Google, OpenAI, Microsoft and Anthropic will support a shared robots.txt signal allowing publishers to reject AI training. But the new agreement does not stop AI companies from crawling websites, using content in AI-generated answers or relying on material they have already collected.
The relationship between artificial intelligence companies and the websites that supply their information is entering a new phase. For years, website owners have faced an increasingly complicated choice: allow AI companies to access their content, or attempt to block each company’s crawler individually.
That could soon become considerably easier.
On October 30, 2026, Google, OpenAI, Microsoft and Anthropic announced support for a common technical signal that will allow publishers to express whether their website content can be used to train generative AI models. Instead of maintaining separate instructions for multiple AI developers, website owners will be able to communicate their preference through a shared policy token in their existing robots.txt file.
The proposed mechanism, called __usage_ai-training, comes from the Internet Engineering Task Force’s (IETF) AI Preferences working group. The proposal was submitted by Kevin Kelley, an Anthropic technical lead involved in internet standards.
Anthropic has already begun supporting the directive, while Microsoft plans to implement it from February 1, 2027. Google expects to follow in early 2027, and OpenAI says it intends to introduce support in the coming months.
At first glance, this looks like a major victory for independent publishers, journalists, bloggers and other website operators who want greater control over how their work is used by AI systems.
But there is a significant detail that deserves more attention: the new signal controls AI training preferences, not necessarily AI access to the content itself.
A website could tell participating companies not to use its articles to train future models while still allowing those companies to crawl its pages, retrieve information and potentially use that information to generate answers.
That distinction could determine how valuable the new agreement becomes for the publishing industry.
Table of Contents
What exactly have Google, OpenAI, Microsoft and Anthropic agreed to?
Until now, website owners who wanted to prevent AI training had to understand and manage different crawler instructions.
For example, OpenAI operates GPTBot, Anthropic operates ClaudeBot, and Google provides Google-Extended as a separate control for certain generative AI uses.
Microsoft also introduced publisher controls through its NoArchive and NoCache mechanisms.
Maintaining these instructions individually creates additional work, particularly for smaller publishers who do not have dedicated technical teams.
The new proposal attempts to simplify that process.
Under the shared mechanism, a website can add a dedicated User-agent group to its robots.txt file:
User-agent: __usage_ai-training
Disallow: /This expresses a preference against using the website’s content for generative AI training.
Website owners can also make exceptions for selected directories.
For example:
User-agent: __usage_ai-training
Disallow: /
Allow: /press-kit/This communicates that the publisher does not want its general website content used for AI training but is willing to permit training use of material in the press-kit directory.
The proposal uses the existing robots.txt structure rather than introducing an entirely new file format or a new set of crawling directives.
That is an important technical advantage: participating AI companies can adopt a common interpretation without requiring website owners to learn a completely different system.
The proposal also allows preferences to be expressed at the individual path level rather than forcing publishers to make an all-or-nothing decision for an entire domain.
The IETF draft describes the signal as a content-use preference, not a conventional crawling instruction.
The biggest misunderstanding: AI training, crawling and AI answers are not the same thing
This is arguably the most important part of the announcement.
Many website owners may assume that adding the new directive prevents AI companies from accessing their articles altogether.
That is not what it does.
There are at least three different activities involved in how AI systems interact with online content.
1. AI training: Teaching future models
Training involves using content to modify the learned parameters of a generative AI model.
For example, an AI developer might collect large amounts of publicly available text, including articles, educational resources and technical documentation, to improve a model’s ability to understand and generate language.
The new policy token is specifically designed to express whether website owners permit this type of use.
A publisher who disallows AI training is communicating that its content should not be used for this purpose by participating providers.
2. AI crawling: Accessing and retrieving website content
Crawling is the process through which automated systems discover and retrieve web pages.
AI companies may operate crawlers for different purposes, including model development, search, user-requested retrieval and gathering information for AI-powered services.
The new policy token does not automatically prevent these crawlers from visiting a website.
If a publisher wants to prevent a particular crawler from accessing its pages, it still needs the appropriate crawler-specific rules in robots.txt.
For example:
User-agent: GPTBot
Disallow: /This is a separate instruction from the shared training preference.
The IETF proposal explicitly says that content-use preferences and crawler traffic-management signals are independent. A participating provider may choose to avoid retrieving content when a relevant use is disallowed, but the shared token itself is not a crawling prohibition.
3. AI grounding: Using content to generate answers
Grounding is another activity that deserves attention.
When someone asks an AI assistant a question, the system may retrieve relevant information from websites and use it to construct an answer.
For instance, someone could ask an AI assistant to explain a recent technology announcement. The assistant might retrieve several news articles, combine information from those sources and produce a summary with links or citations.
This is different from using those articles to train the model’s underlying parameters.
A publisher might therefore be comfortable allowing AI systems to reference its latest reporting while refusing permission to use that reporting for future model training.
The new shared signal does not independently provide this broader choice across all four companies.
Microsoft’s announcement is particularly interesting because it introduces separate controls for AI training and AI grounding.
This distinction could become one of the most important issues in the next phase of the relationship between publishers and AI platforms.
Microsoft’s announcement goes further than the shared agreement
Microsoft has introduced an additional layer of publisher control that separates AI training from the use of content in generated answers.
According to its October 30 announcement, Bing will support two independent preferences.
AI Training: Website owners can express whether their content can be used to train generative AI models. This includes the shared IETF policy token, with Microsoft also extending the preference to individual pages.
AI Grounding: Publishers can separately control whether their content is used to inform AI-generated responses.
Microsoft is also introducing a more granular grounding option called DisallowExtended.
This setting allows publishers to permit grounding in Microsoft’s own experiences, including AI Mode, AI Overviews, Copilot and Microsoft 365, while excluding third-party grounding services such as Web IQ and grounding through APIs in Microsoft Foundry.
A full grounding restriction can prevent content from being used to inform AI-generated answers.
However, Microsoft makes an important commercial point: restricting grounding can reduce a publisher’s visibility in AI experiences.
Conversely, allowing grounding makes content eligible to inform AI responses, potentially with links and citations directing users back to the original website.
Neither choice guarantees that a publisher will receive traffic, be included in an answer or receive payment.
Microsoft’s implementation timeline
| Date | What publishers need to know |
|---|---|
| December 15, 2026 | Temporary grounding transition controls open in Bing Webmaster Tools for eligible sites using NoArchive or NoCache. |
| February 1, 2027 | Microsoft begins honoring the new training and grounding preferences as it crawls websites. |
| March 1, 2027 | Deadline for eligible publishers using legacy NoArchive or NoCache controls to complete the specified transition steps for grounding restrictions. |
| September 30, 2027 | Temporary grounding preferences expire, unless replaced earlier following a recrawl. |
There is an important qualification here. The December 15 and March 1 transition steps apply specifically to websites using NoArchive or NoCache under Microsoft’s stated conditions. They are not universal deadlines requiring every website to make changes.
Microsoft also says that the new preferences will not affect a website’s eligibility for ordinary Bing search results.
That separation matters because publishers have long worried that restricting AI usage could also damage their conventional search visibility.
Source: Microsoft Bing’s official publisher announcement.
Will blocking AI training affect Google rankings?
For publishers, this may be the most commercially important question.
The short answer is that the shared training signal is not designed to block Google Search indexing.
Google already offers Google-Extended, a robots.txt control that allows website owners to manage certain uses of content for Gemini-related model training and grounding.
Google has now announced that it plans to interpret the shared training opt-out as equivalent to disallowing Google-Extended, provided the website has not explicitly specified Google-Extended.
Google also states that Google-Extended does not affect a website’s inclusion in Google Search or act as a Google Search ranking signal.
This means publishers should not automatically assume that refusing AI training will cause their articles to disappear from traditional search results.
However, there is an important distinction between conventional search visibility and visibility inside AI-powered products.
A website might remain indexed in Google Search but have different eligibility for certain AI-related uses.
Publishers therefore need to understand which specific permission they are changing before modifying their robots.txt file.
For a website that depends heavily on organic traffic, confusing these controls could create unnecessary problems.
Google’s official explanation.
The economic question nobody can ignore: Who benefits when AI uses publishers’ work?
The announcement is also part of a much bigger debate about the economics of the internet.
Traditional publishing operates through a relatively understandable business model.
A publisher invests in research, reporting, writing, editing, hosting and distribution. Readers visit its website, and the publisher earns revenue through advertising, subscriptions, sponsorships or other commercial arrangements.
AI changes that relationship.
Instead of visiting multiple websites, a user may ask an AI assistant a question and receive a single consolidated answer.
The AI system may use information from numerous publishers without requiring the user to visit every original article.
This creates a potential economic imbalance.
The publisher bears the cost of producing original information, while an AI platform can use that information to improve its products and deliver answers within its own interface.
Even when an AI answer includes a citation, the reader may have little reason to click through to the original website.
This raises three important questions:
- Should publishers receive compensation when their original reporting contributes to commercial AI products?
- Should AI companies be required to distinguish between content used for training and content retrieved for individual answers?
- Should publishers have the ability to negotiate different permissions for different types of content?
The shared robots.txt signal addresses only part of this debate.
It makes it easier for publishers to communicate a preference against AI training. It does not establish a payment system, guarantee attribution or create a licensing agreement.
It also does not guarantee that a website refusing training will receive more visitors.
In other words, the agreement gives publishers a clearer way to say no, but it does not answer how they should be paid when they say yes.
Why smaller publishers could benefit more than large media companies
Large publishers often have legal teams, technical specialists and commercial departments that can negotiate licensing agreements or manage crawler policies.
Independent publishers and small websites usually do not have those resources.
A website operated by one person may need to manage multiple AI crawlers while also handling search engine optimisation, website security, content production and advertising.
A shared policy token could significantly reduce that administrative burden.
Consider a small educational website publishing original explanations, diagrams and research-based articles.
Its owner may want Google to index the articles, allow readers to find them through search engines and permit AI assistants to retrieve relevant information when users ask questions.
At the same time, the owner may not want the entire content library used to train commercial generative AI models without an agreement.
A shared training preference provides a simpler way to communicate that position to participating companies.
However, small publishers face another challenge: refusing AI training may not automatically improve their business.
They still need to attract visitors, build a recognisable brand, earn backlinks and develop direct relationships with readers.
The signal is a tool for managing content permissions, not a replacement for a sustainable publishing strategy.
What about content AI companies have already collected?
This is one of the biggest limitations of the announcement.
The new directive is forward-looking.
It cannot automatically retrieve or delete articles, images or other material that companies collected before a publisher expressed its preference.
If an AI developer has already acquired a publisher’s content through earlier crawling, licensing, data partnerships or other sources, adding the new robots.txt rule does not reverse that historical collection.
Nor does the shared signal itself establish a mechanism for removing previously collected material from existing datasets or trained models.
This creates a difficult practical question.
If a publisher opts out today, how can it know whether its content was already used in an earlier training process?
The new mechanism does not provide a universal audit system answering that question.
It also does not establish a standard public reporting process showing how much content each AI provider has collected from a particular website.
For publishers seeking meaningful control, transparency about historical data use may ultimately be just as important as future opt-out controls.
Can AI companies simply ignore the new robots.txt signal?
Technically, the mechanism relies on participating companies choosing to interpret and respect the policy token.
The IETF proposal is not a technical barrier that makes it impossible for another system to access a website.
Robots.txt has historically worked as a communication mechanism between website owners and cooperative automated systems.
It is not authentication, encryption or an access-control firewall.
A crawler that does not recognise the policy token may simply ignore that group. A malicious or non-compliant crawler can also disregard robots.txt instructions altogether.
The draft itself makes clear that this is a preference signal and that implementations may have their own precedence rules.
There is another important detail: the IETF document is still an Internet-Draft, not a completed, universally adopted internet standard.
An Internet-Draft is a work in progress. It can be revised, replaced or abandoned as the standards process develops.
That means the four companies’ announcements are significant industry commitments, but they should not be confused with a legally enforceable global rule.
Enforcement, transparency and accountability remain unresolved.
Why Meta’s position matters
Another major question is whether other AI companies will join the agreement.
Meta is particularly important because of its large AI ecosystem and its use of publicly available information to develop AI products.
The company has expressed support for the general idea of a common signal but has not made a confirmed commitment to implement this specific directive, according to reporting by The Verge.
That leaves an important gap.
A common standard becomes substantially more valuable when the companies that collect and use large volumes of online information follow the same rules.
If some major AI developers adopt the token while others do not, website owners will still need to maintain additional crawler-specific instructions.
The same problem could arise with smaller AI developers, data brokers and third-party content collection services.
The success of the proposal will therefore depend not only on the four initial participants but also on wider industry adoption.
What website owners should do now
For publishers, the announcement is a reason to review existing crawler policies rather than immediately changing every robots.txt rule.
Here are the practical steps worth considering.
1. Audit your existing robots.txt file
Identify the crawler-specific rules already present on your website.
Check whether you have separate instructions for GPTBot, ClaudeBot, Google-Extended, Bingbot and other relevant crawlers.
Do not remove existing rules simply because a shared policy token has been announced.
2. Decide whether you want to allow AI training
There is no universally correct answer.
A publisher producing original research, premium educational material or exclusive reporting may prefer to restrict training.
Another publisher may want its content to be used by AI models because broader exposure could support its brand or business strategy.
The decision should reflect the website’s commercial objectives.
3. Keep training and crawling permissions separate
If your intention is to prevent training but preserve conventional search discovery, do not accidentally block Googlebot or Bingbot.
The new token is not a substitute for ordinary crawler instructions.
4. Understand AI grounding before restricting it
If your website receives visitors from AI-generated answers, a broad grounding restriction could reduce its eligibility to appear as a source in those experiences.
Publishers should evaluate the potential trade-off between protecting content and maintaining visibility.
5. Monitor implementation announcements
The companies have different implementation schedules.
Anthropic has already started supporting the shared directive, while the other announced timelines extend into the coming months and early 2027.
Publishers should verify actual support before assuming that every participating provider is already respecting the signal.
6. Monitor your server logs and referral traffic
Server logs can help identify which recognised crawlers are visiting your website.
Analytics can help reveal whether visitors are arriving from AI products, conventional search engines or other sources.
This information can help publishers make better decisions about whether to permit or restrict particular uses of their content.
What does this mean for independent websites such as Wisdom Imbibe?
For independent digital publishers, the announcement deserves particular attention.
A website such as Wisdom Imbibe invests in creating original technology coverage, educational explanations, analysis and other information for readers.
That content has value beyond the number of visitors who arrive on a particular day. It can also become material that AI systems retrieve, summarise, reference or potentially use during model development.
The new shared signal creates an opportunity to establish a clearer distinction between allowing a website to be discovered and allowing its content to be used for AI training.
But there is no reason to rush into a blanket AI block.
For an advertising-supported publisher, conventional search traffic remains important. AI-generated answers may also provide another discovery channel, although the amount of traffic they send varies.
A sensible strategy is to preserve access for legitimate search crawlers, review AI training permissions separately and evaluate the impact of AI grounding before restricting it.
Most importantly, publishers should continue investing in original reporting, first-hand insights, useful explanations and recognisable editorial expertise.
The more distinctive a publisher’s work becomes, the more valuable it can be to readers and the more difficult it is to replace with generic summaries.
The bigger picture: The internet is moving towards permission-based AI content use
The shared policy token may look like a small technical change, but it represents a meaningful shift in how the publishing industry communicates with AI developers.
For decades, robots.txt primarily helped website owners communicate crawling preferences to automated systems.
Now, the same familiar file is becoming part of a broader conversation about how online information can be used.
The distinction between discovering content, training on it and using it to generate answers is becoming increasingly important.
The four companies’ agreement is a step towards making those distinctions understandable to machines.
However, the bigger challenge remains unresolved: how can the internet support powerful AI systems without undermining the publishers, researchers, educators and independent creators whose work makes those systems useful?
A shared opt-out signal cannot solve that entire problem.
But it does give website owners something they have been asking for: a simpler way to communicate their preferences across several major AI platforms.
The next test will be whether those companies consistently honour the signal, whether other providers join and whether publishers eventually gain meaningful transparency and economic benefits from the use of their work.
For website owners, the future of AI content control may depend less on whether AI companies can access their websites and more on what those companies are allowed to do with the information they find.
Frequently Asked Questions
1. What is the new AI training opt-out signal?
It is a proposed shared robots.txt policy token called __usage_ai-training. It allows website owners to express whether their content can be used for generative AI model training by participating companies.
2. Which AI companies support the new robots.txt rule?
Google, OpenAI, Microsoft and Anthropic have announced support for the shared signal. Their implementation schedules differ.
3. Does blocking AI training prevent Google from indexing my website?
No. The shared training preference is separate from conventional Google Search crawling and indexing. Google says its Google-Extended control does not affect search inclusion or rankings.
4. Does the new rule stop AI companies from crawling websites?
No. It is a content-use preference, not a crawling prohibition. Website owners need separate crawler-specific rules if they want to restrict access.
5. Can website owners allow AI search answers but block AI training?
The shared signal alone does not provide a universal training-versus-grounding choice. Microsoft is introducing separate controls for these activities, while Google already has Google-Extended controls covering certain training and grounding uses.
6. Can the new rule remove content already used to train AI models?
No. It does not automatically remove previously collected content or reverse earlier model training.
7. Is the new AI opt-out rule legally enforceable?
The proposal is an Internet-Draft within the IETF standards process. The companies have announced voluntary support, but the draft itself does not establish a universal legal enforcement mechanism.
8. When will the new robots.txt signal become available?
Anthropic has already begun supporting it. Microsoft plans to start on February 1, 2027, Google expects implementation in early 2027, and OpenAI aims to introduce support in the coming months.
Also Read: