An AI training opt out used to mean one robots.txt rule per AI company: Google-Extended for Google, GPTBot for OpenAI, ClaudeBot for Anthropic, a noarchive tag for Microsoft. On October 30, 2026, that started to change. Google, Microsoft, OpenAI and Anthropic said they will honor one shared rule, User-Agent: __usage_ai-training, that tells every participating company “don’t train generative models on this content” in a single step (Google Search Central, Vanessa Fox).
The short answer for site owners: nothing is enforced yet. Microsoft starts reading the new rule on February 1, 2027 and Google says “early 2027”. The rule covers training only, not AI answers, not search, and not pages a chatbot opens because a user asked. And the spec behind it is an individual IETF draft that “has no formal standing in the IETF standards process” (IETF Datatracker). You can add it now at no risk, but you still need the company-specific rules, and you need to decide separately whether you want to be cited in AI answers.
Below: what the rule says, who supports it and from when, what it doesn’t block, Microsoft’s separate grounding controls, and the exact robots.txt lines for the three most common choices.
Key takeaways
- One line, many companies.
User-Agent: __usage_ai-trainingwithDisallow: /states a preference against AI training to every company that supports the token. - Nothing changes until 2027. Microsoft honors it from February 1, 2027; Google says early 2027; OpenAI “in the coming months”; Anthropic has started implementing it with no date given.
User-agent: *does not count. Your existing wildcard block expresses no training preference under this draft. You must name the token.- At Google, the shared opt-out also removes Gemini grounding unless you give Google-Extended its own Allow group. It has no effect on Google Search, AI Overviews or AI Mode.
- Bing users on noarchive/nocache must act by March 1, 2027, or Microsoft stops reading those tags as AI preferences.
- It’s a preference, not a wall. robots.txt is “not a form of access authorization”; bots that ignore it still get in.
What changed (dates)
| Date | Event |
|---|---|
| Oct 30, 2026 | Anthropic’s Kevin Kelley submits draft-kelley-aipref-training-preference-signal-00 to the IETF. Google, Microsoft, OpenAI and Anthropic announce support. |
| Oct 30, 2026 | Microsoft announces separate AI Training and AI Grounding preferences replacing NoArchive/NoCache. |
| Dec 15, 2026 | Temporary grounding preferences open in Bing Webmaster Tools (only for sites using NoArchive/NoCache on that date). |
| Feb 1, 2027 | Microsoft starts honoring the new preferences as it crawls. |
| Early 2027 | Google plans to support __usage_ai-training. |
| Mar 1, 2027 | Microsoft stops reading NoArchive/NoCache as training or grounding preferences. |
| May 3, 2027 | The -00 draft expires unless it’s updated or replaced (normal for Internet-Drafts). |
| Sep 30, 2027 | Bing’s temporary grounding preferences expire. |
What the AI training opt out rule actually says
The mechanism comes from the IETF’s AI Preferences (AIPREF) working group. The draft was written by Kevin Kelley, listed with Anthropic as his affiliation, and submitted on October 30, 2026 (IETF Datatracker). Its idea is deliberately boring: no new robots.txt grammar. You write an ordinary robots.txt group, but instead of a crawler name on the User-Agent line you use a “policy token” that stands for a purpose.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That example from the draft reads as “a preference against AI training for everything except /press-kit/, and a preference in favor of it there,” as Kelley put it when he announced it on the working group’s mailing list (IETF mail archive). Path matching and longest-match precedence work exactly as in the Robots Exclusion Protocol, RFC 9309, so anyone who has written a Disallow line already knows the syntax.
What “AI training” means here
The draft borrows its definition from the working group’s vocabulary document: AI training is “using an asset to modify the learned parameters of a generative AI model.” Models used only for non-generative tasks are excluded, and so are models used “for classification, ranking, or scoring.” In plain terms, the token is about building chatbots and image generators, not search ranking.
Three details people will get wrong
- Wildcards don’t match. The draft says “wildcard groups (i.e., User-Agent: *) MUST NOT be interpreted as matching the policy token.” A site with only
User-agent: *rules has expressed no training preference at all. - No group means no preference. If the file contains no
__usage_ai-trainingline, “no preference has been expressed.” Kelley pointed out this is true of every robots.txt file published today (IETF mail archive). - It isn’t a crawl rule. “This policy token is not a directive for crawling behavior.” Disallowing training doesn’t stop Googlebot or Bingbot from fetching your pages. Crawling is still controlled by the rules for each crawler’s own name.
Who supports the shared AI training opt out, and from when
All four companies announced on the same day, but their timelines and wording differ. This is the state as of November 2, 2026.
| Company | Existing training control | Shared token status | How it’s treated |
|---|---|---|---|
| Google-Extended (since 2023) | “Early 2027” | As if you disallowed Google-Extended, unless Google-Extended is named explicitly | |
| Microsoft (Bing, Copilot) | NoArchive / NoCache tags | From Feb 1, 2027 | Training preference; page-level tag can override robots.txt |
| OpenAI | GPTBot | “In the coming months” | GPTBot rules continue to apply |
| Anthropic | ClaudeBot | Started implementing, no date | No details published beyond the draft |
Sources: Google, Microsoft Bing Blog, Vanessa Fox (OpenAI and Anthropic status). OpenAI’s own announcement page blocked automated reading during my research, so its status here comes from Fox and from PPC Land, which reported that “OpenAI has published its own commitment to the shared signal.”
Google: the Gemini grounding catch
Google’s post is the one to read carefully. “When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended” (Google Search Central).
Google-Extended is not only a training switch. Google’s crawler documentation says it also controls grounding in Gemini Apps and Grounding with Google Search on Vertex AI (Google crawler docs). Google’s post adds that “websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps.” So a blanket shared opt-out also takes you out of Gemini’s cited answers, unless you add a separate Google-Extended Allow group.
What it doesn’t touch: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” (Google crawler docs). Fox’s reference table lists the effect on AI Overviews and AI Mode as “no effect” (Vanessa Fox). Google also still has its separate Search Console opt-out for Search’s generative AI features (Search Engine Roundtable).
OpenAI and Anthropic: keep the named rules
OpenAI’s crawler docs still describe GPTBot as the training switch: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.” Its search crawler, OAI-SearchBot, is separate, and the two settings are independent (OpenAI). Anthropic works the same way: “When a site restricts ClaudeBot access, it signals that the site’s future materials should be excluded from our AI model training datasets,” while Claude-SearchBot and Claude-User handle search and user requests (Anthropic). Until both companies say the shared token is live, keep the GPTBot and ClaudeBot groups.
What the shared rule does not block
This is where most of the confusion will come from. The token is narrow on purpose.
| Use of your content | Covered by __usage_ai-training? |
What controls it instead |
|---|---|---|
| Training generative models | Yes (once each company supports it) | Also Google-Extended, GPTBot, ClaudeBot |
| Classic search indexing and ranking | No | Googlebot, Bingbot rules; noindex |
| Citations in AI answers (grounding) | No, except Gemini via the Google-Extended mapping | Bing ai-grounding tags; OAI-SearchBot; Claude-SearchBot; Search Console opt-out |
| Pages a chatbot opens because a user asked | No | Mostly not robots.txt; server or CDN blocks |
| Bots that ignore robots.txt | No | Firewall / CDN bot management |
User-triggered fetches
Fox’s summary is that the signal “says nothing about web search, AI answers, or user-triggered requests” (Vanessa Fox). OpenAI says of ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply” (OpenAI). Publisher Nate Hake, who took part in the working group, said on X that he is still unhappy the current draft “basically exempts the preferences from applying to anything a user does,” and that “some significant areas of dispute remain” (quoted by Search Engine Roundtable).
It’s a request, not a lock
RFC 9309 states that robots.txt rules “are not a form of access authorization” (RFC 9309). Google’s Gary Illyes made the same point on the Odys Podcast: “Anyone can impersonate Googlebot in about 30 seconds,” and robots.txt is a suggestion, not a security control; verify Googlebot by IP and reverse DNS and use a firewall or CDN for real protection (PPC Land). Anthropic also warns that blocking its crawler by IP “may not work correctly or persistently guarantee an opt-out,” because it stops the bot from reading your robots.txt (Anthropic).
Bing’s separate training and grounding controls
Microsoft went further than the others. Besides honoring the shared training token, it is replacing its 2023 NoArchive/NoCache controls with two independent preferences (Microsoft Bing Blog):
- AI Training: the shared robots.txt token for a site or path, plus a page-level tag that overrides robots.txt for that page.
- AI Grounding: page level only. Allow (the default), DisallowExtended (your content can inform Microsoft’s own “AI Mode, AI Overview, Copilot, and Microsoft 365” but not grounding services Microsoft sells to third parties, such as Web IQ and the grounding API in Foundry), or Disallow. “There is no site-wide robots.txt grounding preference.”
Page-level syntax from Microsoft’s post:
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
X-Robots-Tag: Bingbot: ai-training:disallow
X-Robots-Tag: Bingbot: ai-grounding:disallowextendedThree conditions Microsoft spells out: these settings “govern use of content, not crawl access”; “publishing a setting is not the same as activating it”, because Bing must recrawl the page first (IndexNow can help but doesn’t guarantee a date); and “neither choice guarantees inclusion, traffic, or payment.” Classic Bing rankings aren’t affected either way (Search Engine Roundtable).
If you use noarchive or nocache today
After March 1, 2027, Microsoft will no longer read NoArchive or NoCache as a training or grounding preference. To keep a grounding restriction through the switch, publish the new ai-grounding tag on each page and record a temporary preference in Bing Webmaster Tools, which opens on December 15, 2026. If you don’t, those pages default to Allow for grounding until Bing recrawls them (Microsoft Bing Blog).
Should you opt out? What the evidence says about AI visibility
Blocking training is a values and licensing decision. Blocking grounding is a traffic decision, and there’s very little public data on it. The one fresh data point is small. Cyrus Shepard used Google’s opt-out controls to remove one site from AI Overviews and AI Mode, then compared before and after. He estimated a click-through rate of “~0.9%” on AI impressions, which were about 8% of the site’s total impressions. The surprise: “regular organic clicks ALSO DROPPED when AI features were disabled” (Search Engine Roundtable).
Treat that carefully. It’s one site, AI Overviews still appeared on the results page during the test (only this site was missing from them), and sitewide average position moved because of new posts and brand searches (PPC Land). It’s a reason to test before you block AI answers, not proof either way.
My view for marketers and businesses that sell services: keep grounding open, because citations in AI answers are a discovery channel, and Microsoft itself warns that grounding Disallow or DisallowExtended “will significantly reduce your content’s visibility in AI-generated experiences.” Decide on training separately. If your content is your product (a publisher, a course, a paid research library), the training opt-out is worth adding now.
Want this applied to your accounts? Get in touch with Zain. I reply on WhatsApp, email and LinkedIn.
Robots.txt setup: three common choices
All three keep search crawling exactly as it is. Add the groups below to your existing file; don’t replace your current rules. The draft says the token is matched “exactly, apart from case,” like any product token, so copy it exactly, with both leading underscores (IETF Datatracker).
1. Opt out of training everywhere, stay in search and AI answers (except Gemini)
# Shared AI training opt out (IETF draft; Microsoft Feb 1 2027, Google early 2027)
User-Agent: __usage_ai-training
Disallow: /
# Company rules that work today
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Note that disallowing Google-Extended also removes you from Gemini grounding (Google crawler docs). For Bing before February 2027, Fox suggests a noarchive tag; from that date the shared group above applies (Vanessa Fox).
2. Opt out of training for others, but keep Gemini answers
User-Agent: __usage_ai-training
Disallow: /
User-agent: Google-Extended
Allow: /The draft says specific expressions “SHOULD take precedence over general ones” and lets companies treat their own tokens as more specific (IETF Datatracker). Fox describes this exact pattern: Google may use the content (including for training), while other companies that support the token may not (Vanessa Fox). That’s the trade-off: you can’t separate Gemini training from Gemini grounding with Google-Extended.
3. Protect one section only
User-Agent: __usage_ai-training
Disallow: /courses/
Disallow: /members/Everything outside those paths has an explicit “no preference” in this group. Remember that anything behind a login should be protected by the login, not by robots.txt.
Checklist
| When | Action |
|---|---|
| This week | Decide training and grounding separately. Write it down per site and per section. |
| This week | Add the __usage_ai-training group if you want out of training. Keep GPTBot, ClaudeBot, Google-Extended rules. |
| This week | Check your CDN or host settings: some write robots.txt for you. |
| Before Dec 15, 2026 | List pages using noarchive/nocache. |
| Dec 15, 2026 to Mar 1, 2027 | Add ai-training / ai-grounding tags and record the temporary grounding preference in Bing Webmaster Tools. |
| Feb–Mar 2027 | Watch Google’s crawler docs and the IETF draft for changes before treating the rule as final. |
How it fits with Cloudflare content signals and the EU AI Act
Cloudflare’s Content-Signal line
Cloudflare launched its own Content Signals Policy in September 2025, with a line such as Content-Signal: search=yes, ai-train=no, and said it had turned on managed robots.txt for “over 3.8 million domains” (Cloudflare). Cloudflare itself says content signals “are not technical countermeasures against scraping.” In July 2026 Google’s John Mueller said that, as far as he knew, “none of the crawlers / llms use the “content-signal” robots.txt directives” (Search Engine Roundtable). The new token is the one the four big model providers have actually committed to. If you’re on Cloudflare, check what its managed robots.txt already says before you edit the file. Interestingly, the AIPREF working group is chaired by Mark Nottingham of Cloudflare and Suresh Krishnan of Cisco (IETF Datatracker).
The EU angle
Google says the proposal is consistent with the EU’s Code of Practice for general-purpose AI model providers, “that recognizes robots.txt as a primary method for web publishers to express machine-readable preferences” (Google Search Central). The Code was published on July 10, 2025, is voluntary, and its signatories include Amazon, Anthropic, Google, Microsoft and OpenAI (European Commission). That’s why a robots.txt line matters more than it used to: for signatories it’s part of how they show copyright compliance in Europe. It still doesn’t create a licence or payment, as Microsoft notes for its own controls.
For a related case of an AI company changing how it uses site content, see my post on Meta’s commercial terms and controlling its crawlers. If you want the wider picture on getting found in search and AI answers, my free book on getting found in search and AI answers covers it, and my free SEO master checklist covers the technical basics.
FAQ
How do I opt out of AI training on my website?
Add a robots.txt group with User-Agent: __usage_ai-training and Disallow: /, and keep the company rules that work today: GPTBot (OpenAI), ClaudeBot (Anthropic) and Google-Extended (Google). Microsoft reads the shared rule from February 1, 2027; Google plans early 2027.
Does the AI training opt out remove my site from Google Search or AI Overviews?
No. Google-Extended, which the shared rule maps to at Google, “does not impact a site’s inclusion in Google Search,” and Fox lists no effect on AI Overviews or AI Mode. It does remove you from Gemini Apps grounding unless you explicitly allow Google-Extended.
Does User-agent: * block AI training?
Not under this draft. Wildcard groups “MUST NOT be interpreted as matching the policy token.” You need a group that names __usage_ai-training.
Is __usage_ai-training an official standard?
Not yet. It’s an individual Internet-Draft dated October 30, 2026, which the IETF says “has no formal standing in the IETF standards process.” Participants say disputes remain, so the details can change.
What’s the difference between AI training and AI grounding?
Training changes a model’s learned parameters. Grounding uses your page as input when an AI writes an answer, often with a citation. Bing lets you control them separately; the shared token covers training only.
Want this applied to your accounts? Get in touch with Zain. I reply on WhatsApp, email and LinkedIn.
Sources
- Google Search Central Blog: Supporting a shared AI training control for websites (Oct 30, 2026)
- Google: Google’s common crawlers (Google-Extended)
- Microsoft Bing Blog: Your content, your choice (Oct 30, 2026)
- IETF: draft-kelley-aipref-training-preference-signal-00
- IETF ai-control mailing list: New draft announcement (Oct 30, 2026)
- Search Engine Roundtable: Google Adopts New AI Training Control
- Search Engine Roundtable: Microsoft Bing AI training and grounding controls
- Search Engine Roundtable: AI Overviews & AI Mode 0.9% CTR opt-out study
- Search Engine Roundtable: Mueller on Content Signals
- PPC Land: Google and Bing back one robots.txt rule to opt out of AI training
- Vanessa Fox: Complete reference to blocking LLM crawlers
- OpenAI: Overview of OpenAI crawlers
- Anthropic: How site owners can block the crawler
- Cloudflare: Content Signals Policy
- RFC 9309: Robots Exclusion Protocol
- European Commission: The General-Purpose AI Code of Practice
