__usage_ai-training: one robots.txt signal for AI training
Google, OpenAI, Microsoft and Anthropic plan to honor one shared training signal. What it covers, what it doesn't, and where the Google catch lies.
by Jean Pierre Kolb ·
My robots.txt has 21 entries for AI bots, each one maintained by hand. If you want to refuse AI training today, you keep a list like that provider by provider and hope it's complete. That's what is meant to change: on 30 October 2026, Google, OpenAI, Microsoft and Anthropic announced they will honor one shared signal, __usage_ai-training. I came across it through Jon Henshaw's report on Coywolf (opens in a new tab), then read the draft and the announcements myself.
My verdict up front: this is a good thing, and I've set the token. But it does less than some headlines suggest. For now it replaces nothing, and at Google it comes with a catch that can cost you visibility in Gemini.
The short version
| Question | Answer |
|---|---|
| What is it? | A shared token for an ordinary robots.txt group: User-Agent: __usage_ai-training |
| What does it cover? | Whether you want your content used to train generative AI models, per path |
| What doesn't it cover? | According to the draft: crawling, classic search, real-time AI answers (with an exception at Google, see below), content already retrieved |
| Does it work yet? | Barely. Anthropic has started implementing, Microsoft follows on 1 February 2027, Google in early 2027, OpenAI "in the coming months" |
| Is it a standard? | No, so far it's an Internet-Draft by a single author |
| Enforceable? | No. It's a preference, like everything else in robots.txt |
| What to do? | Set the token, keep your existing bot rules, check the Google catch, sort out Bing first if you use NoArchive/NoCache |
What was actually agreed
The basis is the draft "Machine-Readable Training Preference Signal" (opens in a new tab) by Kevin Kelley (Anthropic), dated 30 October 2026 and valid until 3 May 2027. The idea is strikingly simple: no new directive, no new syntax, just a defined name in the User-Agent line. The example from the draft:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/This means: no training on the whole site, except /press-kit/. A disallowed path expresses a preference against training, an allowed path a preference in favor. Paths, wildcards and the longest-match rule work as always. Parsers that don't know the token treat the group like that of any other unknown bot, so they ignore it. For those parsers, nothing changes.
The four commitments vary in firmness:
| Provider | Timeline | What was actually said |
|---|---|---|
| Anthropic | "over the coming months", no date | Kelley's mail to the working group (opens in a new tab): implementation has started, but rolling it out across all services will take time, and the interpretation may change as standards evolve. I found no page of Anthropic's own on this |
| OpenAI | "in the coming months" | OpenAI's announcement (opens in a new tab): voluntary, details to follow, existing crawler controls stay in effect, the approach may change as the IETF talks evolve. It doesn't name the token |
| Microsoft | 1 February 2027 | Bing blog (opens in a new tab): applied at crawl time from then on, effective only with the page's next recrawl |
| early 2027 | Google Search blog (opens in a new tab): aiming for 2027, without naming the token. Crawler docs (opens in a new tab): __usage_ai-training, early 2027 |
Coywolf writes that Anthropic is the only company using the token today. That goes one step too far for me: Kelley's mail says "started implementing" and expects something usable only "over the coming months", and Anthropic's crawler page (opens in a new tab) doesn't mention the token yet. So don't rely on it being active anywhere yet.
I found no commitment from Meta, Apple, Amazon, Perplexity, Mistral, xAI, ByteDance or Common Crawl. The draft's acknowledgments do list people from Meta, Amazon Web Services and Common Crawl, but expressly for identification only, and that isn't a commitment. The current state is tracked in Vanessa Fox's reference on blocking LLM crawlers (opens in a new tab).
What the token doesn't do
This is where it gets practical, because the naive reading fails on these five points.
It doesn't control crawling. The draft says so verbatim: "This policy token is not a directive for crawling behavior." A crawler may take the preference into account but doesn't have to stay away because of it. So your server load won't necessarily drop. If you want to limit fetching, you still need groups for individual bots or a block at server or CDN level.
User-Agent: * doesn't count. The wildcard group expressly must not be applied to the token. A robots.txt with User-Agent: * and Disallow: / therefore expresses no training preference at all in the sense of the draft. No group of its own, no signal.
It isn't retroactive. According to the draft, the robots.txt in effect when the content was retrieved is what counts. The token won't pull back what's already in training data.
It only covers generative training. Models for classification, ranking or scoring fall outside the definition. Feeding content into AI answers at runtime (grounding, RAG) is also something else.
Private agreements beat it. A licensing contract may override the preference. That's in the draft too.
The Google catch: training and grounding are tied together
For me this is the most important point, especially from a GEO perspective. Google plans to treat an opt-out via the new token as if you had disallowed Google-Extended, unless you name Google-Extended explicitly. And according to Google's crawler docs (opens in a new tab), Google-Extended governs not only training of future Gemini models but also grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.
The result: once Google implements this, this seemingly pure training opt-out also takes you out of grounding in Gemini Apps and on Vertex AI.
User-Agent: __usage_ai-training
Disallow: /Google names the way out itself: if you explicitly allow Google-Extended, you can appear in Gemini Apps answers and receive clicks via grounding. But then, by Google's definition, you also allow Gemini training, because both are tied to Google-Extended.
User-Agent: __usage_ai-training
Disallow: /
User-Agent: Google-Extended
Allow: /So via robots.txt, Google makes you choose per path: both or neither. Google Search itself isn't affected, since according to the docs Google-Extended plays no part in inclusion in the index or as a ranking signal. You don't control AI Overviews or AI Mode with it either. For those, a separate Search Console control (opens in a new tab) has been available worldwide since 31 August 2026, and according to Google it has no effect on AI training in turn.
Microsoft separates things more cleanly. There, training and grounding are two independent settings, and the Bing blog says explicitly that neither touches classic Bing results.
What Microsoft adds on top
Microsoft uses the occasion for its own finer-grained system that applies to Microsoft only. The training preference also exists at page level, where it overrides robots.txt. For grounding there is no robots.txt rule, only the page level:
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">The same works as a header, for instance X-Robots-Tag: Bingbot: ai-training:disallow. The value disallowextended allows grounding only in Microsoft's own applications such as Copilot and excludes grounding services Microsoft provides to third parties, such as Web IQ and the API in Foundry. An allow doesn't lift a robots.txt rule that forbids Bingbot from crawling.
Important if you use NoArchive or NoCache to limit AI use at Bing. These tags lose that meaning:
| Date | What happens |
|---|---|
| 15 December 2026 | Temporary grounding setting in Bing Webmaster Tools for sites using NoArchive/NoCache |
| 1 February 2027 | New preferences apply at crawl time. If Microsoft detects new preferences on a site, it disregards NoArchive/NoCache there |
| 1 March 2027 | Deadline for new preferences and the temporary grounding setting. After that, NoArchive and NoCache no longer count as a preference for training or grounding |
| 30 September 2027 | Temporary grounding settings expire, at the latest by then |
The 1 February row is the stumbling block: if you've been limiting grounding through NoArchive and now set the training token, you could lose that limit from then on, since "Allow" applies without a grounding preference. Microsoft doesn't say explicitly whether a robots.txt rule alone triggers this, but the wording suggests it does. In that case, set ai-grounding first.
The token and the other signals
Several mechanisms now compete, some of them overlapping. An overview so you don't mix them up:
| Mechanism | Where | What | Status |
|---|---|---|---|
__usage_ai-training |
robots.txt group |
Training of generative models | Individual draft, four commitments |
Content-Usage: train-ai=n |
HTTP header and its own robots.txt directive |
Working group categories: train-ai, ai-use, search |
Working group draft (opens in a new tab), still open |
Content-Signal: ai-train=no |
robots.txt and HTTP header |
ai-train, search, ai-input |
From Cloudflare, see Content Signals & C2PA |
GPTBot, ClaudeBot, Google-Extended … |
robots.txt group per provider |
Crawling or use per provider | Still in force, the providers keep their controls |
ai-training/ai-grounding at Bing |
Meta tag, X-Robots-Tag |
Training and grounding per page | Microsoft only, from February 2027 |
| RSL (opens in a new tab) | License: reference in robots.txt |
License and payment terms | Different layer: license rather than preference |
About Content-Signal, which I use myself, an honest note: according to a July 2026 report by Search Engine Journal (opens in a new tab), John Mueller wrote that, as far as known, no crawler or LLM uses this directive. So the token has something that, from what I could find, Content-Signal doesn't have from these four providers: public commitments. In return, Content-Signal is broader: it also covers real-time use, and ai-train isn't limited to generative models. I'm keeping both. They don't conflict as long as the values match.
How solid is the ground?
The signal rests on two unfinished documents, and you should know that before treating it as settled.
First, Kelley's text is an individual draft that the working group hasn't adopted. Kelley's own mail calls it input to the group's work on attaching preferences to content. Second, it does say the token corresponds to the definition the AIPREF working group has arrived at. But the underlying vocabulary draft, version 08 (opens in a new tab), states in capital letters that its contents don't reflect consensus of the group, and marks the very section with the definitions as not yet having consensus. The minutes of the 21 October interim meeting (opens in a new tab) quote the chair: "We haven't declared consensus on anything yet." Kelley's draft only softly asks implementers to update their definitions ("SHOULD consider"). The meaning can still shift.
There is criticism too. According to Search Engine Roundtable (opens in a new tab), Nate Hake, also named in the acknowledgments, points out that the IETF process isn't finished and that significant disputes remain, and criticizes that the preferences don't apply to actions users trigger themselves. By my reading, that concerns the "AI Use" category, not the training definition the token builds on. An open question in the vocabulary points the same way: what "directly provided by the user" means, including whether a URL handed over counts, is unresolved.
I still think it's right that the big providers are moving ahead of completion. A standard without implementations doesn't help you, and the syntax is so close to RFC 9309 that later corrections should do little damage. It just means: keep watching, don't tick it off.
And legally?
This isn't legal advice, but three points belong in your assessment.
Google writes that the proposal is consistent with the EU Code of Practice for general-purpose AI models. The Code's copyright chapter (opens in a new tab) does commit signatories to follow the Robot Exclusion Protocol per RFC 9309, including later versions whose feasibility the IETF demonstrates. On top of that come other protocols that standards bodies have adopted or that are state of the art, widely adopted and generally agreed. Whether a token from an individual draft falls under that remains to be seen.
In Germany, the rights reservation under section 44b(3) of the Copyright Act (opens in a new tab) depends on machine-readability for works accessible online. On 10 December 2025, the Hanseatic Higher Regional Court (opens in a new tab) (press release in German, case no. 5 U 104/24) rejected a reservation worded in natural language on a photo agency's website: the plaintiff hadn't shown it was machine-readable at the time of the 2021 download. The court also based the use on the research exception in section 60d. In the judgment (opens in a new tab) (in German), it calls the required form technology-neutral and mentions robots.txt only as an option discussed in the literature. The ruling isn't final: the Federal Court of Justice heard the case on 3 September 2026 and plans to rule on 17 December 2026 (opens in a new tab) (in German, I ZR 281/25). So what exactly suffices is open, and 17 December is the date to keep in mind for this topic.
The EU Commission consulted on machine-readable opt-out protocols until January 2026 (opens in a new tab), held a workshop (opens in a new tab) on them in June and intends to publish a list of generally agreed solutions. I haven't been able to find such a list as of today.
My conclusion: if you really want to reserve your rights, don't rely on this one token. Use several routes in parallel: the token, the provider-specific rules and a legally worded declaration reviewed by a professional.
What I recommend
You want to allow training (my case). A small group is enough. It isn't technically necessary, since Microsoft for one treats "Allow" as the default anyway. But it makes your position unambiguous:
User-Agent: __usage_ai-training
Allow: /You want to refuse training. Set the token with Disallow: / and keep your bot rules. Until rollout, GPTBot, ClaudeBot, Google-Extended and friends are what actually applies, and they're the only answer for every provider without a commitment. Decide consciously about the Google catch and, if you use NoArchive, about grounding at Bing.
You only want to exclude some areas. Watch out for a subtlety: Disallow: /premium/ alone isn't neutral. In an existing group, any path not disallowed counts as allowed, so by my reading of the draft and RFC 9309 you express a preference in favor of training for everything else. If you don't want that, disallow broadly and allow selectively, as in the press-kit example.
Plus four things to do:
- A separate group, cleanly apart. Consecutive
User-Agentlines share their rules, and under RFC 9309 even a blank line doesn't separate them. Some parsers, such as Python'surllib.robotparser, see that differently. If the token sits without rules of its own directly aboveUser-Agent: GPTBot, the GPTBot group therefore applies to the token as well. So always give it rules of its own. You can do that on purpose too: the draft allows mixing tokens and bots in one group. WithDisallow: /, you then block the bot and express the preference at the same time. - Every subdomain separately. A
robots.txtapplies per host, as always. Nobody evaluates a file in a subfolder such as/blog/robots.txt. - Align your signals.
Content-Signal: ai-train=yesnext to__usage_ai-trainingwithDisallow: /is a contradiction no machine will resolve for you. - Check. My SEO & GEO Analyzer shows the token as its own card in the GEO Score tab, with the preference for the analyzed URL and a warning if it contradicts a site-wide
Content-Signal. It earns no points. The Check robots.txt tab in robots.txt & Sitemap reads it too. For a quick look,curlis enough:
curl -sL https://www.example.com/robots.txt | grep -iE -A5 '^[[:space:]]*user-agent:[[:space:]]*__usage_ai-training'What I did on jpkc.com
This knowledge base is expressly built for AI access, and Content-Signal: ai-train=yes already sits in robots.txt and in the headers. So the group with Allow: / now sits there as well. I don't set Crawl-delay on it, because the token isn't a crawler and fetches nothing. The Host line and all existing bot groups stay as they are, and Google-Extended is already explicitly set to Allow in my file. While I was at it, I added Claude-User and Claude-SearchBot, which used to fall under * and now have a group of their own like the other AI bots.
Before deploying, I compared the old and new versions: with the parser from my tools, which reads groups per RFC 9309, and with the one from Python's standard library, which doesn't and stands in for older parsers here. I tested the 69 user agents already named (including the joke entries Bender and Gort) plus one unknown agent against ten paths. Differences in allow/disallow and Crawl-delay: none. That's to be expected, since no bot carries the token and the two new Claude groups apply only to themselves. So the test only shows that the new groups leave their neighbors unchanged, but with a file your entire visibility depends on, that's exactly what I wanted to measure.
FAQ
Does __usage_ai-training stop AI training immediately?
No. Three of the four providers have only announced support, and Anthropic has started implementing it. On top of that, it's a preference, and bots without a commitment or with bad intentions don't care. For a hard block, you need rules at server or CDN level.
Can I delete my GPTBot and ClaudeBot rules now?
No. Google, OpenAI and Anthropic explicitly keep their own controls, and those partly govern something else: a Disallow for GPTBot affects fetching, the new token doesn't. Besides, the shared rule only covers the providers that have committed.
Does a training opt-out hurt my Google ranking?
According to Google's docs, no. Google-Extended affects neither inclusion in Search nor ranking. But once Google implements this, you lose grounding in Gemini Apps and on Vertex AI there unless you explicitly allow Google-Extended. That's the price many are likely to overlook.
Do I stay visible in ChatGPT, Claude and Copilot if I block training?
According to the draft, the token is defined for training only. OpenAI and Anthropic run separate bots for search and user fetches. You control search separately there, but for ChatGPT-User, OpenAI warns that robots.txt rules may not apply to user fetches. Microsoft explicitly separates training and grounding. Only at Google does the token reach grounding, through Google-Extended.
Is this a legally sound rights reservation?
Nobody can promise you that today. Case law on machine-readability isn't settled, and Kelley's draft says nothing about legal effect. For binding questions, you need a professional.
What does the token do for me if I allow training?
Mainly clarity. Without a group you've expressed no preference in the sense of the draft; with Allow: / you've expressed an unambiguous one. For a knowledge base like this one, that fits the purpose, and the two lines cost nothing.
Further reading
The signal's older sibling, including the use field and path rules, is covered in Content Signals & C2PA. How a request becomes a block at network level is shown in 15 September 2026: Cloudflare Changes the Crawler Defaults. The rest of crawler management is in Structured Data & Technical GEO, and What is GEO? provides the framework.