# __usage_ai-training: one robots.txt signal for AI training

> Google, OpenAI, Microsoft and Anthropic plan to honor one shared training signal. What it covers, what it doesn't, and where the Google catch lies.

Source: https://www.jpkc.com/db/en/blog/usage-ai-training-robots-txt/

My `robots.txt` has 21 entries for AI bots, each one maintained by hand. If you want to refuse AI training today, you keep a list like that provider by provider and hope it's complete. That's what is meant to change: on 30 October 2026, Google, OpenAI, Microsoft and Anthropic announced they will honor one shared signal, `__usage_ai-training`. I came across it through [Jon Henshaw's report on Coywolf](https://coywolf.com/news/seo/google-openai-microsoft-and-anthropic-back-a-shared-robotstxt-ai-training-signal/), then read the draft and the announcements myself.

My verdict up front: this is a good thing, and I've set the token. But it does less than some headlines suggest. For now it replaces nothing, and at Google it comes with a catch that can cost you visibility in Gemini.

## The short version

| Question | Answer |
| --- | --- |
| **What is it?** | A shared token for an ordinary `robots.txt` group: `User-Agent: __usage_ai-training` |
| **What does it cover?** | Whether you want your content used to train generative AI models, per path |
| **What doesn't it cover?** | According to the draft: crawling, classic search, real-time AI answers (with an exception at Google, see below), content already retrieved |
| **Does it work yet?** | Barely. Anthropic has started implementing, Microsoft follows on 1 February 2027, Google in early 2027, OpenAI "in the coming months" |
| **Is it a standard?** | No, so far it's an Internet-Draft by a single author |
| **Enforceable?** | No. It's a preference, like everything else in `robots.txt` |
| **What to do?** | Set the token, keep your existing bot rules, check the Google catch, sort out Bing first if you use `NoArchive`/`NoCache` |

## What was actually agreed

The basis is the draft ["Machine-Readable Training Preference Signal"](https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/) by Kevin Kelley (Anthropic), dated 30 October 2026 and valid until 3 May 2027. The idea is strikingly simple: no new directive, no new syntax, just a defined name in the `User-Agent` line. The example from the draft:

```text
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```

This means: no training on the whole site, except `/press-kit/`. A disallowed path expresses a preference *against* training, an allowed path a preference *in favor*. Paths, wildcards and the longest-match rule work as always. Parsers that don't know the token treat the group like that of any other unknown bot, so they ignore it. For those parsers, nothing changes.

The four commitments vary in firmness:

| Provider | Timeline | What was actually said |
| --- | --- | --- |
| **Anthropic** | "over the coming months", no date | [Kelley's mail to the working group](https://mailarchive.ietf.org/arch/msg/ai-control/mddjatOxSR0h6GmLRBWlc839c7w/): implementation has started, but rolling it out across all services will take time, and the interpretation may change as standards evolve. I found no page of Anthropic's own on this |
| **OpenAI** | "in the coming months" | [OpenAI's announcement](https://openai.com/index/helping-websites-express-ai-training-preferences/): voluntary, details to follow, existing crawler controls stay in effect, the approach may change as the IETF talks evolve. It doesn't name the token |
| **Microsoft** | 1 February 2027 | [Bing blog](https://blogs.bing.com/search/2026/10/your-content-your-choice-clearer-preferences-for-training-and-grounding/): applied at crawl time from then on, effective only with the page's next recrawl |
| **Google** | early 2027 | [Google Search blog](https://developers.google.com/search/blog/2026/10/ai-training-control): aiming for 2027, without naming the token. [Crawler docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended): `__usage_ai-training`, early 2027 |

Coywolf writes that Anthropic is the only company using the token today. That goes one step too far for me: Kelley's mail says "started implementing" and expects something usable only "over the coming months", and [Anthropic's crawler page](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) doesn't mention the token yet. So don't rely on it being active anywhere yet.

I found no commitment from Meta, Apple, Amazon, Perplexity, Mistral, xAI, ByteDance or Common Crawl. The draft's acknowledgments do list people from Meta, Amazon Web Services and Common Crawl, but expressly for identification only, and that isn't a commitment. The current state is tracked in Vanessa Fox's [reference on blocking LLM crawlers](https://www.vanessafox.com/complete-reference-blocking-llm-crawlers/).

## What the token doesn't do

This is where it gets practical, because the naive reading fails on these five points.

**It doesn't control crawling.** The draft says so verbatim: "This policy token is not a directive for crawling behavior." A crawler may take the preference into account but doesn't have to stay away because of it. So your server load won't necessarily drop. If you want to limit fetching, you still need groups for individual bots or a block at server or CDN level.

**`User-Agent: *` doesn't count.** The wildcard group expressly must not be applied to the token. A `robots.txt` with `User-Agent: *` and `Disallow: /` therefore expresses no training preference at all in the sense of the draft. No group of its own, no signal.

**It isn't retroactive.** According to the draft, the `robots.txt` in effect when the content was retrieved is what counts. The token won't pull back what's already in training data.

**It only covers generative training.** Models for classification, ranking or scoring fall outside the definition. Feeding content into AI answers at runtime (grounding, RAG) is also something else.

**Private agreements beat it.** A licensing contract may override the preference. That's in the draft too.

## The Google catch: training and grounding are tied together

For me this is the most important point, especially from a GEO perspective. Google plans to treat an opt-out via the new token as if you had disallowed `Google-Extended`, unless you name `Google-Extended` explicitly. And according to [Google's crawler docs](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended), `Google-Extended` governs not only training of future Gemini models but also grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

The result: once Google implements this, this seemingly pure training opt-out also takes you out of grounding in Gemini Apps and on Vertex AI.

```text title="Training off — and at Google, Gemini grounding too"
User-Agent: __usage_ai-training
Disallow: /
```

Google names the way out itself: if you explicitly allow `Google-Extended`, you can appear in Gemini Apps answers and receive clicks via grounding. But then, by Google's definition, you also allow Gemini training, because both are tied to `Google-Extended`.

```text title="Training off in general, Google stays open — for both"
User-Agent: __usage_ai-training
Disallow: /

User-Agent: Google-Extended
Allow: /
```

So via `robots.txt`, Google makes you choose per path: both or neither. Google Search itself isn't affected, since according to the docs `Google-Extended` plays no part in inclusion in the index or as a ranking signal. You don't control AI Overviews or AI Mode with it either. For those, a [separate Search Console control](https://support.google.com/webmasters/answer/16908024?hl=en) has been available worldwide since 31 August 2026, and according to Google it has no effect on AI training in turn.

Microsoft separates things more cleanly. There, training and grounding are two independent settings, and the Bing blog says explicitly that neither touches classic Bing results.

## What Microsoft adds on top

Microsoft uses the occasion for its own finer-grained system that applies to Microsoft only. The training preference also exists at page level, where it overrides `robots.txt`. For grounding there is **no** `robots.txt` rule, only the page level:

```html title="Page level at Microsoft"
<meta name="Bingbot" content="ai-training:disallow">
<meta name="Bingbot" content="ai-grounding:disallow">
<meta name="Bingbot" content="ai-grounding:disallowextended">
```

The same works as a header, for instance `X-Robots-Tag: Bingbot: ai-training:disallow`. The value `disallowextended` allows grounding only in Microsoft's own applications such as Copilot and excludes grounding services Microsoft provides to third parties, such as Web IQ and the API in Foundry. An allow doesn't lift a `robots.txt` rule that forbids Bingbot from crawling.

Important if you use `NoArchive` or `NoCache` to limit AI use at Bing. These tags lose that meaning:

| Date | What happens |
| --- | --- |
| 15 December 2026 | Temporary grounding setting in Bing Webmaster Tools for sites using `NoArchive`/`NoCache` |
| 1 February 2027 | New preferences apply at crawl time. If Microsoft detects new preferences on a site, it disregards `NoArchive`/`NoCache` there |
| 1 March 2027 | Deadline for new preferences and the temporary grounding setting. After that, `NoArchive` and `NoCache` no longer count as a preference for training or grounding |
| 30 September 2027 | Temporary grounding settings expire, at the latest by then |

The 1 February row is the stumbling block: if you've been limiting grounding through `NoArchive` and now set the training token, you could lose that limit from then on, since "Allow" applies without a grounding preference. Microsoft doesn't say explicitly whether a `robots.txt` rule alone triggers this, but the wording suggests it does. In that case, set `ai-grounding` first.

## The token and the other signals

Several mechanisms now compete, some of them overlapping. An overview so you don't mix them up:

| Mechanism | Where | What | Status |
| --- | --- | --- | --- |
| `__usage_ai-training` | `robots.txt` group | Training of generative models | Individual draft, four commitments |
| `Content-Usage: train-ai=n` | HTTP header and its own `robots.txt` directive | Working group categories: `train-ai`, `ai-use`, `search` | [Working group draft](https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/), still open |
| `Content-Signal: ai-train=no` | `robots.txt` and HTTP header | `ai-train`, `search`, `ai-input` | From Cloudflare, see [Content Signals & C2PA](https://www.jpkc.com/db/en/blog/content-signals-c2pa/) |
| `GPTBot`, `ClaudeBot`, `Google-Extended` … | `robots.txt` group per provider | Crawling or use per provider | Still in force, the providers keep their controls |
| `ai-training`/`ai-grounding` at Bing | Meta tag, `X-Robots-Tag` | Training and grounding per page | Microsoft only, from February 2027 |
| [RSL](https://rslstandard.org/) | `License:` reference in `robots.txt` | License and payment terms | Different layer: license rather than preference |

About `Content-Signal`, which I use myself, an honest note: according to a July 2026 report by [Search Engine Journal](https://www.searchenginejournal.com/google-answers-question-about-llms-author-txt-for-seo/581547/), John Mueller wrote that, as far as known, no crawler or LLM uses this directive. So the token has something that, from what I could find, `Content-Signal` doesn't have from these four providers: public commitments. In return, `Content-Signal` is broader: it also covers real-time use, and `ai-train` isn't limited to generative models. I'm keeping both. They don't conflict as long as the values match.

## How solid is the ground?

The signal rests on two unfinished documents, and you should know that before treating it as settled.

First, Kelley's text is an individual draft that the working group hasn't adopted. Kelley's own mail calls it input to the group's work on attaching preferences to content. Second, it does say the token corresponds to the definition the AIPREF working group has arrived at. But the underlying [vocabulary draft, version 08](https://www.ietf.org/archive/id/draft-ietf-aipref-vocab-08.txt), states in capital letters that its contents don't reflect consensus of the group, and marks the very section with the definitions as not yet having consensus. The [minutes of the 21 October interim meeting](https://datatracker.ietf.org/doc/minutes-interim-2026-aipref-15-202610212000/) quote the chair: "We haven't declared consensus on anything yet." Kelley's draft only softly asks implementers to update their definitions ("SHOULD consider"). The meaning can still shift.

There is criticism too. According to [Search Engine Roundtable](https://www.seroundtable.com/google-ai-training-control-42430.html), Nate Hake, also named in the acknowledgments, points out that the IETF process isn't finished and that significant disputes remain, and criticizes that the preferences don't apply to actions users trigger themselves. By my reading, that concerns the "AI Use" category, not the training definition the token builds on. An open question in the vocabulary points the same way: what "directly provided by the user" means, including whether a URL handed over counts, is unresolved.

I still think it's right that the big providers are moving ahead of completion. A standard without implementations doesn't help you, and the syntax is so close to RFC 9309 that later corrections should do little damage. It just means: keep watching, don't tick it off.

## And legally?

This isn't legal advice, but three points belong in your assessment.

Google writes that the proposal is consistent with the EU Code of Practice for general-purpose AI models. The [Code's copyright chapter](https://ec.europa.eu/newsroom/dae/redirection/document/118115) does commit signatories to follow the Robot Exclusion Protocol per RFC 9309, including later versions whose feasibility the IETF demonstrates. On top of that come other protocols that standards bodies have adopted or that are state of the art, widely adopted and generally agreed. Whether a token from an individual draft falls under that remains to be seen.

In Germany, the rights reservation under [section 44b(3) of the Copyright Act](https://www.gesetze-im-internet.de/englisch_urhg/englisch_urhg.html) depends on machine-readability for works accessible online. On 10 December 2025, the [Hanseatic Higher Regional Court](https://justiz.hamburg.de/resource/blob/1126574/2495057737fed4883eec42dce4d2430b/ki-und-urheberrecht-hanseatisches-oberlandesgericht-weist-berufung-zurueck-data.pdf) (press release in German, case no. 5 U 104/24) rejected a reservation worded in natural language on a photo agency's website: the plaintiff hadn't shown it was machine-readable at the time of the 2021 download. The court also based the use on the research exception in section 60d. In the [judgment](https://www.itm.nrw/wp-content/uploads/2025/12/5-u-104-24.pdf) (in German), it calls the required form technology-neutral and mentions `robots.txt` only as an option discussed in the literature. The ruling isn't final: the [Federal Court of Justice heard the case on 3 September 2026 and plans to rule on 17 December 2026](https://www.bundesgerichtshof.de/SharedDocs/Termine/DE/Termine/IZR281-25.html) (in German, I ZR 281/25). So what exactly suffices is open, and 17 December is the date to keep in mind for this topic.

The EU Commission [consulted on machine-readable opt-out protocols until January 2026](https://digital-strategy.ec.europa.eu/en/consultations/commission-launches-consultation-protocols-reserving-rights-text-and-data-mining-under-ai-act-and), [held a workshop](https://www.cepic.org/post/european-commission-publishes-summary-report-following-tdm-opt-out-protocol-workshop) on them in June and intends to publish a list of generally agreed solutions. I haven't been able to find such a list as of today.

My conclusion: if you really want to reserve your rights, don't rely on this one token. Use several routes in parallel: the token, the provider-specific rules and a legally worded declaration reviewed by a professional.

## What I recommend

**You want to allow training** (my case). A small group is enough. It isn't technically necessary, since Microsoft for one treats "Allow" as the default anyway. But it makes your position unambiguous:

```text
User-Agent: __usage_ai-training
Allow: /
```

**You want to refuse training.** Set the token with `Disallow: /` and **keep your bot rules**. Until rollout, `GPTBot`, `ClaudeBot`, `Google-Extended` and friends are what actually applies, and they're the only answer for every provider without a commitment. Decide consciously about the Google catch and, if you use `NoArchive`, about grounding at Bing.

**You only want to exclude some areas.** Watch out for a subtlety: `Disallow: /premium/` alone isn't neutral. In an existing group, any path not disallowed counts as allowed, so by my reading of the draft and RFC 9309 you express a preference *in favor of* training for everything else. If you don't want that, disallow broadly and allow selectively, as in the `press-kit` example.

Plus four things to do:

1. **A separate group, cleanly apart.** Consecutive `User-Agent` lines share their rules, and under RFC 9309 even a blank line doesn't separate them. Some parsers, such as Python's `urllib.robotparser`, see that differently. If the token sits without rules of its own directly above `User-Agent: GPTBot`, the GPTBot group therefore applies to the token as well. So always give it rules of its own. You can do that on purpose too: the draft allows mixing tokens and bots in one group. With `Disallow: /`, you then block the bot and express the preference at the same time.
2. **Every subdomain separately.** A `robots.txt` applies per host, as always. Nobody evaluates a file in a subfolder such as `/blog/robots.txt`.
3. **Align your signals.** `Content-Signal: ai-train=yes` next to `__usage_ai-training` with `Disallow: /` is a contradiction no machine will resolve for you.
4. **Check.** My [SEO & GEO Analyzer](https://www.jpkc.com/db/en/tools/seo/) shows the token as its own card in the *GEO Score* tab, with the preference for the analyzed URL and a warning if it contradicts a site-wide `Content-Signal`. It earns no points. The *Check robots.txt* tab in [robots.txt & Sitemap](https://www.jpkc.com/db/en/tools/robots-sitemap/) reads it too. For a quick look, `curl` is enough:

```bash
curl -sL https://www.example.com/robots.txt | grep -iE -A5 '^[[:space:]]*user-agent:[[:space:]]*__usage_ai-training'
```

## What I did on jpkc.com

This knowledge base is expressly built for AI access, and `Content-Signal: ai-train=yes` already sits in `robots.txt` and in the headers. So the group with `Allow: /` now sits there as well. I don't set `Crawl-delay` on it, because the token isn't a crawler and fetches nothing. The `Host` line and all existing bot groups stay as they are, and `Google-Extended` is already explicitly set to `Allow` in my file. While I was at it, I added `Claude-User` and `Claude-SearchBot`, which used to fall under `*` and now have a group of their own like the other AI bots.

Before deploying, I compared the old and new versions: with the parser [from my tools](https://www.jpkc.com/db/en/tools/robots-sitemap/), which reads groups per RFC 9309, and with the one from Python's standard library, which doesn't and stands in for older parsers here. I tested the 69 user agents already named (including the joke entries `Bender` and `Gort`) plus one unknown agent against ten paths. Differences in allow/disallow and `Crawl-delay`: none. That's to be expected, since no bot carries the token and the two new Claude groups apply only to themselves. So the test only shows that the new groups leave their neighbors unchanged, but with a file your entire visibility depends on, that's exactly what I wanted to measure.

## FAQ

### Does `__usage_ai-training` stop AI training immediately?

No. Three of the four providers have only announced support, and Anthropic has started implementing it. On top of that, it's a preference, and bots without a commitment or with bad intentions don't care. For a hard block, you need rules at server or CDN level.

### Can I delete my GPTBot and ClaudeBot rules now?

No. Google, OpenAI and Anthropic explicitly keep their own controls, and those partly govern something else: a `Disallow` for `GPTBot` affects fetching, the new token doesn't. Besides, the shared rule only covers the providers that have committed.

### Does a training opt-out hurt my Google ranking?

According to Google's docs, no. `Google-Extended` affects neither inclusion in Search nor ranking. But once Google implements this, you lose grounding in Gemini Apps and on Vertex AI there unless you explicitly allow `Google-Extended`. That's the price many are likely to overlook.

### Do I stay visible in ChatGPT, Claude and Copilot if I block training?

According to the draft, the token is defined for training only. OpenAI and Anthropic run separate bots for search and user fetches. You control search separately there, but for `ChatGPT-User`, OpenAI warns that `robots.txt` rules may not apply to user fetches. Microsoft explicitly separates training and grounding. Only at Google does the token reach grounding, through `Google-Extended`.

### Is this a legally sound rights reservation?

Nobody can promise you that today. Case law on machine-readability isn't settled, and Kelley's draft says nothing about legal effect. For binding questions, you need a professional.

### What does the token do for me if I allow training?

Mainly clarity. Without a group you've expressed no preference in the sense of the draft; with `Allow: /` you've expressed an unambiguous one. For a knowledge base like this one, that fits the purpose, and the two lines cost nothing.

## Further reading

The signal's older sibling, including the `use` field and path rules, is covered in [Content Signals & C2PA](https://www.jpkc.com/db/en/blog/content-signals-c2pa/). How a request becomes a block at network level is shown in [15 September 2026: Cloudflare Changes the Crawler Defaults](https://www.jpkc.com/db/en/blog/cloudflare-crawler-defaults/). The rest of crawler management is in [Structured Data & Technical GEO](https://www.jpkc.com/db/en/blog/structured-data-technical-geo/), and [What is GEO?](https://www.jpkc.com/db/en/blog/was-ist-geo/) provides the framework.

