Robots.txt AI Training Opt-Out Gets One Shared Token

A small text file from the early days of the web is back in the news. Robots.txt, the plain note that sits at the root of almost every website, now carries the weight of one of the biggest arguments in tech: who gets to train AI on your writing. And on October 30, 2026, the robots.txt AI training opt out got something it never had before, a single shared way to say no.

That day, Google, Microsoft, OpenAI and Anthropic each said they would support one common policy token. Until now, if you wanted to opt out, you had to learn the name of every company's crawler and write a rule for each. That is a bit like putting a different lock on the same door for every neighbour. The new idea is one lock, and it lands in the same month that OpenAI stopped training on the live internet, so the rules about what AI may read are clearly being written right now.

It is a good idea, but much less finished than the headlines made it sound. Here is what was announced, how the rule works, and what a site owner can sensibly do today.

What was announced on October 30, 2026

The word "announced" covers four slightly different stories.

Google wrote on its Search Central blog that it plans to support a shared AI training control in robots.txt. John Mueller, who wrote the post, said Google aims to support it, along with other AI providers, in 2027. Google's crawler documentation reportedly puts it more precisely as early 2027.

Microsoft gave the most concrete date. It said it will start honoring the new training and grounding preferences on February 1, 2027, as its crawler revisits each site. Its own warning: neither choice guarantees inclusion, traffic or payment.

OpenAI gave no date. It said it aims to implement the signal "in the coming months," called the effort voluntary, and said the signal will not replace the controls it already offers.

Anthropic went furthest. Kevin Kelley, who wrote the draft, told the IETF working group that Anthropic has already started implementing support, but it will take time to cover all its services.

The honest summary: four big names agree on the idea, one says it has started, and the rest sit somewhere between February 2027 and "soon." A headline that says the shared opt-out "works now" is wrong, at least for now. Dates like these are worth tracking, much as the Deno Deploy shutdown date was for the people building on that platform.

How the robots.txt AI training opt out works

The token is called __usage_ai-training. It comes from an Internet-Draft titled "Machine-Readable Training Preference Signal," written by Kelley and dated October 30, 2026. It is a draft, not a finished standard, and reports describe it as an individual submission without formal standing at the IETF.

The clever part is how little it changes. There is no new command to learn. You write an ordinary robots.txt group, but where the crawler's name normally goes, you put the token. Then you use the same Disallow and Allow lines you already know. The draft's own example looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Read that as: please do not train AI on this site, except the press kit. Disallow states a preference against training for a path; Allow states a preference in favour.

Three small rules catch people out:

• A wildcard does not count. A group that starts with User-agent: * is not read as speaking for the token. If no group names it, the draft says no preference has been expressed. So a file that already blocks "all bots" says nothing about AI training.
• It is not a crawl block. The token states what you want done with your content, not who may visit. To stop a crawler from fetching pages, you still write a rule for that crawler.
• Specific beats general, and a private licensing deal can override the preference.

So think of a robots.txt line as a polite, public sign on the gate, not a steel door.

Google-Extended robots.txt rules and the new token

Google has offered its own opt-out since 2023, called Google-Extended. This is where the new token touches existing files, so the Google-Extended robots.txt rules deserve a careful look.

Google says that if a site opts out through the shared token and does not name Google-Extended anywhere, Google plans to treat the file as if the site had disallowed Google-Extended. In other words, one line can quietly do two jobs.

Two things to keep straight. First, Google-Extended does not decide whether your pages appear or rank in Google Search. Second, Google's documentation, as reported, says the choice also touches grounding in Gemini Apps and Vertex AI (we covered the Gemini API price increase separately). Grounding is the step where an AI answer pulls fresh information from a page; how AI actually works covers the basics behind it. So opting out of training can also mean opting out of being a source in some Gemini answers. Many site owners do not expect that.

If you want "no training, yes to being a source," name Google-Extended in its own group and say so. Make that decision on purpose, not by accident.

The token is not a lock

Let's be blunt. Robots.txt has always been a request. It works because well-behaved crawlers choose to obey it, and nothing in the file physically stops anyone.

Microsoft says its settings "govern use of content, not crawl access," so a rule that blocks its crawler still has to be written separately. And the obedience is not perfect even now. PPC Land cited data from TollBit, a company that tracks AI bots, finding that about 15% of AI page fetchers in Europe reached URLs that were marked off-limits in August 2026.

The token also looks forward, not back. Reports say that changing your robots.txt does not remove anything already collected into existing datasets or trained models.

Anyone who wants enforcement, not just a request, has to block bots at the network level. Cloudflare sells tools for that, and it also built its own robots.txt extension called Content Signals, which lets a site mark "search," "ai-input" and "ai-train" separately. It is the same company whose deal with Deno we covered when asking is Deno dead, and a reminder of how much of the web's plumbing now sits with a few firms. If you want to see how far the control question reaches beyond websites, our piece on whether AI has a kill switch is the other end of the same argument.

Does robots.txt stop AI training?

Only for companies that choose to respect it, and only for what they collect from now on.

Today you can already opt out of specific training crawlers by name, such as OpenAI's GPTBot and Google's Google-Extended. The new token adds one shared line on top, but most companies have not switched it on yet. And it all depends on goodwill.

One more trap: training bots and search bots are different things. OpenAI, for example, separates the crawler that gathers training data from the one that shows your pages in ChatGPT search answers. Copy a "block every AI bot" snippet from a forum and you can remove yourself from the tools that send you readers. Check each bot's name first. The same split applies to AI agents: Meta Muse can only touch what you connect, which is a different question from what a crawler collected.

What a site owner can sensibly do now

You do not need to rush, but you do need to decide. A calm order of work:

1. Decide what you want. Quoted everywhere, out of training but still a source, or out of everything. Write it down first.
2. Open yoursite.com/robots.txt and read it. Old copied rules are common, and a blanket block can hide inside them.
3. Name the training crawlers you care about, and leave search crawlers alone unless you mean to block them.
4. Add the shared token group if you wish. Software that does not know the token ignores it, per the draft, so adding it early costs nothing.
5. Say what you mean for Google. If you want to stay a source in Gemini answers, name Google-Extended on purpose rather than letting the token decide for you.
6. Check again in 2027, against each company's own documentation.

If you build with AI coding tools, a tidy rules file for your assistant, like the one our CLAUDE.md generator produces, is the same kind of thing: plain text that tells a machine how to behave around your work. And if that machine can reach your whole computer, the questions in macOS Full Disk Access for AI agents come next.

What it means for everyone else

Nothing changes for most readers this week, but the direction matters. For years the choice was all or nothing: let AI read everything, or block everything and vanish from the new ways people search, the uses of AI beyond chatbots included. A path-level preference is a step toward something in between, where you can say "my guides are fine, my paid research is not."

It also shows how much is still being made up as it goes. The companies that want to train on the web are helping write the rules for opting out. I read that as both progress and a conflict of interest, and I would watch the dates more than the press releases.

The pattern is familiar. Providers decide prices, limits and permissions and announce them with a date. Defaults are theirs; settings are yours, if you look. WhatsApp's new profile cover photo arrives with "Everyone" as the default.

The bottom line

The robots.txt AI training opt out now has a shared token, __usage_ai-training, backed in principle by Google, Microsoft, OpenAI and Anthropic. It is simple, it reuses a file you already have, and it can say no to training for some paths and yes for others. It is also a draft, it is voluntary, it does not touch what has already been collected, and for most companies it does not work yet.

My advice is plain. Decide what you want, check your file, add the line if it suits you, and look again in early 2027 when the first company dates arrive.

FAQs

What is the __usage_ai-training token?

It is a policy token from an Internet-Draft written by Anthropic's Kevin Kelley and dated October 30, 2026. You put it on the User-Agent line of a robots.txt group, and the Disallow and Allow lines under it say whether you prefer AI training on those paths. Software that does not understand it ignores it.

Does robots.txt stop AI training?

Only for companies that choose to honor it, and only for content collected from now on. It does not remove data already gathered, and it does not physically block anyone. It works as a public statement of your preference.

How do I opt out of AI training with robots.txt?

Add a group that names the token, with a Disallow line for the paths you want to protect. For the whole site, that is User-Agent: __usage_ai-training followed by Disallow: /. You can also block specific training crawlers by name. Keep search crawlers allowed if you want traffic from them.

How do I block AI crawlers in robots.txt?

For the new shared token, add a group that names __usage_ai-training with Disallow: /. You can also add a group for each training crawler by name. Remember that robots.txt is a request, so only companies that choose to honor it will follow it.

Should I block AI crawlers?

That depends on how you feel about your writing being used to train models. Blocking training does not have to cost you search traffic, because search crawlers can stay allowed. If you want to be cited in AI answers, think twice before blocking everything, since reports say a training opt out may also remove you as a source in some Gemini answers.

Does the shared AI training opt out hurt my Google rankings?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. Be careful, though: reports say it can affect grounding in Gemini Apps, so a training opt out may also remove you as a source in some Gemini answers.

What does Google-Extended do in robots.txt?

It is Google's own control, available since 2023, for how your content is used by Google's AI models. Under Google's plan, a shared-token opt out will be treated as a Google-Extended disallow when you have not named Google-Extended yourself.

When will the shared AI training opt out start working?

It depends on the company. Microsoft said it will start honoring it from February 1, 2027. Google aims for 2027, with documentation pointing to early 2027. OpenAI says "the coming months." Anthropic says it has started implementing support but that it will take time.