One robots.txt rule to say no to AI training

Four AI providers have committed to reading the same robots.txt instruction to learn whether a website accepts its content being used to train generative AI models. Google announced it on its Search Central blog, OpenAI on its website, Microsoft on the Bing blog and Anthropic on the IETF working group's mailing list.

The technical basis is an individual IETF draft, "Machine-Readable Training Preference Signal", published on 30 October 2026 by Kevin Kelley of Anthropic. It takes its definition of AI training from the vocabulary of the AI Preferences (AIPREF) working group: using content to modify the learned parameters of a generative model.

It helps to keep this in proportion: an individual draft is not yet an approved standard, and each company has given its own timeline. What changes now is the direction: for the first time, one line works for several providers at once.

The signal reuses ordinary robots.txt syntax, with no new directives. Instead of a crawler name, the User-Agent line carries a "policy token", __usage_ai-training, and the Disallow and Allow rules state on which paths you accept or refuse training.

Three details prevent surprises:

• The wildcard does not count. User-Agent: * rules do not apply to this token, so your current robots.txt expresses no training preference even if it blocks paths.
• It does not block crawling. It is a preference about how content is used; what each bot may fetch is still decided by its own rules. The draft does ask crawlers not to fetch what they will not be able to use.
• It allows exceptions by path. You can refuse training across the site and accept it, say, in your press kit folder, with the most specific rule winning.

None of the four applies it fully yet, and the timelines differ. As of 3 November 2026, this is what each has said:

• Google: plans to support it in early 2027. If you use the token to refuse training and do not mention Google-Extended, Google will treat it as if you had blocked Google-Extended, which its documentation says does not affect inclusion or ranking in Search.
• OpenAI: will honour it and aims to implement it in the coming months. Its existing per-crawler controls remain in place.
• Anthropic: has already started implementing it and warns that extending it to all its services will take time.
• Microsoft Bing: will read it in robots.txt from 1 February 2027 and adds page-level preferences for training and for use in answers (grounding). From 1 March 2027 it will no longer read the NoArchive and NoCache tags as a training or grounding preference.

Refusing training does not have to take you out of AI answers, and confusing the two can cost you visibility. Training means using your content to build a model; citing you in an answer (grounding) means consulting it at the moment a specific question is answered.

Providers increasingly keep them apart. Bing, for example, offers one preference for training and another for grounding, and warns that blocking grounding reduces brand visibility in its AI experiences. In ChatGPT, the search crawler and the training crawler were already separate, as we explain in what GEO is.

For many small businesses, the sensible combination is to let answer engines read and cite their site and to decide calmly about training. For those who live off original content (publishers, training providers, proprietary research), refusing training may make more sense.

You do not need to change anything today to keep appearing in Google, Bing or ChatGPT: the signal is voluntary and, if you do not publish it, you express no preference. The useful step is to prepare the decision for when providers apply it in 2027:

• Decide your position on training and write it down, separately from visibility in AI answers.
• Review your robots.txt, CDN and firewall so you do not accidentally block the search crawlers you do want. It is part of technical SEO.
• If you use NoArchive or NoCache, check what will happen with Bing from March 2027.
• Measure what you gain or lose in AI answers before and after any change, as we set out in our GEO service.