Block AI training, Nov 2026: the shared robots.txt token
Google, Microsoft and OpenAI said on October 30, 2026 they will support a shared robots.txt token that lets websites opt out of AI training in one place.
· 5 min read
Read in other languages한국어Español简体中文繁體中文日本語
Key takeaways (as of November 10, 2026)
- The shared token “__usage_ai-training” comes from an individual draft that Anthropic’s Kevin Kelley submitted to the IETF on October 30, 2026; it is not yet an IETF-approved standard.
- Google aims to support the token in 2027, while Microsoft will begin applying the new settings on February 1, 2027 and will stop reading the old NoArchive and NoCache directives on March 1, 2027.

Two lines in robots.txt may soon be enough to opt a website out of AI training by several companies at once. On October 30, Google, Microsoft and OpenAI said they would support a shared “policy token” written into robots.txt. The AI Preferences working group of the Internet Engineering Task Force (IETF) meets next week, on November 16 and 19. An opt-out that site owners used to spell out company by company is turning into common web syntax.
Blocking AI training today means a different name for every company
robots.txt is a short text file placed at the root of a website. It is a long-standing convention that tells search crawlers which parts of a site they may read. AI training was the problem. Since 2023 Google has accepted training opt-outs under a separate name, Google-Extended, and other companies each use their own. Every time a new AI company appeared, site operators had to update their list again. Writing on the Google Search Central Blog on October 30, Google Search Advocate John Mueller said that instead of managing separate instructions for each AI developer, website owners will be able to set broad preferences for all participating AI companies through a shared “policy token” in robots.txt.
| Company | Current method | Response to shared token | Timing |
|---|---|---|---|
| Google-Extended (since 2023) | An opt-out via the token is read as a Google-Extended opt-out | Support targeted for 2027 | |
| Microsoft (Bing) | NoArchive · NoCache | Replaced by training and grounding settings | Applied from February 1, 2027 |
| OpenAI | GPTBot (reported) | Supports the shared token | Within months (reported) |
The table is based on Google’s and Microsoft’s official blog posts of October 30, 2026. OpenAI’s original announcement could not be accessed; the OpenAI row reflects a November 2 report by the Japanese IT publisher Gijutsu-Hyoron (gihyo.jp). According to that report, OpenAI will keep letting sites configure its training crawler, GPTBot, and its search crawler, OAI-SearchBot, separately as they do now.
Name the token instead of using the asterisk (*)
The syntax is short. Writing 'Disallow: /' under 'User-Agent: __usage_ai-training' means the whole site is excluded from training. Specific folders can also be left open. The asterisk is the catch. The draft states that the 'User-Agent: *' group, which addresses all crawlers, must not be interpreted as applying to this token. If no line names the token, the site is treated as having expressed no preference. The token is also not a command to stop crawling. The draft explains that a crawler reading a page and the uses made of what it reads are handled separately. Where an individual agreement exists, that agreement may take precedence.
| Category | Label | Meaning (IETF vocabulary draft, version 08) |
|---|---|---|
| AI training | train-ai | Use to change the trained parameters of a generative AI model |
| AI use | ai-use | Use of material not directly provided by the user as input to generative AI |
| Search | search | Use that directs users to the original location, excluding summary generation |
The categories are taken from the IETF working group document “A Vocabulary For Expressing AI Usage Preferences”, version dated September 14, 2026. That document also notes that it is a draft and does not yet reflect working group consensus.

Sites can stay in search while opting out of training
What worries operators most is search visibility. Microsoft said the new settings do not change a site’s eligibility to appear in Bing search results. Bing has gone a step further, letting sites choose training and “grounding” separately for each page. Grounding is when an AI pulls in web documents as evidence while writing an answer. Training can be set per site or path through robots.txt, and through page meta tags where needed. Grounding is set only through page meta tags; robots.txt has no grounding setting. Google said sites that explicitly allow Google-Extended can appear as sources in answers in the Gemini app and receive clicks. Whether to block training or remain a cited source in AI answers is now a trade-off for each operator to weigh.
| Date | Microsoft Bing schedule |
|---|---|
| December 15, 2026 | Temporary grounding setting opened to sites using NoArchive · NoCache |
| February 1, 2027 | New training and grounding settings applied during crawling |
| March 1, 2027 | NoArchive · NoCache no longer read as preference signals |
| September 30, 2027 | Temporary setting expires |
What blogs and publishers can do now
Neither the announcements nor the IETF draft mention Korean search or AI services, and it has not been decided whether Korean search and AI companies will follow the same token. Operators running their own domain should first open 'yourdomain/robots.txt' to check what it currently says. Because the shared token will be applied in stages from 2027, sites that want to opt out of training right away are safer keeping the company-specific names in place as well. Our November 5 article on the expansion of AI text watermarks (https://jemiboo.com/en/tech/2026-11-05/) looked at marks left on text written by AI; this token is a “please don’t use this” mark attached to text written by people.
Frequently asked questions
Will opting out of AI training in robots.txt remove my site from search?
The shared token is a statement of preference about use, not a command to block crawling. Microsoft said the new settings do not change a site’s eligibility to appear in Bing search results.
When does the __usage_ai-training token take effect?
Google aims to support it in 2027, and Microsoft will begin applying it on February 1, 2027. OpenAI has been reported to plan support within months.
Does blocking with User-Agent: * also opt out of AI training?
No. The IETF draft specifies that the asterisk group must not be interpreted as applying to this token. To opt out of AI training, the name '__usage_ai-training' has to be written out separately.
The draft is still an individual submission from one person, and it can enter the standards track only if the working group adopts it. Once the minutes of next week’s San Francisco meeting are posted, the first thing to check is whether the token name survives unchanged.
Sources
- Google Search Central Blog, Supporting a shared AI training control for websites (Oct 30, 2026) — developers.google.com
- Microsoft Bing Blog, Your content, your choice: clearer preferences for training and grounding (Oct 30, 2026) — blogs.bing.com
- IETF, draft-kelley-aipref-training-preference-signal-00 (Oct 30, 2026) — datatracker.ietf.org
- IETF, A Vocabulary For Expressing AI Usage Preferences, draft-ietf-aipref-vocab-08 (Sep 14, 2026) — datatracker.ietf.org
- IETF, IETF 127 Agenda (updated Nov 9, 2026) — datatracker.ietf.org
- Gijutsu-Hyoron gihyo.jp, report on robots.txt AI training controls (Nov 2, 2026) — gihyo.jp




