Jemiboo Magazine

Block AI training, Nov 2026: the shared robots.txt token

Google, Microsoft and OpenAI said on October 30, 2026 they will support a shared robots.txt token that lets websites opt out of AI training in one place.

Jemiboo Magazine

· 5 min read

Read in other languages한국어Español简体中文繁體中文日本語

Key takeaways (as of November 10, 2026)

  • The shared token “__usage_ai-training” comes from an individual draft that Anthropic’s Kevin Kelley submitted to the IETF on October 30, 2026; it is not yet an IETF-approved standard.
  • Google aims to support the token in 2027, while Microsoft will begin applying the new settings on February 1, 2027 and will stop reading the old NoArchive and NoCache directives on March 1, 2027.
A small crawler robot stopped in front of a server whose switch is turned off

Two lines in robots.txt may soon be enough to opt a website out of AI training by several companies at once. On October 30, Google, Microsoft and OpenAI said they would support a shared “policy token” written into robots.txt. The AI Preferences working group of the Internet Engineering Task Force (IETF) meets next week, on November 16 and 19. An opt-out that site owners used to spell out company by company is turning into common web syntax.

Blocking AI training today means a different name for every company

robots.txt is a short text file placed at the root of a website. It is a long-standing convention that tells search crawlers which parts of a site they may read. AI training was the problem. Since 2023 Google has accepted training opt-outs under a separate name, Google-Extended, and other companies each use their own. Every time a new AI company appeared, site operators had to update their list again. Writing on the Google Search Central Blog on October 30, Google Search Advocate John Mueller said that instead of managing separate instructions for each AI developer, website owners will be able to set broad preferences for all participating AI companies through a shared “policy token” in robots.txt.

CompanyCurrent methodResponse to shared tokenTiming
GoogleGoogle-Extended (since 2023)An opt-out via the token is read as a Google-Extended opt-outSupport targeted for 2027
Microsoft (Bing)NoArchive · NoCacheReplaced by training and grounding settingsApplied from February 1, 2027
OpenAIGPTBot (reported)Supports the shared tokenWithin months (reported)

The table is based on Google’s and Microsoft’s official blog posts of October 30, 2026. OpenAI’s original announcement could not be accessed; the OpenAI row reflects a November 2 report by the Japanese IT publisher Gijutsu-Hyoron (gihyo.jp). According to that report, OpenAI will keep letting sites configure its training crawler, GPTBot, and its search crawler, OAI-SearchBot, separately as they do now.

Name the token instead of using the asterisk (*)

The syntax is short. Writing 'Disallow: /' under 'User-Agent: __usage_ai-training' means the whole site is excluded from training. Specific folders can also be left open. The asterisk is the catch. The draft states that the 'User-Agent: *' group, which addresses all crawlers, must not be interpreted as applying to this token. If no line names the token, the site is treated as having expressed no preference. The token is also not a command to stop crawling. The draft explains that a crawler reading a page and the uses made of what it reads are handled separately. Where an individual agreement exists, that agreement may take precedence.

CategoryLabelMeaning (IETF vocabulary draft, version 08)
AI trainingtrain-aiUse to change the trained parameters of a generative AI model
AI useai-useUse of material not directly provided by the user as input to generative AI
SearchsearchUse that directs users to the original location, excluding summary generation

The categories are taken from the IETF working group document “A Vocabulary For Expressing AI Usage Preferences”, version dated September 14, 2026. That document also notes that it is a draft and does not yet reflect working group consensus.

A hand on a laptop whose screen shows lines resembling a configuration file

Sites can stay in search while opting out of training

What worries operators most is search visibility. Microsoft said the new settings do not change a site’s eligibility to appear in Bing search results. Bing has gone a step further, letting sites choose training and “grounding” separately for each page. Grounding is when an AI pulls in web documents as evidence while writing an answer. Training can be set per site or path through robots.txt, and through page meta tags where needed. Grounding is set only through page meta tags; robots.txt has no grounding setting. Google said sites that explicitly allow Google-Extended can appear as sources in answers in the Gemini app and receive clicks. Whether to block training or remain a cited source in AI answers is now a trade-off for each operator to weigh.

DateMicrosoft Bing schedule
December 15, 2026Temporary grounding setting opened to sites using NoArchive · NoCache
February 1, 2027New training and grounding settings applied during crawling
March 1, 2027NoArchive · NoCache no longer read as preference signals
September 30, 2027Temporary setting expires

What blogs and publishers can do now

Neither the announcements nor the IETF draft mention Korean search or AI services, and it has not been decided whether Korean search and AI companies will follow the same token. Operators running their own domain should first open 'yourdomain/robots.txt' to check what it currently says. Because the shared token will be applied in stages from 2027, sites that want to opt out of training right away are safer keeping the company-specific names in place as well. Our November 5 article on the expansion of AI text watermarks (https://jemiboo.com/en/tech/2026-11-05/) looked at marks left on text written by AI; this token is a “please don’t use this” mark attached to text written by people.

Frequently asked questions

Will opting out of AI training in robots.txt remove my site from search?

The shared token is a statement of preference about use, not a command to block crawling. Microsoft said the new settings do not change a site’s eligibility to appear in Bing search results.

When does the __usage_ai-training token take effect?

Google aims to support it in 2027, and Microsoft will begin applying it on February 1, 2027. OpenAI has been reported to plan support within months.

Does blocking with User-Agent: * also opt out of AI training?

No. The IETF draft specifies that the asterisk group must not be interpreted as applying to this token. To opt out of AI training, the name '__usage_ai-training' has to be written out separately.

The draft is still an individual submission from one person, and it can enter the standards track only if the working group adopts it. Once the minutes of next week’s San Francisco meeting are posted, the first thing to check is whether the token name survives unchanged.

Sources

  1. Google Search Central Blog, Supporting a shared AI training control for websites (Oct 30, 2026) — developers.google.com
  2. Microsoft Bing Blog, Your content, your choice: clearer preferences for training and grounding (Oct 30, 2026) — blogs.bing.com
  3. IETF, draft-kelley-aipref-training-preference-signal-00 (Oct 30, 2026) — datatracker.ietf.org
  4. IETF, A Vocabulary For Expressing AI Usage Preferences, draft-ietf-aipref-vocab-08 (Sep 14, 2026) — datatracker.ietf.org
  5. IETF, IETF 127 Agenda (updated Nov 9, 2026) — datatracker.ietf.org
  6. Gijutsu-Hyoron gihyo.jp, report on robots.txt AI training controls (Nov 2, 2026) — gihyo.jp

block AI trainingrobots.txtAI training opt-outIETF AIPREFGoogle-ExtendedBing grounding

More from the same day

More in Tech