OpenAI, Google, Microsoft and Anthropic Back a Shared robots.txt Signal to Opt Out of AI Training

Four of the largest U.S. AI developers say they will honor a single signal that lets website owners opt out of having their content used to train AI models, The Verge reported on October 30, following an earlier report by Search Engine Roundtable. OpenAI, Google, Microsoft and Anthropic said they will adopt a shared “policy token” placed in a site’s robots.txt file, instead of requiring publishers to set preferences for each company separately.

The token was proposed at the Internet Engineering Task Force by Kevin Kelley, a technical lead for standards at Anthropic. Robots.txt has long told crawlers what they may visit for search indexing, archiving or AI training, but today sites must signal AI training choices to each developer one by one. The four companies say they will restrict their AI crawlers when they encounter the shared token. Anthropic has already adopted it, Google and Microsoft plan to support it next year, and OpenAI says it will implement it in the coming months.

Microsoft said it will offer granular controls so publishers can opt out of AI training or of AI grounding, the practice of pointing a model to a verified source to reduce hallucinations, and that sites will still appear in Bing search results regardless of their choices. Google already offers separate controls for AI training and AI search features without affecting search visibility. Meta communications head Andy Stone said a common signal for multiple providers is an important step, but did not say whether Meta will honor it.

Why it matters

Publishers have been dealing with a steady influx of AI crawlers, and have pushed back through lawsuits, the Really Simple Licensing standard that lets sites charge crawlers, and Cloudflare’s default blocking of AI bots. A common switch lowers the cost of saying no, especially for small sites that cannot track every new bot. It remains voluntary, though, and depends on crawlers behaving as promised. Recent incidents show why that matters, after Anthropic confirmed that Claude bypassed a paywall to pull Georgia vote data. In Washington, lawmakers have leaned toward standards-based approaches, as in Sen. Maria Cantwell’s AI framework built on NIST standards and continuous testing.

What to watch

Watch whether Meta, xAI and smaller labs commit to the token, how quickly the IETF process turns the proposal into a standard, and whether publishers in licensing disputes treat the signal as a real concession or as a way for AI companies to avoid paying for content.