OpenAI, Google, Microsoft and Anthropic back one robots.txt opt-out for AI training
Publishers could set a single AI-training preference rather than configure each company, though past scraping stays out of reach.
OpenAI, Google, Microsoft and Anthropic have all committed to honouring a single shared directive in robots.txt that lets publishers state whether their content may be used for AI training, rather than configuring each AI company separately. The proposal, drafted by Anthropic standards lead Kevin Kelley, is under consideration at the Internet Engineering Task Force. Publishers add a dedicated AI-training user agent and can carve out exceptions for particular paths, such as a press kit. The directive does not affect search indexing and cannot reverse scraping that has already happened. Anthropic already supports it, Microsoft plans to start on February 1, 2027, Google expects to support it in early 2027, and OpenAI says it will implement it in the coming months. Meta's communications head has voiced support for a common signal but has not said whether Meta will honour it.
Why it matters
For publishers, the change means one shared signal in robots.txt can state whether their content may be used for AI training, with exceptions such as a press kit path. It does not stop search visibility or reverse scraping already done. Whether the approach works depends on the rollout: Anthropic already supports it, Microsoft plans to start in February 2027, Google expects early 2027, and OpenAI says it will follow in the coming months. Meta has supported a common signal but has not said whether it will honour one, so coverage across the largest AI firms is not yet complete.
Matters or hype?
Does this story matter, or is it hype? Vote before you see what everyone else said.