The biggest names in AI are backing a new way to opt out of scraping

| Source: The Verge AI

Tags: robots.txt, IETF, OpenAI, Google, Microsoft, Anthropic, web scraping

OpenAI, Google, Microsoft, and Anthropic plan to adopt a shared robots.txt policy token that lets publishers opt out of AI training scraping across all of them at once.

Details

OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that lets publishers set AI training preferences once instead of per company. The shared 'policy token' was proposed at the IETF by Kevin Kelley, a technical lead for standards at Anthropic, and lives in a site's robots.txt file. The companies say their AI crawlers will stay away from sites that carry it. Microsoft will offer finer controls to separate opting out of training from opting out of grounding, and says sites will still appear in Bing search results either way. The move arrives as publishers push back on scraping through efforts like RSL licensing and Cloudflare's default blocking of AI crawlers.