Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Key takeaways
- Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training.
- That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise.
- The move could impact how AI model providers are able to access web content for training purposes and to help power their agentic services.
Why this matters: a development in AI with implications for how people work, create, and decide.
Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training. Starting on September 15, 2026, Cloudflare s default settings will block mixed-use crawlers from any pages that host ads, the company announced on Wednesday.
That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise. These changes to the defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says.
The move could impact how AI model providers are able to access web content for training purposes and to help power their agentic services.