Cloudflare will filter out web crawlers that serve AI companies
Key takeaways
- The hosting platform wants sites to have more control over how AI companies use their content.
- Samuel Boivin/Shutterstock Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time.
- "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Matthew Prince, Cloudflare's CEO and co-founder shared in a statement.
The hosting platform wants sites to have more control over how AI companies use their content.
Samuel Boivin/Shutterstock Cloudflare has announced plans to automatically block mixed-use web crawlers that index websites for search engines and act as AI agents and trainers at the same time. The company previously offered its customers the optional ability to prevent crawlers from scraping their sites for AI chatbots, but now Cloudflare's stance is becoming more defensive by default.
"Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge," Matthew Prince, Cloudflare's CEO and co-founder shared in a statement. "Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training."