Cloudflare Blocks AI Bots by Default, Launches

Cloudflare, a leading internet infrastructure provider, has made a significant move to reshape how AI crawlers interact with web content. As of July 1, 2025, Cloudflare now blocks AI crawlers by default for all new domains that sign up for their services. This means website owners will have explicit control over whether AI companies can access their content for purposes like model training, inference, or search.

This fundamental shift empowers content creators and publishers, who have often seen their intellectual property scraped by AI models without permission or compensation. Previously, Cloudflare offered an option to block AI crawlers, which over a million customers had opted into. Now, the default is to block, and users must actively choose to allow AI crawlers if they wish.

Beyond the default blocking, Cloudflare has launched a new initiative called “Pay Per Crawl.” This program, currently in private beta, allows publishers to set fees for AI companies to access their content. The idea is to create a new economic model where original content is valued and compensated, moving away from the “content-for-traffic” model that has traditionally driven the web. Cloudflare will act as the merchant of record for these transactions, providing the technical infrastructure to facilitate payments.

Key aspects of this change include:

Permission-Based Model: AI companies will now need explicit permission from website owners to scrape their content.

Granular Control: Website owners can define whether they want to allow, charge, or completely block specific AI crawlers, and even specify the purpose of the crawl (training, inference, or search).

Verified Bots Program: Cloudflare is working on ways for AI bots to authenticate themselves, ensuring transparency and allowing website owners to identify legitimate crawlers.

Managed Robots.txt: Cloudflare can now manage robots.txt files on behalf of customers, automatically including directives to prevent popular AI bot operators from using content for training, while maintaining SEO friendliness.

AI Labyrinth: Cloudflare has also developed a “honeypot” system called “AI Labyrinth,” which traps misbehaving bots in an endless loop of decoy pages, preventing them from scraping legitimate content.