Launches Pay Per Crawl 

a significant move that could redefine the economics of the internet in the age of AI, Cloudflare, a leading connectivity cloud company, has announced that it will now block AI crawlers by default for all newly onboarded websites and is launching a “Pay per Crawl” initiative. This shift empowers website owners and content creators with greater control over their intellectual property and opens up new avenues for monetization.

Here’s a breakdown of this major development:

1. Default Blocking of AI Bots:

Permission-Based Model: As of July 1, 2025, Cloudflare has transitioned to a permission-based model for AI crawlers. This means that, by default, AI bots will be blocked from accessing content on new websites using Cloudflare’s services unless explicit permission is granted by the website owner.

Addressing Uncompensated Scraping: This change is a direct response to the growing concern among publishers and content creators about AI models scraping their content for training and generating responses without compensation or driving referral traffic back to the original sources.

Identification and Intent: Cloudflare’s technology leverages its advanced bot detection infrastructure to identify AI crawlers and understand their purpose (e.g., AI training, inference, or chatbot search responses). This allows website owners to make informed decisions about which crawlers to allow.

Leading Publishers in Support: Numerous major content, media, and technology companies, including ADWEEK, The Associated Press, The Atlantic, Condé Nast, TIME, and Universal Music Group, have publicly supported this move, highlighting the need for a more sustainable future that values original content.

2. Launch of “Pay per Crawl”:

Monetization Mechanism: Alongside the default blocking, Cloudflare has launched “Pay per Crawl,” an experimental marketplace (currently in private beta) that enables website owners and media publishers to charge AI bots for accessing their content.

Publisher Control: Publishers can set custom pricing for each individual bot crawl, depending on the use case (e.g., training large language models, powering real-time search results, or assisting independent bots). Cloudflare acts as an intermediary, managing the payment processing and revenue distribution.

Transparency Tools: The system also includes transparency tools, such as dashboards that show how often bots visit a site and what data they are collecting, allowing publishers to differentiate between beneficial crawlers (like Google Search) and AI bots that may be extracting content without driving traffic.

Addressing Traffic Imbalance: Cloudflare has released data highlighting the significant imbalance between traffic generated and data consumed by AI crawlers. For example, OpenAI’s crawlers reportedly make 1,700 requests per referral, and Anthropic’s reach a staggering 73,000-to-1, compared to Google’s more balanced 14-1 crawl-to-click ratio. This initiative aims to rectify this imbalance.

New Revenue Stream: For content creators and smaller publishers, “Pay per Crawl” offers a potential new revenue stream, providing a direct economic link between content usage and compensation, rather than solely relying on ad revenue driven by traffic.

Implications and the Future of the Web:

Reshaping the AI Data Economy: Cloudflare’s move positions it as a significant gatekeeper and broker in the emerging AI data economy. Given that Cloudflare manages traffic for approximately 20% of the web, this change has the potential to fundamentally alter how AI companies acquire data.

Pressure on AI Companies: AI companies will now be under increased pressure to either pay for access to high-quality content or develop more sophisticated methods of content acquisition that respect publishers’ rights.