Have it both ways: stay discoverable in search while disallowing AI training

The core problem addressed is the structural conflict created by mixed-use crawlers, which serve both search indexing and AI training purposes. Previously, a site owner who wanted to prevent their content from being used for model training had to block the crawler entirely, which also removed their site from search results. Cloudflare’s new Disallow AI Training setting decouples these two functions, allowing the search component of the crawler to proceed while blocking the training component. This shifts the burden from the site owner to the network layer, which can identify the specific intent of the crawl.

The implementation relies on a new classification system that distinguishes between Search, Training, and Agent behaviors. A single bot can exhibit multiple behaviors, and the new setting specifically targets the Training behavior of mixed-use crawlers. This is distinct from a simple robots.txt directive, which Cloudflare argues is insufficient because it cannot identify the crawler's intent or enforce compliance. Instead, Cloudflare uses its network position to classify the crawl, publish the preference, and block non-compliant requests, providing a more robust enforcement mechanism than standard text-based directives.

A significant part of this announcement is the introduction of the 'Accountable' designation for bot operators. This designation requires operators to provide mechanisms for opting out of training and summaries, offer URL-level visibility into content usage, and guarantee that opting out of training does not impact search results. Apple, Google, and Microsoft are cited as meeting these qualifications, either through current capabilities or time-bound commitments. This creates a tiered system where 'Accountable' operators are treated differently from those that do not meet these standards, potentially influencing how site owners configure their defenses.

The change also alters the behavior of existing controls. Previously, 'Block' and 'Block on pages with ads' settings did not apply to mixed-use crawlers to avoid accidentally removing sites from search. With the introduction of Disallow AI Training, these block settings now apply to all training crawlers, including mixed-use ones. This means that site owners who previously had to leave mixed-use crawlers unblocked to maintain search visibility can now use the new setting to refuse training while still using block settings for other unwanted traffic. The granularity increases, but it also requires site owners to understand the distinction between the new setting and the existing block rules.

The long-term goal extends beyond training to AI summaries, where the issue is not just whether content is used, but how much of it appears. Cloudflare plans to allow site owners to control the extent of content inclusion in summaries by early next year, set once on Cloudflare rather than per operator. This suggests a broader strategy of centralizing content usage preferences at the network level. For engineers and site operators, this means a shift from managing individual crawler relationships to configuring high-level policies that are enforced by the network, reducing the operational overhead of monitoring and blocking specific bots.