Cloudflare has moved content-use intent from crawler policy prose into an API parameter. For research agents, that creates a useful fail-closed boundary between finding a source and ingesting it for a particular purpose.

What Changed

On August 31, 2026, Cloudflare announced that its Browser Run crawl endpoint now respects the Content Signals use directive. Callers can set contentUse to reference or full. If a target site's robots.txt expresses a more restrictive level, Cloudflare rejects the request with a 400 response.

The default is full, so an operator that only needs a citation or reference should declare the narrower intent explicitly rather than inherit the most permissive request shape.

Purpose And Use Are Separate Controls

Cloudflare's endpoint documentation also exposes crawlPurposes values for search, ai-input, and ai-train. A site can disallow one declared purpose even when another remains acceptable.

That distinction matters. A company may allow discovery while declining model training, or allow reference while declining full-content reuse. One generic crawler permission cannot represent those choices accurately.

Build The Refusal Into The Pipeline

  • Declare the narrowest purpose and use level before the crawl begins.
  • Record the declared intent, source URL, time, and resulting permission status.
  • Treat a content-signal rejection as a stop condition for that source and use.
  • Do not retry through another provider to evade a publisher's restriction.
  • Keep provenance attached when allowed content moves into notes or a knowledge base.

Scope And Limits

This is enforcement inside Cloudflare's crawl product, not a universal protocol or a legal ruling about copyright and licensing. Sites without a relevant signal will not produce the same refusal, and other crawlers may behave differently.

The practical value is still real: agent teams now have an implementation example where content intent is declared before collection and a conflict becomes a machine-readable failure instead of an ignored preference.

Related: Read our field notes on purpose-built agent web infrastructure and source-linked search tooling.