What it is
Cloudflare’s AI Crawl Control offers multiple tools: AI Audit dashboard shows which bots crawl your site, managed robots.txt simplifies blocking, and a pay-per-crawl marketplace lets publishers monetize AI training access.
Evidence trail
- Nov 17, 2025 Content Signals launched
- Sep 10, 2025 AI Crawl Control open beta launched
- Jun 30, 2025 AI Audit and marketplace features launched
Adoption signals
Users
1M+ customers enabled AI bot blocking
- AI Audit and marketplace features launched Jun 30, 2025
Data volume
1B+ 402 responses/day
- AI Crawl Control open beta launched Sep 10, 2025
Examples
Real-world robots.txt blocklist Source
Cloudflare's Robotcop post uses this abbreviated news-site policy as a concrete AI crawler blocklist example.
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: / Content Signals robots.txt example Source
Cloudflare's Content Signals post shows how a site can allow search while disallowing AI training in robots.txt.
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /