What it is
Cloudflare’s AI Crawl Control offers multiple tools: AI Audit dashboard shows which bots crawl a site, managed robots.txt simplifies blocking, redirects can steer verified AI training crawlers toward canonical content, and pay-per-crawl lets publishers monetize automated access.
Evidence trail
- Apr 17, 2026 Redirects for AI Training launched
- Sep 24, 2025 Content Signals Policy launched
- Aug 28, 2025 AI Crawl Control general availability announced
- Jul 01, 2025 AI Audit and marketplace features launched
Adoption signals
Users
3.8M+ domains on managed robots.txt
- Content Signals Policy launched Sep 24, 2025
Data volume
1B+ 402 responses/day
- AI Crawl Control general availability announced Aug 28, 2025
Examples
Real-world robots.txt blocklist Source
Cloudflare's Robotcop post uses this abbreviated news-site policy as a concrete AI crawler blocklist example.
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: / Content Signals robots.txt example Source
Cloudflare's Content Signals post shows how a site can allow search while disallowing AI training in robots.txt.
User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /