Enforcement
Voluntary preference signal. Communicates a requested condition or restriction to automated systems.
It only affects actors that detect and honor the signal; it does not itself prevent reuse.
Catalog status describes public availability, not legal validity, adoption, or proven effectiveness.
What it is
Websites can use robots.txt and related directives to communicate whether AI crawlers may access content. OpenAI, Google, Cloudflare, and other operators document crawler-specific handling, but compliance depends on the crawler and use case.
X-Robots-Tag sends crawler directives through HTTP response headers, which supports URL-level or response-level handling beyond a sitewide robots.txt file.
Cloudflare’s managed robots.txt setting adds known AI crawler disallow rules and Content Signals such as search, ai-input, and ai-train. Cloudflare is also testing a separate content-use extension for describing how accessed content may be stored or reused. The file expresses preferences; it does not block requests by itself.
Google’s user-triggered fetcher documentation notes that some user-initiated fetches generally ignore robots.txt, including AI-agent fetches under Google-Agent. That keeps robots-style controls useful but incomplete for AI access governance.
Evidence trail
- Jul 01, 2026 · Primary sourceCloudflare documents managed robots.txt content signals
- May 08, 2026 · Primary sourceGoogle documents user-triggered fetcher handling
- Sep 17, 2025 · Primary sourceOpenAI bots documentation
Adoption signals
No public adoption figure found.