Initiative profile

DIY robots handling (robots.txt++)

Live

robots.txt can be used to express AI crawler access preferences. See example from OpenAI. Additionally, the X-Robots-Tag response header allows servers to send crawler directives via HTTP response headers.

Visit website Back to catalog
Website
platform.openai.com/docs/bots
Latest update
Sep 16, 2025 OpenAI bots documentation
Primary approach
Preference signal

What it is

Websites can use robots.txt and related directives to communicate whether AI crawlers may access content. Many AI crawlers (such as GPTBot) publish user-agent strings and honor disallow/allow rules.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Robots-Tag

X-Robots-Tag allows servers to send crawler directives via HTTP response headers, enabling granular control per URL and response type beyond robots.txt.

See also https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/

Evidence trail