A community-curated index of initiatives for controlling AI data flow — licensing, preference signals, and technical enforcement.
Maintained by Nick Vincent. Contributions welcome. New to these concepts? See the Glossary.
Select a goal below to filter the catalog.
Click a goal to filter, click again to clear
Machine-readable hints for AI agents/crawlers; easy to adopt, rely on compliance.
Legal terms and institutions that define and negotiate reuse rights.
Perimeter and platform measures that block, meter, or monetize automated access.
Emerging standards, protocols, or infra that carry preference and licensing metadata across the AI pipeline.
robots.txt can be used to express AI crawler access preferences. See example from OpenAI. Additionally, the X-Robots-Tag response header allows servers to send crawler directives via HTTP response headers.
A machine-readable licensing schema for clearly signaling reuse permissions and conditions (including payment or use restriction).
Fastly CDN partnership announced
Add subdomains to make content accessible to AI with blocking and monetization.
State of the Bots post (https://tollbit.com/bots/25q2/)
Approach from OpenMined
Launch target
'The simple meta tag that makes AI companies pay for your content.'
Actively onboarding creators
Certify that models are fairly trained
Launch target
Proposed signals for communicating reuse preferences to AI and web agents.
Response to feedback (https://creativecommons.org/2025/08/27/we-asked-you-answered-how-your-feedback-shapes-cc-signals/)
Internet Engineering Task Force is working on a standardized preference signal for AI agents and crawlers ("building blocks that allow for the expression of preferences about how content is collected and processed for Artificial Intelligence (AI) model development, deployment, and use.")
Milestone for protocol specifications
A set of tools to block or charge for scraping; includes AI Audit dashboard, managed robots.txt, and pay-per-crawl marketplace.
AI Audit and marketplace features launched
A 50/50 revenue-share platform connecting publishers with AI companies, with 700+ publishers signed up including major news outlets.
Gist search engine launched
W3C specification for expressing text and data mining permissions via a well-known JSON file, designed for EU DSM Directive compliance.
W3C Community Group active
{
"tdm": "optout",
"license": "https://example.com/license"
}
A proposed convention for AI-specific crawler directives via an `ai.txt` file.
User guide post (https://spawning.substack.com/p/the-spawning-guide-to-rights-reservations)
A simple anti-scraping tool intended to protect datasets from basic crawlers/scrapers.