Named crawlers, named opt-outs
Major AI companies publish specific user-agent strings identifying the crawlers they operate for collecting AI training data — GPTBot (OpenAI), CCBot (Common Crawl, widely used as a training data source), Bytespider (ByteDance), ClaudeBot and anthropic-ai (Anthropic), and Google-Extended (Google's training-specific crawler, distinct from its search-indexing crawler) among the most prominent. A site owner wanting to opt out of AI training collection specifically needs to disallow each named crawler explicitly, since a generic robots.txt rule doesn't automatically cover crawlers that weren't anticipated when it was written.
Why training and search/answer crawling are often separate
Some AI companies distinguish between a crawler used for training data collection and a separate crawler used for live, query-time retrieval to power an answer engine or search feature — Google-Extended specifically exists as a training-only crawler, distinct from Googlebot's search-indexing function. This distinction matters for a site that wants its content to remain discoverable and citable in AI answer engines while still opting out of having that content used to train underlying models.
Why robots.txt is a request, not an enforcement mechanism
robots.txt operates entirely on an honor system — a well-behaved crawler checks the file and respects its directives voluntarily, but nothing technically prevents a crawler from simply ignoring it. Major AI companies that have published named crawlers generally do respect robots.txt directives for their own named bots, which is exactly why naming them explicitly works in practice, even though the underlying mechanism has no technical enforcement.
Why a meta tag adds a complementary signal
An HTML meta tag like noai, placed in a page's head, provides an additional, page-level opt-out signal that some tools and platforms recognize independently of robots.txt — using both together covers more of the ecosystem's various opt-out recognition mechanisms than either alone.
What actually provides stronger enforcement
For a site owner who needs enforcement beyond an honor-system request, server or CDN-level bot blocking — such as Cloudflare's dedicated AI Scraper blocking feature — can technically deny access to identified AI crawlers rather than merely requesting they not access certain paths, providing a meaningfully stronger control for content owners with significant scraping concerns.
Why this list needs periodic revisiting
New AI crawlers are announced and named on a rolling basis as more companies build training data collection infrastructure — a robots.txt AI opt-out block created once and never revisited will miss newer crawlers that didn't exist at the time it was written, making periodic review and updates to the named crawler list worthwhile.
A deliberate choice, not a default
Opting out of AI training crawling is a legitimate choice for content owners who prefer their material not be used this way, but it's worth making deliberately — some site owners prefer their content remain fully open to AI training as a form of broader distribution and visibility, making this a decision to actively choose rather than assume.
Frequently Asked Questions
Major AI companies operate their own distinctly named crawlers — GPTBot, CCBot, Bytespider, ClaudeBot, Google-Extended, and others — and a generic robots.txt rule doesn't automatically cover crawlers that weren't anticipated when it was written. Explicitly naming each one is what makes the opt-out actually effective for that specific crawler.
Some AI companies run separate crawlers for training data collection versus live, query-time retrieval that powers an answer engine — Google-Extended, for example, is training-only and distinct from Googlebot's search-indexing crawler. Blocking only training crawlers lets your content remain discoverable and citable in AI answer engines while opting out of training use specifically.
robots.txt operates on an honor system — it requests that a crawler not access certain content, but nothing technically prevents a crawler from ignoring it. Major AI companies that have published named crawlers generally do respect these directives for their own bots, which is why explicit naming works in practice despite the lack of technical enforcement.
Yes — server or CDN-level bot blocking, such as Cloudflare's dedicated AI Scraper blocking feature, can technically deny access to identified AI crawlers rather than merely requesting they not access certain paths, providing meaningfully stronger enforcement for site owners with significant scraping concerns.
Yes — the AI Scraping Opt-Out Header Generator produces a correctly formatted robots.txt block covering 15 named AI training crawlers, plus a complementary HTML meta tag, with the choice to block all crawlers or exclude answer-engine crawlers specifically.