Every AI crawler identifies itself with a distinct user-agent string, and the
strings do not map cleanly onto "AI" as one category. Some train models. Some
power a live answer with a citation. Some only fetch a page because a person
pasted its URL into a chat window a minute earlier. A robots.txt rule written
against "AI bots" as if they were one thing usually does something other than
what its author intended.
This is the reference list: every crawler we could verify, who operates it,
what it actually does, and the specific robots.txt line for it. Use it to
write the rule you actually mean, not the closest wildcard you found in
someone else's blog post.
The three jobs a crawler can be doing
Before the table, the distinction that the whole list hangs on. A crawler identifying as AI-related is doing exactly one of three jobs:
- Training. It fetches your pages to become part of a dataset a model is trained on, sometime later, in a batch you have no visibility into. Blocking it has no effect on anything that happens today.
- Retrieval and citation. It fetches your pages to build or refresh an index that a live answer engine — ChatGPT Search, Perplexity, Copilot — reads from at answer time. Blocking it removes you from those answers, full stop.
- User-triggered fetch. Nobody crawls anything. A person pasted your URL, or asked about it by name, and the assistant fetched that one page, once, live, to answer that one question. This isn't a crawl in the ordinary sense — it's the assistant acting as the user's browser for a single request.
We laid out the training-versus-retrieval split in more depth in why ChatGPT can't see your site; this post exists to give every crawler its own row instead of two examples.
OpenAI
| Crawler | Job | Recommendation |
|---|---|---|
GPTBot | Trains OpenAI's models. Respects robots.txt, per OpenAI's own published crawler documentation. | Block if you want out of training; costs nothing in ChatGPT's answers. Allow if you're fine being training data. |
OAI-SearchBot | Powers ChatGPT Search's retrieval and citation index. A separate crawler from GPTBot, launched alongside ChatGPT Search in October 2024. | Allow, unless you want to be invisible to ChatGPT Search entirely. |
ChatGPT-User | User-triggered: someone inside ChatGPT pasted or asked about your specific URL and the assistant fetched it live. Not a routine crawl. | Allow. Blocking it means a real person who linked to you gets nothing back, for no gain — there's no dataset being built to opt out of. |
Anthropic
| Crawler | Job | Recommendation |
|---|---|---|
ClaudeBot | Anthropic's primary training crawler, functionally parallel to GPTBot. | Block for the same reason as GPTBot: opts out of training, changes nothing about Claude's live answers. |
Claude-SearchBot | Retrieval and citation crawler, parallel to OAI-SearchBot. | Allow if you want to be citable in Claude's answers. |
Claude-User | User-triggered fetch, parallel to ChatGPT-User. | Allow, same reasoning as ChatGPT-User. |
OpenAI and Anthropic both explicitly support the training/retrieval split — that's the entire reason two separate crawlers exist per company instead of one. If you want out of training but still want to be citable, that's not a compromise you're forcing on the system; it's the exact use case the split was built for.
Perplexity
| Crawler | Job | Recommendation |
|---|---|---|
PerplexityBot | Perplexity's primary crawler, builds its retrieval and citation index. | Allow if you want to appear in Perplexity's answers. Block only if you want out entirely — there's no separate training-only variant to opt out of instead. |
Perplexity-User | User-triggered fetch, parallel to ChatGPT-User. | Allow, same reasoning as the other user-triggered fetchers. |
| Crawler | Job | Recommendation |
|---|---|---|
Googlebot | Ordinary search crawler. Required to rank in Google Search at all. | Allow. This is not an AI-crawler decision — blocking it removes you from Google Search, not from an AI feature. |
Google-Extended | An opt-out signal specifically for Google's AI training uses (Gemini, AI features). Separate from Googlebot. | Block if you want out of Google's AI training. Blocking it does not affect Google Search ranking — that's the entire point of it existing as its own token. |
This is the pair people mix up most often, because both names start with
"Google" and only one of them touches your search rankings. Googlebot and
Google-Extended are read by Google's systems for entirely different
purposes, and disallowing the wrong one either does nothing or costs you your
search visibility, depending on which way you get it backwards.
Apple
| Crawler | Job | Recommendation |
|---|---|---|
Applebot | Ordinary crawler behind Siri and Spotlight search features. | Allow, unless you specifically want out of Apple's search surfaces. |
Applebot-Extended | Apple's AI-training opt-out signal, parallel in purpose to Google-Extended. Separate from ordinary Applebot. | Block if you want out of Apple's AI training without touching Siri/Spotlight visibility. |
Microsoft
| Crawler | Job | Recommendation |
|---|---|---|
Bingbot | Microsoft's ordinary search crawler — and also the retrieval backbone behind Copilot's web-grounded answers. Plays double duty the way Googlebot does for Google, except Microsoft hasn't split it into two tokens. | Allow. Blocking it costs you both Bing Search ranking and Copilot citations at once, because there's no separate opt-out for one without the other. |
Meta
| Crawler | Job | Recommendation |
|---|---|---|
Meta-ExternalAgent | Meta's AI-training crawler, for Llama and Meta AI features. | Block if you want out of training for Meta's models. |
Meta-ExternalFetcher | User-triggered fetch, parallel to ChatGPT-User, for links a user shares with Meta AI. | Allow, same reasoning as the other user-triggered fetchers. |
Everyone else on the list
| Crawler | Operator | Job | Recommendation |
|---|---|---|---|
Amazonbot | Amazon | Crawls for Amazon's AI/Alexa-related features. | Allow unless you specifically want out of Amazon's AI features — no training/retrieval split is documented for it. |
DuckAssistBot | DuckDuckGo | Powers DuckDuckGo's AI-assisted answer feature. | Allow if you want to be citable in DuckDuckGo's AI answers; this is a retrieval crawler, so blocking it removes you from those answers. |
Bytespider | ByteDance (owner of TikTok) | Crawls to power TikTok search, content recommendations, and ByteDance's own AI features. | Your call — bundles search, recommendations and AI training into one crawler with no separate opt-out for any one piece. |
CCBot | Common Crawl | A nonprofit that crawls the open web and republishes the dataset for anyone to use — researchers and AI labs included. Not itself an AI company, but its dataset is a common ingredient in many models' training data. | Block if your goal is keeping content out of AI training generally, since the dataset feeds more than one lab at once. Allow if you're fine contributing to open research datasets. |
Diffbot | Diffbot (commercial web-data extraction) | Crawls on behalf of its own customers to supply structured data for AI training, search, and retrieval-augmented generation products. | Treat like CCBot: it's a pass-through to unknown downstream customers rather than one named consumer product, so block it if you want a hard boundary on who resells your content. |
Timpibot | Timpi | Builds a decentralized search engine's index and collects training data for LLMs. Smaller and less well-known than the crawlers above, but real and documented. | Allow or block on the same logic as any other combined search/training crawler — there's no split to take advantage of. |
cohere-ai | Cohere | Training/retrieval crawler for Cohere's models. | Allow if you want to be usable by Cohere's products; block to opt out. LetsLaunch's own robots.txt names this one explicitly rather than relying on a wildcard. |
MistralAI-User | Mistral AI | Training/retrieval crawler for Mistral's models. | Same logic as cohere-ai. LetsLaunch's robots.txt names this one too. |
What LetsLaunch actually does
Our own robots.txt (src/app/robots.ts in this codebase) names sixteen
agents explicitly in an AI_SEARCH_AGENTS group and allows all of them,
including cohere-ai and MistralAI-User alongside the bigger names — a
listing cited inside any of these answer engines is worth as much to a maker
as an ordinary search result, so we chose visibility over opting out. The code
comment on that group is direct about the user-triggered fetchers too:
ChatGPT-User, Claude-User and Perplexity-User ignore robots.txt by
design, so naming them there is a statement of intent, not a control that
actually stops anything.
That's our call for a directory that exists to get products seen. It is not
a universal recommendation. A publisher with a subscription paywall, or
content it doesn't want folded into a model's training data regardless of
citation upside, has a legitimate reason to block the training crawlers on
this list while leaving the retrieval and user-triggered ones alone — that
split is exactly what separate tokens like GPTBot/OAI-SearchBot and
ClaudeBot/Claude-SearchBot are for.
Writing the actual rule
Robots.txt groups by user-agent are independent — a crawler that finds a group
naming it by name ignores the * group entirely, so a specific agent needs
its own disallow lines if you want to block just that one:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Allow: /
That example opts out of OpenAI and Anthropic training specifically while
staying allowed everywhere else, including for OAI-SearchBot's retrieval
index. Copy the shape, not the specific agents — decide per crawler, from the
tables above, rather than pasting a blanket AI-blocking snippet that treats
every name on this page as the same decision.
Checking whether it's actually working
A robots.txt rule is a request, not an enforcement mechanism — the polite
crawlers on this list respect it, but the file itself can't verify anything.
Once you've written the rule you mean, two different checks matter and they
answer different questions. Our AI crawler check
fetches your page as each named crawler once and shows you the status code and
the content each one received right now — useful for confirming a rule change
took effect. For whether a crawler actually shows up over time, on its own
schedule, we wrote a separate guide to
grepping logs and reading Cloudflare's AI Crawlers dashboard — that's an
ongoing question a one-time fetch can't answer.
If you're launching a product and want it reachable by name across these answer engines in the first place, a LetsLaunch listing is one more independent page describing it the same way your own site does.
Does blocking GPTBot remove me from ChatGPT's answers?
No. GPTBot only collects training data; OAI-SearchBot is the separate
crawler that builds the index behind ChatGPT Search's citations. Blocking
GPTBot opts you out of training runs and has no effect on whether ChatGPT
can cite you — only blocking OAI-SearchBot does that.
Does blocking Google-Extended hurt my Google ranking?
No. Google-Extended is a distinct opt-out token for Google's AI training
uses, separate from Googlebot, which is the ordinary search crawler required
for ranking in Google Search at all. Blocking Google-Extended leaves
Googlebot and your search ranking untouched.
Should I block the user-triggered fetchers like ChatGPT-User?
Usually not. ChatGPT-User, Claude-User, Perplexity-User and
Meta-ExternalFetcher don't crawl your site on a schedule — they fetch one
page, once, because a real person pasted your link or asked about it by name
inside that assistant. Blocking them doesn't opt you out of any training
dataset; it means that specific person gets nothing back.