Should You Block GPTBot? The Decision, Not the Panic

Somebody on a founder Slack posts a robots.txt snippet that blocks GPTBot, someone else says "wait, doesn't that get you removed from ChatGPT," and the thread ends in fifteen replies of confident, contradictory advice. It shouldn't take fifteen replies. This is a narrow, mechanical question with a narrow, mechanical answer.

The one fact that resolves most of the confusion

Blocking GPTBot has no effect on whether ChatGPT can cite you in an answer.

That's it. That's the fact everyone in that thread needed. ChatGPT's search and citation feature is powered by a different crawler, OAI-SearchBot, and we've covered the split in detail in why ChatGPT can't see your site: GPTBot collects training data, OAI-SearchBot fetches pages for the index behind ChatGPT's citations, and OpenAI operates them as independent crawlers. Disallowing one in robots.txt says nothing to the other. If your actual goal is "I want ChatGPT to be able to find and quote my page," this entire post is irrelevant to you — go read that one instead, because the crawler you care about isn't the one this post is about.

If your actual goal is "I don't want OpenAI training a model on my content," keep reading, because that's a real and separate question, and it deserves an honest answer about what blocking GPTBot actually buys you.

What blocking GPTBot actually does

Per OpenAI's own published crawler documentation, and consistent with independent server-log analyses of how major AI crawlers behave, GPTBot generally respects a Disallow rule in robots.txt that names it specifically. The same is true of the other big named crawlers — Anthropic's ClaudeBot, Google's Google-Extended. This is worth being precise about, because it is an industry norm, not a legal requirement. Nothing compels a crawler operator to honor robots.txt; a rule aimed at a named user-agent can in principle be ignored, or the crawler could lie about its own identity. But the major, named crawlers have an established track record of compliance, and it's checkable — you don't have to take anyone's word for it. Pull your own server logs and look for the user-agent string after you make the change; if requests from it stop, the rule is being honored. If you want the full list of every named crawler and the rule that matches each one, see our AI crawler cheat sheet — this post is the decision framework for one crawler on that list.

Given that it works, what you actually get is narrower than most people assume going in:

• You opt your pages out of being used as training data going forward, starting whenever OpenAI's crawler next respects the rule.
• You get nothing retroactive. If your content was already fetched in a past crawl and used in a training run that already happened, blocking the crawler today does not undo that. There is no unlearning button on the other end of a robots.txt edit.
• You get no change to ChatGPT search or citations — covered above, worth repeating because it's the thing people actually came here worried about.
• You get no change to Google Search — GPTBot has nothing to do with Google, and blocking it doesn't touch your rankings there.
• You get no change to any other AI product's retrieval — Perplexity, Claude's web search, anything else that fetches your page live to answer a question is a separate crawler making a separate request, governed by whatever rule you have (or don't have) aimed at it.

So the actual decision on the table is small and specific: do you want this one company not training on this content from this point forward. Everything else people worry about when they think about this — search visibility, being "in ChatGPT," getting delisted somewhere — is not actually affected by this switch.

The case for blocking it

The underlying reason opt-out mechanisms like this exist at all is a real, ongoing disagreement in the industry: some publishers and rights-holders have raised copyright and compensation concerns about AI companies training models on their content without consent or payment. That's been a live, publicly reported point of contention between publishers and AI companies, and it's the actual origin of both the training/retrieval split and the robots.txt-based opt-outs that OpenAI, Anthropic and Google all support. Whether training on public web content should require consent or payment is not a question this post is going to answer for you — it's genuinely unsettled, reasonable people land on different sides, and your view on it is a fine reason to block the crawler on its own, independent of anything else in this post.

If that's your reason, block it. You don't need a further business case. It's a values call about your own content, and it's yours to make.

The case for not blocking it

The case against is more speculative, and worth stating as speculative rather than dressing it up as a strategy.

Some founders and publishers want their product to be part of what a model "knows" about when it's later asked general questions — the idea being that training data shapes a model's baseline familiarity with a topic or brand, which is a different thing from live retrieval and citation. That's a plausible-sounding motivation. It is not a proven one. We've written before, in what makes AI engines cite a source, about how thin the actual evidence is for anything that reliably causes a model to recommend or cite a specific product — and being present in a training run is a different, further-upstream bet than the citation question that post covers, with even less established evidence behind it. Nobody has shown that a product being in GPTBot's training data causes ChatGPT to mention that product later. It might contribute something to how the model talks about a category in general terms. It might do nothing measurable at all. There's no study to point at either way.

So "leave it open in case it helps the model know about us" is a bet, not a fact, and it should be weighed as one — against a concern (uncompensated training use of your content) that, for some site owners, is not speculative at all.

The middle path both companies explicitly support

Most people asking this question don't actually want an all-or-nothing choice, and they don't have to make one. OpenAI and Anthropic both explicitly support splitting training from retrieval at the crawler level, because they operate them as separate user-agents in the first place:

• OpenAI: disallow GPTBot (opt out of training), explicitly allow OAI-SearchBot (stay eligible for citation in ChatGPT's answers) and ChatGPT-User (stay reachable when a person pastes your link into a chat).
• Anthropic: the same shape — disallow ClaudeBot, allow Claude-SearchBot and Claude-User.
• Google: disallow Google-Extended if you want to opt out of Gemini/AI feature training, but never touch ordinary Googlebot — that one indexes you for Google Search itself, and blocking it removes you from Google entirely, which is a completely different and much larger consequence than anything else in this post.

This is the setting most sites end up at once they understand the split: opted out of training, still eligible to be cited, still reachable by a person with a link. It isn't a compromise so much as just correctly stating what you want, at the level of granularity the crawlers actually offer.

How to decide, in practice

Skip the philosophy question for a second and ask two narrower ones:

1. Do I have an actual objection to my content training a model, independent of any effect on my visibility? If yes, disallow GPTBot — that's the whole ask, and blocking it delivers exactly that with no side effects on search or citation.
2. Am I blocking it because I think it'll somehow hurt me if I don't, or help me if I do, in terms of getting found? If that's the actual reason, go re-read the mechanics above — there isn't one. Training and retrieval don't talk to each other. Decide on the merits of the first question instead, because the second one isn't actually load-bearing here.

If you're not sure which of those you are, that uncertainty is itself useful information: it usually means the real answer is "block GPTBot, allow the search and user-agent crawlers," and move on. It costs you nothing you were actually using, and it settles the one part of this that was a genuine, values-based decision rather than a guess about SEO mechanics.

Getting found in the first place is a separate problem from any of this — one more listing that names your product consistently is a small, low-cost input to that, and submitting to LetsLaunch takes a few minutes if you want one.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects training data; ChatGPT's search and citation feature is powered by a separate crawler, OAI-SearchBot. Blocking GPTBot opts you out of training runs and has no effect on whether you can be cited in ChatGPT's answers, because that depends entirely on OAI-SearchBot staying allowed.

Does blocking GPTBot undo past training?

No. A robots.txt rule only affects future crawls. If your content was already fetched and used in a training run before you added the disallow rule, blocking the crawler today does not remove it from a model that has already been trained.

Is GPTBot legally required to respect robots.txt?

No. Respecting robots.txt is an industry norm, not a legal obligation — nothing compels a crawler to honor a disallow rule aimed at it, and a rule naming a specific user-agent could in principle be ignored or spoofed. In practice, the major named crawlers, including GPTBot, ClaudeBot and Google-Extended, have a documented track record of compliance, and you can verify it yourself by checking your own server logs for the user-agent after adding the rule.