Our own AI crawler cheat sheet says the named
crawlers "have an established track record of compliance" with robots.txt.
That line needs a footnote, and the footnote is Perplexity.
In a blog post published in August 2025, Cloudflare accused Perplexity of
"stealth crawling" — disguising its crawler's identity to keep pulling content
from sites that had explicitly blocked Perplexity's declared crawler, either
through robots.txt or a network-level block. This is not a vague industry
rumor. Cloudflare published a specific experiment, specific traffic numbers,
and a specific response. Perplexity disputes the characterization. Both of
those things are true at once, and this post is about being precise on which
parts are documented fact and which part is a live dispute.
What Cloudflare says it found
Cloudflare's test was simple enough to describe in two sentences. It stood up
brand-new domains that were never linked from anywhere and never submitted to
any search engine or index, so the only way to know what was on them was to
actually fetch and read them. Each domain's robots.txt disallowed every
crawler, with no exceptions.
Then, according to Cloudflare's blog post, they asked Perplexity's own AI assistant about the content on those exact test domains — and got back detailed, accurate answers about pages the assistant should have had no legitimate way to see. Nothing was indexed. Nothing was linked. The only path from "page exists" to "assistant can describe it" was a crawl that ignored the disallow rule.
Cloudflare says the traffic behind this broke down into two distinct patterns:
Perplexity-User, Perplexity's openly declared, user-triggered agent, running at roughly 20-25 million requests a day. This is the crawler named on our own cheat sheet and it isn't the accusation — a declared agent fetching a page a user asked about is normal and expected.- A second, undeclared crawler, at roughly 3-6 million requests a day, identifying itself as an ordinary Chrome browser on macOS. According to Cloudflare, this traffic used IP ranges outside Perplexity's official published ranges and rotated across different network providers — different ASNs — in a pattern consistent with avoiding detection as a single identifiable source.
That second pattern is the entire allegation. Not that Perplexity crawls the web — every retrieval-based assistant does that, openly, and we cover the mechanics of it elsewhere. The allegation is that when the declared crawler was blocked, a second, disguised one kept going anyway.
What Cloudflare did about it
Cloudflare didn't just publish a blog post and leave it there. It removed Perplexity from its "verified bots" program, added detection rules for the disguised traffic pattern to its managed rule sets, and extended the ability to block that traffic to every customer — including free-tier sites that previously had no way to act on this kind of pattern even if they noticed it.
That's the detail worth sitting with. A robots.txt disallow line is
something any site can write, for free, unilaterally. Detecting a crawler that
is actively trying not to look like a crawler — rotating IP ranges and network
providers to defeat pattern-matching — is not something most site owners can
do themselves. It needs traffic volume and infrastructure at Cloudflare's
scale to even notice the pattern, let alone block it. According to Cloudflare,
over 2.5 million websites added AI-crawler blocking rules within a month of
the related tooling becoming available — a number worth reading as
approximate, since it's Cloudflare's own reporting on adoption of its own
feature, not an independently audited count.
What Perplexity says
Perplexity disputes the characterization. Multiple outlets reported a company spokesperson pushing back, disputing that the undeclared crawler belonged to Perplexity at all, or disputing that any blocked content was actually accessed. We're stating that as what it is: a public rebuttal, not a concession, and not something we can independently verify from here.
We're not going to adjudicate that dispute for you. What we can say plainly is that Cloudflare's evidence is unusually concrete for this kind of claim — a controlled experiment on domains nobody could have found any other way, compared to the more common pattern of "our traffic logs look suspicious," which is much easier to wave away. That doesn't make the allegation settled. It makes it a specific, checkable claim rather than a vibe, which is more than most accusations like this get.
This isn't how OpenAI's crawlers have been reported to behave
Worth one paragraph of context: multiple reports have described OpenAI's
crawlers — the same GPTBot and OAI-SearchBot covered on our
cheat sheet — actually stopping when a site
disallows them via robots.txt. That's a reported contrast, not a claim that
OpenAI's crawlers are flawless everywhere at all times; we've said elsewhere
that robots.txt compliance across the industry is a norm the major crawlers
have generally followed, not a legal guarantee that binds any of them. This
incident is the sharpest documented exception to that norm we're aware of. It
is not proof the norm doesn't exist.
What this actually changes for a site owner
Nothing about this changes the mechanics we've written about elsewhere — see
why ChatGPT can't see your site for
the separate, much more common problem of crawlers simply not rendering your
JavaScript. What it changes is how much you can trust a robots.txt disallow
line to actually be the whole story.
The honest takeaway is narrow. A robots.txt rule is a request you cannot
verify is being honored by every crawler that might read it, and this incident
is the concrete case for why — not a reason to assume the file is theater,
since the record shows most named crawlers do follow it. If you want to check
whether your own site is actually being blocked the way your robots.txt
claims, our AI crawler check fetches your page as
each named crawler and shows you what came back, which tells you what's
happening to declared traffic. It cannot tell you whether an undeclared
crawler is also getting through — that's the part this whole story shows a
text file was never going to catch, and why Cloudflare's fix lived at the
network layer instead of in anyone's robots.txt.
And a separate reminder, because it's easy to conflate the two: none of this tells you anything about whether being crawled — declared or not — makes an assistant more likely to cite you. That's a different, unresolved question, and we've written about what actually correlates with citation in what makes AI engines cite a source. Getting crawled and getting cited are not the same problem, and this story is only about the first one.
Did Perplexity admit to stealth crawling?
No. Cloudflare published detailed findings — including a controlled experiment on unlinked test domains — alleging that an undeclared, disguised crawler kept accessing sites that had blocked Perplexity's declared crawler. Perplexity publicly disputed the characterization through a company spokesperson. This is a documented, disputed claim, not a settled admission.
Does this mean robots.txt doesn't work against AI crawlers?
No. The major named crawlers, including OpenAI's, have a general track record
of respecting robots.txt, which is what our cheat sheet
is built around. This incident is the clearest publicly documented exception
to that track record, not evidence that compliance across the industry is
fake. Treat it as a reason to verify rather than a reason to give up on the
file entirely.
How did Cloudflare detect the alleged stealth crawling?
By creating test domains with a robots.txt disallowing every crawler, never
linking or indexing them anywhere, and then asking Perplexity's AI assistant
about content on those exact domains. According to Cloudflare, the assistant
answered with accurate detail about pages it had no legitimate way to know
about, which pointed to a crawler ignoring the disallow rule rather than any
public index.