Cloudflare Says Perplexity Ignored Robots.txt. We Checked the Record.

Our own AI crawler cheat sheet says the named crawlers "have an established track record of compliance" with robots.txt. That line needs a footnote, and the footnote is Perplexity.

In a blog post published in August 2025, Cloudflare accused Perplexity of "stealth crawling" — disguising its crawler's identity to keep pulling content from sites that had explicitly blocked Perplexity's declared crawler, either through robots.txt or a network-level block. This is not a vague industry rumor. Cloudflare published a specific experiment, specific traffic numbers, and a specific response. Perplexity disputes the characterization. Both of those things are true at once, and this post is about being precise on which parts are documented fact and which part is a live dispute.

What Cloudflare says it found

Cloudflare's test was simple enough to describe in two sentences. It stood up brand-new domains that were never linked from anywhere and never submitted to any search engine or index, so the only way to know what was on them was to actually fetch and read them. Each domain's robots.txt disallowed every crawler, with no exceptions.

Then, according to Cloudflare's blog post, they asked Perplexity's own AI assistant about the content on those exact test domains — and got back detailed, accurate answers about pages the assistant should have had no legitimate way to see. Nothing was indexed. Nothing was linked. The only path from "page exists" to "assistant can describe it" was a crawl that ignored the disallow rule.

Cloudflare says the traffic behind this broke down into two distinct patterns:

• Perplexity-User, Perplexity's openly declared, user-triggered agent, running at roughly 20-25 million requests a day. This is the crawler named on our own cheat sheet and it isn't the accusation — a declared agent fetching a page a user asked about is normal and expected.
• A second, undeclared crawler, at roughly 3-6 million requests a day, identifying itself as an ordinary Chrome browser on macOS. According to Cloudflare, this traffic used IP ranges outside Perplexity's official published ranges and rotated across different network providers — different ASNs — in a pattern consistent with avoiding detection as a single identifiable source.

That second pattern is the entire allegation. Not that Perplexity crawls the web — every retrieval-based assistant does that, openly, and we cover the mechanics of it elsewhere. The allegation is that when the declared crawler was blocked, a second, disguised one kept going anyway.

What Cloudflare did about it

Cloudflare didn't just publish a blog post and leave it there. It removed Perplexity from its "verified bots" program, added detection rules for the disguised traffic pattern to its managed rule sets, and extended the ability to block that traffic to every customer — including free-tier sites that previously had no way to act on this kind of pattern even if they noticed it.

That's the detail worth sitting with. A robots.txt disallow line is something any site can write, for free, unilaterally. Detecting a crawler that is actively trying not to look like a crawler — rotating IP ranges and network providers to defeat pattern-matching — is not something most site owners can do themselves. It needs traffic volume and infrastructure at Cloudflare's scale to even notice the pattern, let alone block it. According to Cloudflare, over 2.5 million websites added AI-crawler blocking rules within a month of the related tooling becoming available — a number worth reading as approximate, since it's Cloudflare's own reporting on adoption of its own feature, not an independently audited count.

What Perplexity says

Perplexity disputes the characterization. Multiple outlets reported a company spokesperson pushing back, disputing that the undeclared crawler belonged to Perplexity at all, or disputing that any blocked content was actually accessed. We're stating that as what it is: a public rebuttal, not a concession, and not something we can independently verify from here.

We're not going to adjudicate that dispute for you. What we can say plainly is that Cloudflare's evidence is unusually concrete for this kind of claim — a controlled experiment on domains nobody could have found any other way, compared to the more common pattern of "our traffic logs look suspicious," which is much easier to wave away. That doesn't make the allegation settled. It makes it a specific, checkable claim rather than a vibe, which is more than most accusations like this get.

This isn't how OpenAI's crawlers have been reported to behave

Worth one paragraph of context: multiple reports have described OpenAI's crawlers — the same GPTBot and OAI-SearchBot covered on our cheat sheet — actually stopping when a site disallows them via robots.txt. That's a reported contrast, not a claim that OpenAI's crawlers are flawless everywhere at all times; we've said elsewhere that robots.txt compliance across the industry is a norm the major crawlers have generally followed, not a legal guarantee that binds any of them. This incident is the sharpest documented exception to that norm we're aware of. It is not proof the norm doesn't exist.

What this actually changes for a site owner

Nothing about this changes the mechanics we've written about elsewhere — see why ChatGPT can't see your site for the separate, much more common problem of crawlers simply not rendering your JavaScript. What it changes is how much you can trust a robots.txt disallow line to actually be the whole story.

The honest takeaway is narrow. A robots.txt rule is a request you cannot verify is being honored by every crawler that might read it, and this incident is the concrete case for why — not a reason to assume the file is theater, since the record shows most named crawlers do follow it. If you want to check whether your own site is actually being blocked the way your robots.txt claims, our AI crawler check fetches your page as each named crawler and shows you what came back, which tells you what's happening to declared traffic. It cannot tell you whether an undeclared crawler is also getting through — that's the part this whole story shows a text file was never going to catch, and why Cloudflare's fix lived at the network layer instead of in anyone's robots.txt.

And a separate reminder, because it's easy to conflate the two: none of this tells you anything about whether being crawled — declared or not — makes an assistant more likely to cite you. That's a different, unresolved question, and we've written about what actually correlates with citation in what makes AI engines cite a source. Getting crawled and getting cited are not the same problem, and this story is only about the first one.

Did Perplexity admit to stealth crawling?

No. Cloudflare published detailed findings — including a controlled experiment on unlinked test domains — alleging that an undeclared, disguised crawler kept accessing sites that had blocked Perplexity's declared crawler. Perplexity publicly disputed the characterization through a company spokesperson. This is a documented, disputed claim, not a settled admission.

Does this mean robots.txt doesn't work against AI crawlers?

No. The major named crawlers, including OpenAI's, have a general track record of respecting robots.txt, which is what our cheat sheet is built around. This incident is the clearest publicly documented exception to that track record, not evidence that compliance across the industry is fake. Treat it as a reason to verify rather than a reason to give up on the file entirely.

How did Cloudflare detect the alleged stealth crawling?

By creating test domains with a robots.txt disallowing every crawler, never linking or indexing them anywhere, and then asking Perplexity's AI assistant about content on those exact domains. According to Cloudflare, the assistant answered with accurate detail about pages it had no legitimate way to know about, which pointed to a crawler ignoring the disallow rule rather than any public index.