Most of what gets written about AI crawlers is about visibility — can ChatGPT see your page, will it cite you, should you welcome the traffic. None of that is the problem for a growing list of sites. Their problem is that the traffic showed up, kept showing up, and the bill came due.
The clearest documented case is Wikimedia. The Wikimedia Foundation's own engineering blog reported that bandwidth consumption for multimedia downloads from Wikimedia Commons has surged 50% since January 2024, and it did not hedge about the cause: the surge was not coming from human readers, but from "automated, data-hungry scrapers looking to train AI models." TechCrunch's reporting on the post picked up the same figures. This is not a monitoring question — we covered that one already — it's a cost and reliability question, and it deserves its own answer.
Why bots cost more per request, not just more in total
The intuitive model is that bot traffic is a scaled-up version of human traffic — more requests, same shape, so the cost scales linearly with volume. Wikimedia's explanation is that this isn't what happened, and the mismatch is the actual story.
Human traffic clusters around popular pages, the ones a CDN or edge cache already has warm. Crawler bots, per Wikimedia's own account, tend to "bulk read" — they work through large numbers of pages indiscriminately, including the long tail of obscure pages that were rarely requested before and were never worth caching. Serving those requests means going back to origin infrastructure instead of an edge cache, every time. A bot doesn't just add volume; it adds volume in the expensive shape.
The numbers back this up directly. Bots accounted for roughly 65% of the most resource-intensive traffic Wikimedia serves, despite representing only around 35% of overall pageviews. Read that gap carefully: the group causing two-thirds of the expensive work is a third of the requests. If you only looked at request counts, you would badly underestimate what's driving your bill.
Wikimedia was blunt about what this does to infrastructure built for a different kind of spike: its systems, in the Foundation's own words, were "built to sustain sudden traffic spikes from humans during high-interest events, but the amount of traffic generated by scraper bots is unprecedented and presents growing risks and costs." That's a nonprofit that has spent two decades engineering for viral human traffic, saying the thing it wasn't built for is a bot.
It isn't only Wikimedia
Wikimedia has the size and the engineering culture to publish a number. Most sites carrying the same problem don't, which is a reason to take the anecdotal reports seriously rather than dismiss them for lacking a published percentage.
The same body of reporting that surfaced Wikimedia's figures also names several smaller, independent infrastructure projects describing the identical pattern in their own words: the git-hosting service SourceHut, Diaspora developer Dennis Schubert, the repair-guide site iFixit, and the documentation host Read the Docs have all separately and publicly reported bandwidth spikes and inflated infrastructure costs they attribute to AI-crawler traffic — in some cases severe enough to affect site reliability for real users. Unlike Wikimedia's figures, none of these came with a formal published percentage, so treat them as real, corroborating, but less precise than Wikimedia's own numbers. The pattern showing up independently across a git host, a documentation site, a repair-guide site, and an individual developer's project is still worth something even without a shared methodology behind it.
Wikimedia's response is measurable too: its 2025/2026 annual plan sets an explicit goal of cutting crawler-generated request rate by 20% and bandwidth usage by 30%. That's a nonprofit budgeting engineering time specifically against bot load, not against human growth — which tells you how large the line item got.
What this means if you're not running Wikimedia-scale infrastructure
Almost nobody reading this runs a site at Wikimedia's scale, and that's the point worth making rather than skipping past: the mechanism doesn't require that scale. A "bulk read" crawler that hits your long tail of rarely-visited pages, bypassing your cache the same way, produces the same disproportionate cost on infrastructure a hundredth the size. The absolute dollar figure is smaller. The ratio — request share versus cost share — is the same shape.
If you're behind a CDN, you already have more leverage here than you might realize, and most of it costs nothing to turn on.
Start with rate-limiting, not blocking. LetsLaunch runs behind Cloudflare, the same setup we've described before for watching crawler traffic — the same dashboard that shows you who's visiting can also throttle them. Cloudflare's rate-limiting and bot-management rules can target a specific crawler's request rate directly, which is a much narrower tool than an outright block. A crawler capped at, say, one request per second still finishes crawling your site — just slower, and without hammering your origin in a burst. That's usually the actual goal: less load, not zero bots.
Know what blocking actually trades away. This is the part worth being
honest about, because it points in the opposite direction from most of what
this blog has argued elsewhere. We've spent several other posts
on the case for staying crawlable, because citation eligibility depends on a
crawler being able to reach your page at all. Blocking a retrieval crawler
outright — OAI-SearchBot, PerplexityBot, Claude-SearchBot — solves a
bandwidth problem by creating a visibility problem. If your actual goal is
citation, rate-limiting is almost always the better trade: a slower crawl
still completes, and you stay in whatever index that crawler feeds. Reserve
an outright block for a crawler that's demonstrably causing reliability
problems for real users right now, not for one you're merely annoyed at
seeing in the logs.
Proof-of-work challenges are a real, separate category. Some smaller open-source infrastructure projects have adopted tools that put a lightweight computational challenge in front of a request before serving the page — cheap enough that a normal browser clears it invisibly, expensive enough that sustaining a bulk-scraping pattern across thousands of pages gets computationally costly for whoever is running the scraper. This is a real and growing defense worth knowing about by mechanism, whether or not you adopt it — it's a different lever from a robots.txt rule or a firewall block, because it changes the cost calculus for the crawler operator rather than just refusing the request.
Match the response to the actual problem. The uncomfortable truth is that "should I block AI crawlers" doesn't have one answer, because it's two different questions wearing the same sentence. A site chasing citation eligibility should not blanket-apply advice written for a bandwidth crisis — "just block everything AI-labeled" throws away exactly what you're trying to get. A site with a genuine bandwidth or reliability problem should not feel obligated to stay maximally open just because citation matters to sites in general — if the traffic is actively degrading service for real users, that's a legitimate reason to throttle or block specific offenders, full stop. Check our crawler cheat sheet for what each bot actually is before deciding which category you're in.
Is AI crawler traffic actually costing sites money?
Yes, documented directly. The Wikimedia Foundation's own engineering blog reported a 50% surge in bandwidth for multimedia downloads from Wikimedia Commons since January 2024, attributed to AI-training scrapers rather than human readers, with bots responsible for roughly 65% of its most resource-intensive traffic despite being about 35% of pageviews. Smaller sites — SourceHut, iFixit, Read the Docs, and independent developer Dennis Schubert among them — have separately reported the same pattern, without Wikimedia's level of published detail.
Should I block AI crawlers to save bandwidth?
Only if you have a genuine, demonstrated bandwidth or reliability problem, and rate-limiting is usually the better first move over an outright block. Throttling a crawler's request rate reduces load while letting the crawl still complete, which keeps you eligible for citation in that crawler's index. A hard block trades your bandwidth problem for a visibility problem — worth it if the traffic is actually degrading your site for real users, not worth it if you're just trying to reduce a number in a dashboard.
Why does bot traffic cost more than the same amount of human traffic?
Because of what gets requested, not just how much. Wikimedia's own explanation is that crawler bots "bulk read" through large numbers of pages, including rarely-visited ones that were never popular enough to be cached — so each of those requests goes back to origin infrastructure instead of being served from a CDN edge cache. Human traffic clusters around popular, already-cached pages; bot traffic doesn't, which is why a smaller share of requests can still account for the majority of the most expensive ones to serve.