AI

Wikimedia's own engineering blog reported a 50% bandwidth surge from AI scraping, with bots making up 65% of the most expensive traffic while being a third of pageviews. Smaller sites are reporting the same pattern. Here's what a normal site can actually do about it.

In August 2025, Cloudflare accused Perplexity of disguising its crawler to keep scraping sites that had blocked it. Perplexity disputes the characterization. Here's what Cloudflare actually documented.

Nobody can promise you'll get more ChatGPT or Perplexity referrals. But several named analytics firms independently measured what that traffic does once it arrives — and it converts meaningfully better than organic search, even though the exact multiple depends on who's counting.

GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider and a dozen more — what each one actually does, who runs it, and the robots.txt rule that matches what you're trying to opt into or out of.

Three products, three different audiences, three different ways of deciding what to cite. A founder with limited time needs to know which one actually rewards the effort before spending any of it.

Google has said plainly it doesn't use llms.txt. A 137,000-domain study found 97% of published files get zero requests, ever. We still publish one — here's why that isn't a contradiction.

A curl check tells you what a crawler receives once. It tells you nothing about whether GPTBot, ClaudeBot or PerplexityBot actually show up on a Tuesday in October. Here's how to monitor that on an ongoing basis.

There is no special 'AI schema.' Google's own documentation says AI Overviews eligibility rides on the same requirements as an ordinary search snippet — but that doesn't mean structured data is pointless, and it doesn't mean FAQ markup gets you cited either.

Blocking GPTBot opts your site out of future OpenAI training runs. That's the whole effect. It does nothing to whether ChatGPT can cite you — that's a different crawler entirely.

One Princeton study measured a 22-41% visibility lift from adding quotes, statistics and citations inside a lab benchmark. It's the most solid data that exists — and it isn't the same claim as 'do this and ChatGPT will mention your product.' Here's the actual gap.

Wikipedia and Reddit together account for more than a quarter of ChatGPT's US citations, per a 5W synthesis of 680 million tracked citations. But the two sources aren't interchangeable — and neither is reachable on demand.

The research on AI citation says quotable, specific claims get cited more than vague ones. This is the practical follow-up: how to actually structure a paragraph so it can be lifted out of your page and still be correct.

AI tool directories are a different audience than a general launch board — someone browsing one is already hunting for a tool like yours. We checked the rel attribute on all 27 we track.

Googlebot renders JavaScript. The crawlers behind ChatGPT, Claude and Perplexity do not. A client-rendered site can rank perfectly well in search and still arrive at an assistant as an empty shell.