GPTBot Says It Visits Your Site. Here's How to Check, Every Week.

We wrote before about whether an AI crawler can read your page at all — rendering, robots.txt mistakes, a 403 from your own firewall. That is a one-time question. You fetch the page as OAI-SearchBot, you look at what comes back, you fix it or you don't, and the answer stays true until you change something.

This post is about a different question: whether these crawlers actually show up, on an ongoing basis, once the one-time check passes. A page that was readable to GPTBot last month tells you nothing about whether GPTBot has fetched it since. Readability is a fact about your server. Traffic is a fact about their behavior, and it changes without you touching anything.

What you're actually looking for

Every major AI crawler identifies itself with a distinct user-agent string. The ones worth watching for:

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Bingbot, Amazonbot, DuckAssistBot, Bytespider, CCBot.

LetsLaunch's own robots.txt (src/app/robots.ts in this codebase) names several of these explicitly rather than relying on a wildcard rule, for the same reason you'd want to grep for them by name: a rule or a log line that just says "bot" tells you nothing about which one.

Two of these strings are worth separating in your head before you start counting anything, because they answer different questions.

Training crawlers and retrieval crawlers are not the same signal

We made this distinction in the readability post and it matters just as much for monitoring: GPTBot collects training data. OAI-SearchBot and PerplexityBot fetch pages to serve as citations in an actual answer. They run independently, on their own schedules, for their own purposes.

If you only track GPTBot, a busy week of hits tells you your content might end up in a future training run — and tells you nothing about whether you can be cited in an answer today. If you want to know about citation potential, you need to watch OAI-SearchBot, Claude-SearchBot and PerplexityBot separately, not lump every AI-labeled agent into one counter. A dashboard that reports "247 AI bot hits this week" without splitting training from retrieval is reporting a number nobody can act on.

Grepping your own logs

If you run your own server or have access to raw access logs, this is a grep away. Standard combined log format puts the request path, status code, and user-agent on one line, so a single pipeline gets you counts per crawler:

grep -E "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|Claude-SearchBot" \
  access.log \
  | awk -F'"' '{print $6}' \
  | sort \
  | uniq -c \
  | sort -rn

awk -F'"' splits the line on quote characters, and in combined log format the user-agent is the sixth quoted field — the request line and the referer are the fourth and there's no fifth field before it. sort | uniq -c turns that into a count per exact user-agent string, which is enough to tell you who's showing up and how often.

To see what a specific crawler is actually requesting, rather than just how often it shows up:

grep "PerplexityBot" access.log | awk '{print $7, $9}' | sort | uniq -c | sort -rn

Field 7 is the request path and field 9 is the status code in the default combined format (exact field numbers shift if your log adds extra fields, so check one line by hand before trusting the count). This tells you two useful things at once: which pages a crawler actually cares about, and whether it's getting 200s or something else on the way in.

Run this weekly, or wire it into whatever log-shipping you already have, and you get a trend instead of a single data point. A one-time curl check answers "can it read this page." A weekly grep answers "does it come back."

The version with no log access required

Most sites don't sit in front of raw logs — they sit behind Cloudflare, which is also LetsLaunch's own setup. Cloudflare's dashboard has an AI Crawlers section, under its AI Bots reporting, that shows this without any grep at all: total requests from AI crawlers, a breakdown of allowed versus blocked requests, a per-crawler split naming the major players — Google, OpenAI, Anthropic, Perplexity, ByteDance, Common Crawl, DuckDuckGo among them — and the paths those crawlers hit most.

If your site is already on Cloudflare, this is the first place to look before building anything yourself. It's the same information the grep gives you, already split by crawler, already trending over time, with nothing to instrument. The tradeoff is that it only sees what passes through Cloudflare's edge — if you serve some paths from elsewhere, those don't show up here.

Cloudflare's own traffic data shows AI bot traffic rising sharply year over year. We won't put a number on that, because the precise figure varies by which report and which time window you're reading, and a specific percentage attributed loosely is worse than no percentage at all. The direction is the useful part: this is traffic worth watching, not a rounding error, and it is worth setting up a recurring check rather than a one-time one.

If you just want a fast answer today

Setting up log monitoring or checking a Cloudflare dashboard is worth it if you want a trend. If you just want to know right now whether a specific crawler can reach a specific page, that's a narrower and faster question, and it's what our AI crawler check does: it fetches your page as each crawler, once, and shows you the status code and the text each one received. It won't tell you whether ClaudeBot visited last Tuesday. It will tell you, in the next thirty seconds, whether it could visit at all. Pair that with our guide to making a page AI-readable if the answer comes back wrong.

What a crawler hit does and doesn't prove

Here's the part worth being honest about, because it's the same caution we applied to citations in the readability post. Seeing PerplexityBot in your logs is necessary evidence. It is not sufficient evidence.

A hit tells you the crawler reached your page and (assuming a 200 and readable HTML) received your content. It does not tell you that content was used, indexed, or ever surfaced to a single user. You cannot infer citation frequency from crawl frequency — a crawler can fetch a page a hundred times and cite it never, and there's no dashboard, ours included, that closes that gap.

What monitoring does give you is the other direction, which is just as useful: absence of hits is real information. If OAI-SearchBot has never touched your domain, you cannot be cited from that index, full stop — no amount of good content changes that if the crawler was never there to read it. Monitoring won't tell you you're winning. It will tell you, reliably, when you've been shut out before you've done anything else.

Treat crawler-hit counts as a diagnostic, not a growth metric. They rule things out. They don't rule things in.

Do crawler visits mean my product will be cited by ChatGPT or Perplexity?

No. A crawler hit only confirms the page was fetched and, if it returned a readable 200, that the content was received. Whether it's later cited in an answer is a separate step nobody can currently observe from the outside, and we know of no reliable way to predict it from crawl frequency alone.

Do I need Cloudflare to monitor AI crawler traffic?

No, but it removes the setup work. Cloudflare's AI Crawlers analytics shows per-crawler request counts, allowed-versus-blocked splits, and top crawled paths for any site behind its network, with no log access needed. Without Cloudflare, the same information is available by grepping your raw access logs for the relevant user-agent strings.

What's the difference between monitoring GPTBot and monitoring OAI-SearchBot?

GPTBot hits reflect training-data collection. OAI-SearchBot hits reflect retrieval for citations in ChatGPT's search answers. They're separate crawlers with separate purposes, so tracking only GPTBot tells you nothing about whether you can currently be cited — you have to watch the search/retrieval agents separately to answer that question.