Loading…
Guide
Almost everything written about “AI visibility” skips the part that decides it. Before any question of tone, structure or authority, there is a plain mechanical question: when the crawler behind ChatGPT, Claude or Perplexity requests your URL, does it get your content, or does it get an empty shell, a 403, or nothing at all?
That question has a definite answer, it is the same on every run, and you can check it yourself. This guide covers the three ways sites fail it, in the order they occur.
AI crawlers execute no JavaScript. Not “some,” not “eventually” — none. Vercel and MERJ observed this across more than 500 million GPTBot fetches: the crawler takes the HTML the server hands it and stops there. Whatever your framework would have rendered in the browser never happens.
This is why the problem is so easy to miss. Googlebot renders JavaScript, so a client-rendered single-page app can rank perfectly well in Google search while being, to an assistant, a <div id="root"></div> and a script tag. Your analytics look fine. Your rankings look fine. Your own browser shows the page beautifully, because your own browser runs the JavaScript. Every surface you would normally check is lying to you by omission.
A React or Vue app served as a static shell that fetches its content on mount is the clearest case: the crawler sees a nav, a loading state, and nothing else. But partial versions are more common than total ones. A server-rendered page whose pricing table, FAQ answers, reviews or feature list are fetched client-side after hydration will be read as a page that mentions a product and says almost nothing about it. Content injected by a third-party widget — a review embed, a chat-based docs search, a testimonial carousel loaded from a vendor — is invisible for the same reason.
Infinite scroll and “load more” are the same failure in slower motion: whatever is not in the first HTML response does not exist. So is content revealed by a tab or accordion, if the panel contents are fetched on click rather than shipped in the markup and hidden with CSS. If it is in the HTML and merely hidden, it is read fine.
The fix is not exotic. Server-render the content, or pre-render it at build time. Any framework can do this; most modern ones do it by default and only stop when a component opts out. The test is not which framework you use — it is what comes back over the wire.
There are two honest ways to see the page as a crawler sees it, and neither involves a score.
curl -A "OAI-SearchBot" -sS -o page.html -w "%{http_code}\n" https://example.com/
# then read what actually came back
wc -c page.htmlTwo numbers tell you most of it: the status code, and the size of the body. A 200 with three kilobytes of shell is a failure dressed as a success. Then open page.html in an editor and look for the sentences you would want quoted back to you. If they are not in that file, no assistant will ever see them.
Do it from a machine that is not on your own network, and not from an IP you have allowlisted. Home and office IPs frequently bypass exactly the protections that block the crawlers.
Same idea, done for several crawlers at once, from outside your network, with the raw text shown back to you.
Whichever way you check: the useful output is not a grade. It is the text. Read it and ask whether someone who only had that text could describe your product correctly.
This is the most consequential misunderstanding in the whole subject, and it costs sites their presence in ChatGPT’s answers by accident.
OpenAI operates more than one crawler, and they are not interchangeable. GPTBot collects training data. OAI-SearchBot fetches pages for the index behind ChatGPT’s search citations — the links it shows when it answers a question from the live web.
So the two decisions are genuinely separate. Blocking GPTBot keeps your pages out of training runs and changes nothing about whether ChatGPT can cite you. Blocking OAI-SearchBot removes you from those citations entirely. You may want the first. You almost certainly do not want the second by accident.
With one blanket rule. A snippet copied from a “block AI scrapers” post, a disallow-all group inherited from a staging config, a security plugin’s “AI bots” toggle that matches on a substring — any of these can sweep up the search fetcher alongside the training one. The rule looks like a principled stance against training on your content. Its actual effect includes deleting you from an answer engine.
# The accident: this removes you from ChatGPT's search citations too. User-agent: * Disallow: / # The deliberate version: opt out of training, stay citable. User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: /
Two more things worth knowing about this file. Crawler names change and vendors add new ones, so take the current list from the vendor’s own published documentation rather than from a blog post or a plugin’s defaults — a stale allowlist silently misses the agent that matters. And robots.txt is a request, honoured voluntarily; it is not access control, and it does nothing about content that was already fetched.
Check the file itself
Fetch https://yourdomain.com/robots.txt in a private window and read every group, not just the one you wrote. Rules arrive from CMS plugins, hosting platforms and CDN features, and the file that is served is often not the file in your repository.
You can have server-rendered HTML and a perfect robots.txt and still be unreadable, because the request never reaches your application.
Cloudflare’s AI bot blocking is enabled by default on several plans. It blocks at the edge. From your side everything looks correct — the HTML is fine, the robots file is fine, the site loads instantly in your browser — while the crawler receives a challenge or a 403 and goes away. Nothing in your CMS, your analytics or your rankings reports this. It is the single most common invisible failure we see, precisely because every check people normally run comes back clean.
Cloudflare is not the only source. Any WAF rule, bot-fight mode, rate limit, geographic restriction or aggressive “under attack” setting can do the same, as can a host that serves a JavaScript interstitial before the real page. So can a plugin-level firewall inside the CMS.
Only one way: make the request the way the crawler makes it, from somewhere else, and look at the status code. That is what the curl command above and the AI crawler check both do. If you get a 403, a 429, or a challenge page, the problem is at the edge and no amount of content work will fix it.
If you do want the AI crawlers through, the change is in your CDN’s dashboard, not in your codebase. And this is a real decision, not a formality: some sites deliberately keep them out, and that is a legitimate choice. What is not legitimate is making that choice without knowing you made it.
Once the request succeeds and the body is real, the remaining work is unglamorous and mostly the same work that has always made pages legible.
Say what the thing is, in the text. A page whose first hundred words are a slogan gives a crawler nothing to work with. One plain sentence naming the category, the user and the job — the sort of sentence a marketing review would strike out as obvious — is the sentence most likely to be reused when something describes you.
Keep the specifics on the page. Pricing, limits, platforms, integrations and what you do not do. Concrete facts survive summarisation; adjectives do not.
Use real headings and real links. Headings that describe the section beneath them, and <a href> elements rather than click handlers on divs. A crawler that runs no JavaScript cannot follow a route change that only exists in a router.
Be consistent across pages. The same product name, the same one-line description, the same category wording on your site, your docs, your changelog and anywhere else you appear. Consistent descriptions across independent pages correlate with being named by assistants — which is worth stating carefully: that is a correlation, and no one has shown that any individual page causes it.
Do not gate what you want quoted. Content behind a signup, an email wall or a cookie consent that replaces the body is content that does not exist for a crawler.
The interesting part of this field is how little of it is measurable, and how much is asserted anyway. Sorting the two is the whole discipline.
The status code a given user agent receives. Whether your content is in the HTML without JavaScript. What your served robots.txt says. Which crawlers appear in your server access logs and how often — the logs are the closest thing to ground truth you own, and almost nobody reads them. All of these are deterministic: same input, same answer, every time.
llms.txt. The proposal is appealing and the adoption is not there: the file is essentially never requested — one measurement of files in the wild found 97% never fetched at all. Publishing one costs nothing, but it is not a fix for any of the three failures above, and treating it as one means shipping a file instead of solving the problem.
FAQ schema for citations. Structured data has real uses in classic search. We have seen no evidence that adding FAQ markup produces a lift in how often an assistant cites you, and we are not going to imply otherwise.
Directory listings as a route to AI citations. We run a directory, so read this with appropriate scepticism — and note which way the interest points. There is no evidence that being listed somewhere causes an assistant to cite you. G2 and Capterra sit on enormous authority and receive close to no citations. If sheer domain strength were the mechanism, they would dominate; they do not.
“AI visibility” scores. Asking a model whether it knows your product does not return a stable answer: between 9% and 28% of decisions flip on repeat runs even at temperature zero. A single-shot check rendered as a number out of 100 is a coin flip with a progress bar. Serious measurement repeats each prompt many times and reports a distribution, which is why it is rarely free.
A useful test
When a tool or a post claims a tactic works, ask what was measured and how the counterfactual was established. If the answer is a correlation, a vendor case study, or nothing at all, file it under plausible and move on to the mechanical checks — those you can settle in a minute.
Strictly in this sequence, because each step is pointless if the one before it fails.
There is no sixth step where you buy something. The first three are free, take under an hour, and account for nearly every case where a site is simply not there.
LetsLaunch makes money when people list products here, so treat anything we say about listings with the scepticism it deserves. The checks above do not require you to trust us — every one of them you can run yourself with curl.