Is Cloudflare Blocking AI Crawlers on Your Site? The 2026 Settings, Explained

If your site sits behind Cloudflare and ChatGPT, Claude or Perplexity cannot seem to read it, Cloudflare's AI crawler settings are the first place to look — and they are not the settings most guides describe. The single "Block AI bots" switch from 2024 and the July 2025 "blocked by default" headlines have both been superseded. Since 1 July 2026 Cloudflare splits AI traffic into three categories, Search, Training and Agent, each with its own setting, and on 15 September 2026 it changed what those settings do.

The short version: on a site with default settings, Cloudflare does not block the crawlers that put you into ChatGPT's search answers. What blocks them is an explicit choice someone made — and one of those choices, "Block" on Training, now also blocks Googlebot.

Does Cloudflare block AI crawlers by default?

It depends on when your domain joined Cloudflare and whether it serves ads, but for most sites the answer in October 2026 is: not the ones that matter for AI search. Since 15 September 2026, new domains get one of two presets at onboarding:

Existing domains kept their choices, translated into the new categories. Anyone who had the old "Block AI bots" switch on now has Search: Allow, Training: Disallow AI Training, Agent: Block on pages with ads. Anyone who had it off now has Allow on all three.

Cloudflare's own messaging has not been perfectly consistent about this. A July 2026 press release said Free customers who had not changed their settings by 15 September would be moved to the ad-site defaults; the 15 September blog post says settings "carry over on their own." The later post describes what shipped, but it takes thirty seconds to check your own dashboard instead of trusting either.

What do the Search, Training and Agent categories cover?

Each category groups crawlers by what the company does with the page. Cloudflare's bot reference puts the major AI crawlers here:

This split mirrors how the AI companies document their own crawlers. OpenAI says that sites which opt out of OAI-SearchBot "will not be shown in ChatGPT search answers," while GPTBot is used for training only. Anthropic's crawler documentation draws the same line between ClaudeBot (training), Claude-SearchBot (search) and Claude-User (user requests). So:

• Training controls whether your pages go into future models. It has no effect on whether you are cited today.
• Search controls whether ChatGPT, Claude and Perplexity can find and cite you in answers. Block it and you disappear from AI search.
• Agent controls whether an assistant can open your page when a user pastes your link or asks it to look something up live.

If you have read our decision guide to blocking GPTBot, this is the same distinction, now built into Cloudflare's dashboard.

Does Cloudflare's "Block" setting block Googlebot?

Yes, since 15 September 2026, if you choose it for Training. Googlebot, Bingbot and Applebot are mixed-use crawlers: the same bot fetches pages for search and for AI. Cloudflare's announcement is blunt about the consequence of the Block option: "It will stop Applebot, Bingbot, and Googlebot from reaching your site — search included."

That makes "Block" on Training the single most dangerous setting on the page. It sounds like "keep my content out of AI training," and it also removes you from Google and Bing. The option that does what most people mean is the new one:

• Disallow AI Training publishes a no-training rule in your robots.txt and blocks the training-only crawlers — OpenAI's, Anthropic's, Meta's and Amazon's among them — while the mixed-use crawlers keep crawling for search.
• Block blocks every training crawler, mixed-use ones included, at the edge. Your Google rankings go with it.

Anyone whose Training setting was Block or "Block on pages with ads" before 15 September was migrated to Disallow AI Training automatically, so this trap mainly catches people changing the setting now.

One gap: Bing does not read the no-training rule yet. Cloudflare says Microsoft's support is targeted for early 2027, and until then the Disallow option sends Bing no training signal through robots.txt.

How do I check whether Cloudflare is blocking AI crawlers on my site?

Check the settings, then check the evidence, then — with caution — test.

1. Read the three AI settings. In the Cloudflare dashboard, open your domain and go to Security → Settings, filter by Bot traffic, and open the AI bot policies. You will see Search, Training and Agent, each set to Allow, Block on pages with ads, Block, or (Training only) Disallow AI Training.

2. Look for per-crawler rules. Open AI Crawl Control for the domain. Its Crawlers tab lists individual bots with an Allow or Block action, and a block set there is enforced by a WAF custom rule named "AI Crawl Control" under Security → Security rules. A single blocked crawler here overrides an "Allow" at the category level, and it is easy to forget it exists.

3. Read your live robots.txt. Open https://yourdomain.com/robots.txt in a browser. If Cloudflare manages part of it, its rules sit between # BEGIN Cloudflare Bot Preference Sync and # END markers, above your own. Look for a Disallow: / under OAI-SearchBot, Claude-SearchBot or PerplexityBot. Those are the lines that cost you AI citations. A disallow under GPTBot or ClaudeBot only opts you out of training.

4. Test with curl, and distrust the result. You can request a page while claiming to be a crawler:

# Baseline: an ordinary browser
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://yourdomain.com/

# The ChatGPT search crawler
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)" \
  https://yourdomain.com/

A 403 for the crawler and a 200 for the browser points at a rule aimed at that bot. But Cloudflare identifies real crawlers by IP address and behaviour, not just the user-agent string, and a request from your laptop claiming to be OAI-SearchBot is exactly what a spoofer looks like. In our own tests, one Cloudflare-protected site returned 403 to a plain browser user agent as well as to both OpenAI crawlers, so the test told us nothing about AI bots at all. Always run the browser baseline first, and treat a curl result as a hint.

The authoritative evidence is your own server logs, or the Metrics tab in AI Crawl Control: if real OAI-SearchBot requests from OpenAI's published IP ranges are arriving and getting 200s, you are crawlable. Our guide to monitoring AI crawler traffic walks through the log checks.

How do I allow GPTBot and other AI crawlers through Cloudflare?

Set the category to Allow, then remove anything that overrides it.

• To be cited in AI search but stay out of training — the setting most sites want: Search Allow, Agent Allow, Training Disallow AI Training. ChatGPT, Claude and Perplexity can index and cite you, and assistants can open your pages for users, while GPTBot and ClaudeBot are refused.
• To allow GPTBot as well: set Training to Allow. Then check AI Crawl Control for a per-crawler block on GPTBot, and your robots.txt for a Disallow under User-agent: GPTBot that you or a plugin added.
• Never choose Block on Training unless you intend to leave Google and Bing as well.

Changes apply at Cloudflare's edge immediately, but crawlers notice on their own schedule: OpenAI says changes to OAI-SearchBot access take about 24 hours to take effect in ChatGPT search.

Why do some AI crawlers ignore robots.txt?

Because two kinds of AI traffic do not treat it as binding. OpenAI says of ChatGPT-User, its user-triggered fetcher, that "robots.txt rules may not apply," and Perplexity says Perplexity-User "generally ignores robots.txt," on the reasoning that a person asked for the page. For those, a robots.txt line is a request, and Cloudflare's Agent setting, enforced at the edge, is the control that actually works.

The other kind is crawlers that do not identify themselves. In August 2025 Cloudflare removed Perplexity from its verified-bot list after documenting what it called undeclared crawling — we went through the evidence in Cloudflare vs. Perplexity on stealth crawling. Blocking at the edge by behaviour is what catches that; robots.txt never could.

Why would I let AI companies crawl my site at all?

Because the Search and Agent crawlers are how AI assistants send visitors back, and Training costs you little either way. Cloudflare's own data shows how lopsided the training side is. Its August 2025 analysis put Anthropic at roughly 38,000 pages crawled for every visitor referred, OpenAI at about 1,100 to 1, and Google at about 5 to 1, with training making up 80% of AI crawling over the previous year. Those ratios are the strongest argument for refusing training crawlers. They are no argument against search crawlers, which only fetch pages so that they can be cited and clicked.

That is why "Search: Allow, Training: Disallow AI Training" is the sensible default for a product site. It keeps you visible in ChatGPT, Claude and Perplexity answers without donating your pages to the next model. Whether that visibility turns into visits worth having is a separate question, answered with conversion data in is AI referral traffic worth having, and you can measure the visits with the method in our guide to utm_source=chatgpt.com.

Being crawlable is the precondition; being worth citing is the work. One cheap input is having more independent pages describe your product accurately. LetsLaunch serves every listing as server-rendered HTML with no JavaScript needed, and it allows every AI search crawler, so a free launch is one more of them.