Yes, your content is almost certainly being crawled for AI training unless you have explicitly opted out, and no, blocking training does not have to hurt your AI visibility, as long as you block the right bots. Training bots (GPTBot, ClaudeBot, Google-Extended) collect content to train future models; search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index you so AI can cite you; and live-fetch bots (Claude-User, ChatGPT-User) pull your page when a user’s question needs it. You can block the first group freely. Block the second or third and you disappear from AI answers. This distinction is about to matter for millions of sites, because Cloudflare’s new default flips against blended crawlers on September 15, 2026.
Most of the panic around this topic comes from treating “AI bots” as one switch. It is three switches, and knowing which is which is the difference between protecting your content and making yourself invisible to your buyers.
Three kinds of bot, three different stakes
Training bots collect content that may train future models. GPTBot, ClaudeBot, and Google-Extended live here. Blocking them is a legitimate choice, an IP or philosophical one, and it has zero effect on whether AI recommends you today, because today’s answers come from search and live retrieval, not from this quarter’s training run. Visibility impact of blocking: none.
Search bots index your content so AI can find and cite it. OAI-SearchBot, Claude-SearchBot, and PerplexityBot live here. These are the opposite case: blocking them removes you from AI answers entirely, because an engine cannot cite what it never indexed. Never block these. Visibility impact of blocking: total.
Live-fetch bots fetch your page in real time when a user’s question needs it. Claude-User and ChatGPT-User live here. A block here means AI cannot read your page at the exact moment it is composing an answer that could have cited you. Keep them allowed. Visibility impact of blocking: high.
The single most common self-inflicted wound in AI visibility is a “block all AI bots” rule, added in 2024 and never revisited, that catches the search and live-fetch bots along with the training ones. If AI never cites you, this is the first place to look. The full decision framework for the allow-or-block question is in should you allow or block AI crawlers, the mechanics of Anthropic’s three bots are in how Claude finds and cites sources, and the ChatGPT side in is my website a source for ChatGPT and Perplexity.
The September 15 trap
This is where it stops being theoretical. Cloudflare, which sits in front of a large share of the web, is changing its defaults.
Starting September 15, 2026, Cloudflare will block “mixed-use” crawlers by default on ad-supported pages. The new default applies to new customers, new sites set up by existing customers, and every existing free-tier account. Search-only bots stay allowed; you can opt out of the block; but if you do nothing, the default now works against blended crawlers.
The complication is Googlebot. It crawls for Search and for AI training in a single bot, so it is “mixed-use” by definition. Block training through a blanket rule and you can block Googlebot on those pages too, quietly losing Google Search visibility, the exact thing almost nobody intends. Google’s own Google-Extended directive only partly solves this: it lets you exclude content from training and from Gemini products without affecting Search inclusion, but Googlebot still crawls for Search features including AI Overviews and AI Mode, which is the overlap Cloudflare’s new default is designed to expose.
What to actually do, before the deadline: check your Cloudflare zone settings on purpose rather than letting a default decide them. Allow search and live-fetch bots. If you want to opt out of training, use Google-Extended for Google and the specific training-bot directives for others, rather than a blunt block that catches everything. The deadline is not a threat; it is a reason to audit your bot rules deliberately.
The backdrop nobody mentions
Three numbers explain why this is happening now. By June 2026, training bots made up 50.6% of AI crawl traffic on Cloudflare’s network, while search bots, the ones that historically “paid” for access by sending clicks, had fallen to 10.7%. And more than half of all AI crawl traffic was re-fetching pages that had not changed since the last visit. The old bargain, let bots crawl freely in exchange for referral traffic, broke when the crawling stopped reliably sending traffic back. Cloudflare is renegotiating that bargain in public, and its Pay Per Crawl model is becoming Pay Per Use, paying publishers when AI actually uses their content in an answer rather than merely fetching it.
For a brand, the takeaway is not to pick a side in that fight. It is to make sure your own settings keep you visible while it plays out.
How to check what is happening on your site
Three free checks. Read your robots.txt and confirm you are not blocking OAI-SearchBot, Claude-SearchBot, Claude-User, or PerplexityBot; those are visibility, not training. Grep your server logs for bot user-agents to see who is actually crawling you and how often. And if you are on Cloudflare, open your zone’s bot and AI settings before September 15 and set them deliberately.
A robots.txt that opts out of training while keeping full AI search visibility looks like this:
# Block training bots (no visibility impact)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Search and live-fetch bots: leave crawlable (this is your AI visibility)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /If you want to go beyond checking crawlers to measuring whether AI actually cites and recommends you, that is what a visibility tool tracks; see how to monitor your brand across AI search engines.
FAQ
Is my content being used to train AI models? Almost certainly, unless you have opted out. Training bots like GPTBot, ClaudeBot, and Google-Extended crawl public content by default. You can block them via robots.txt or opt-out directives without affecting your AI search visibility.
Does blocking AI bots hurt my visibility in AI answers? It depends which bots. Blocking training bots (GPTBot, ClaudeBot) does not hurt visibility. Blocking search bots (OAI-SearchBot, Claude-SearchBot) or live-fetch bots (Claude-User, ChatGPT-User) removes you from AI answers. The common “block all AI” rule catches all three.
What is changing with Cloudflare on September 15, 2026? Cloudflare will block “mixed-use” crawlers by default on ad-supported pages, for new customers, new sites, and all free-tier accounts. Search-only bots stay allowed. Because Googlebot is mixed-use, a blanket block can also cost you Google visibility unless you configure it carefully.
How do I block AI training but stay visible in AI search? Block only the training bots (GPTBot, ClaudeBot) and use Google-Extended for Google, while explicitly allowing the search and live-fetch bots (OAI-SearchBot, Claude-SearchBot, Claude-User, ChatGPT-User). Never use a single blanket “block AI” rule.
Will blocking AI training hurt my Google ranking? It can, if done bluntly. Googlebot crawls for both Search and AI training in one bot, so a blanket training block can catch Googlebot. Use Google-Extended, which excludes training without affecting Search inclusion, instead of a broad block.
Once your bot settings are right, measure what actually matters: whether AI cites and recommends you, across all five engines. Start a free 7-day Rankry trial, no card, first report in two minutes.