Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.

Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.

The Direct Answer: Three Decisions, Not One

Block crawlers that ingest your content for model training if you object to that use. Allow crawlers that index your content so assistants can cite and link to you. Allow user-triggered fetchers, because those requests exist only because a real person asked about your page. Most sites should end up with a mixed policy, and almost no small site should end up with a blanket block.

The confusion is understandable, because the same company operates crawlers in more than one category and the names do not announce their purpose. OpenAI runs GPTBot for training collection, OAI-SearchBot for search indexing, and ChatGPT-User for live fetches during a conversation. Disallowing "OpenAI" in a robots.txt file is not a thing you can do; you disallow specific tokens, and which tokens you choose determines whether you are declining to feed a model or declining to be found.

Blanket blocking is popular because it feels like taking a position. It is worth being clear about what the position costs: you keep exactly as much control over training as a targeted block would have given you, and you additionally remove yourself from the surfaces where buyers are increasingly starting. That is a trade some publishers should make deliberately. It is not one to make by accident, through a plugin default or a checkbox labelled "block AI".

The Crawlers, Sorted by What They Give Back

Sort every AI user agent by one question: can a request from this crawler ever result in a person arriving on your site? If yes, it is a distribution channel. If no, it is a pure cost, and the only reason to permit it is that you are content for your work to be used that way.

User agentPurposeCan it send you traffic?
GPTBotOpenAI training data collectionNo
OAI-SearchBotIndexing for ChatGPT search resultsYes, via linked citations
ChatGPT-UserLive fetch triggered by a user's questionYes, and the person is already asking
ClaudeBotAnthropic content collectionNo
Google-ExtendedControls Gemini and Vertex training use, not Search indexingNo, and blocking it does not affect Search
GooglebotGoogle Search index, which also serves AI Overviews and AI ModeYes, and blocking it removes you from Search
PerplexityBotIndexing for Perplexity answersYes, via linked citations
CCBotCommon Crawl, an open dataset many models train onNo

The Googlebot row is the one that catches people. There is no separate crawler for AI Overviews: they are generated from the same index that serves ordinary results, so no robots.txt rule can remove you from AI answers while keeping you in Search. That control exists elsewhere, as the generative AI opt-out and snippet directives described in our read of the Search Console AI reports. Anyone selling you a robots.txt configuration that blocks AI Overviews is selling something that does not exist.

The Ratio That Makes the Decision Concrete

The useful metric is crawl-to-refer: how many times a bot requested your pages for every visitor it subsequently sent. A crawler with a very high ratio is consuming bandwidth and returning nothing. A crawler with a low ratio is a distribution channel that happens to arrive as a bot. Your own server logs are the only place this number exists honestly.

Circulating rules of thumb suggest blocking above roughly a thousand requests per referral and allowing below a couple of hundred. Treat those thresholds as someone else's starting point rather than a standard: the ratio depends on your content type, your traffic volume, and how often assistants have reason to cite you. What transfers is the method, not the numbers.

Running it takes an hour. Pull a month of access logs, group requests by user agent, and pull referral sessions by source from your analytics for the same window. Divide. The result is usually clarifying in a way the debate is not: one or two crawlers will account for most of the load, and the ones you were worried about may turn out to be sending you people. Because referral counting has its own gaps, pair this with the corrected attribution setup in fixing AI traffic in GA4, or you will underestimate the referrals side of every ratio.

What Changed in 2026: Control by Purpose

The tooling caught up with the distinction. Cloudflare shipped controls on 1 July 2026 that classify AI crawlers by use case, separating search, agent, and training purposes so operators can allow one and refuse another, rather than choosing between all access and none.

That builds on infrastructure assembled over the previous year: pay per crawl entered private beta on 1 July 2025, and AI Crawl Control reached general availability with configurable HTTP 402 "Payment Required" responses announced on 28 August 2025. The 402 response is the interesting part conceptually, because it converts the binary of allow-or-block into a price, and Cloudflare has since extended the same mechanism into a broader monetisation gateway built on the x402 payment protocol.

For a small site, the practical upshot is narrower than the headlines suggest. Charging AI companies for crawl access is a real option for publishers with substantial proprietary archives and negligible for a business with forty pages, because nobody is going to negotiate a licence for your service page. What does apply immediately is the use-case split: if your host offers it, you can now decline training while remaining fully available to the crawlers that produce citations, which is the policy most businesses would have chosen all along if the switch had existed.

Writing the Policy

Put the decision in robots.txt because that is where honest crawlers look, and understand that it is a request rather than a fence. Enforcement, if you need it, lives at the CDN or origin, where you can act on verified identity instead of a self-declared user agent string.

  1. Decide the training question first.

    It is a values and licensing question, not an SEO one, and it has no traffic consequence either way. Answer it, write the answer down, and stop revisiting it every time a new crawler appears.

  2. Allow every search and user-triggered agent.

    OAI-SearchBot, ChatGPT-User, PerplexityBot, and their equivalents. These are the tokens that turn into citations and clicks. Blocking them is the version of this decision that actually costs money.

  3. Never disallow Googlebot or Bingbot.

    Both serve conventional search, and Bing's index has downstream relevance to assistant answers as covered in Bing SEO for ChatGPT. A rule intended to limit AI use that catches these is the single most damaging mistake available here.

  4. Verify the file does what you think.

    Ordering and wildcard behaviour in robots.txt surprise people regularly, and a misplaced rule can disallow far more than intended. The parsing rules and the common traps are in robots.txt vs noindex vs canonical.

  5. Re-read it every six months.

    New tokens appear constantly and old ones change purpose. A robots.txt written in 2024 is now making decisions about a landscape that did not exist when it was written, usually by omission.

Questions People Ask About Blocking AI Crawlers

Should I block GPTBot?

Blocking GPTBot stops OpenAI collecting your content for model training and does not stop ChatGPT citing you in search-style answers, because that is a different crawler. If your objection is to training use, blocking GPTBot is the correct and targeted action. If your goal is traffic, blocking it costs you nothing directly, but blocking OAI-SearchBot alongside it would cost you referrals.

What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?

GPTBot collects content that may be used to train OpenAI models. OAI-SearchBot indexes pages so they can be surfaced and linked in ChatGPT's search results, which is the crawler that can send you visitors. ChatGPT-User fetches a page live because a person asked about it in a conversation. They are separate user agent tokens in robots.txt and they deserve separate decisions.

Can I block AI Overviews without losing Google Search traffic?

Not through robots.txt, because AI Overviews are served from Google's main Search index and crawled by Googlebot, not by a separate AI crawler. Blocking Googlebot removes you from Search entirely. The supported controls are the generative AI opt-out in Search Console and snippet-level directives such as nosnippet and max-snippet, which limit what can be shown rather than what can be crawled.

Does robots.txt actually stop AI crawlers?

It stops the ones that choose to honour it, which includes the major named crawlers from OpenAI, Anthropic, Google, and Perplexity, all of whom publish their tokens. It is a request, not an enforcement mechanism, and it has no effect on a scraper that ignores it or disguises itself. If you need enforcement rather than a request, that is a network-layer control at your CDN or origin, not a text file.

Should small businesses block AI crawlers at all?

Usually not, and certainly not indiscriminately. The case for blocking is strongest when your content is the product you sell, when a licensing arrangement makes free ingestion a contract problem, or when crawler volume is a genuine hosting cost. For most small sites the realistic risk is obscurity rather than exploitation, and blocking search-side crawlers to protest training-side behaviour forfeits referrals without affecting the thing being protested.

Primary Sources

SearchHandled Editorial TeamPublished Jul 21, 2026 · Last reviewed Jul 21, 2026. Every factual claim is checked against the linked primary sources; corrections can be submitted through our contact page.