Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.
Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.
Two Layers, One Decision, No Reconciliation
robots.txt asks crawlers not to fetch certain things and depends entirely on their cooperation. Bot management at a CDN or origin enforces access before a request reaches your application. Both control crawler access. Nothing checks that they agree.
They are also usually owned by different people. The robots file is edited by whoever handles SEO; the bot rules are configured by whoever set up the CDN, frequently during a security incident, and frequently years ago. Neither party thinks of the other's settings as related to theirs.
The result is a specific and invisible failure: a site that deliberately permits a retrieval crawler and returns it a 403 at the edge. The policy says allow. The infrastructure says no. Nobody finds out, because the only record of it is in logs almost nobody reads.
Why Nothing Tells You
A crawler that is refused never reaches your site. It leaves no analytics session, no Search Console entry, and no trace in any tool that measures what happened on your pages. From every surface except the log, a blocked crawler and an uninterested one look identical.
This matters most for the AI crawlers, because the tools people buy to measure AI visibility work by sampling prompts rather than by watching access. If a retrieval crawler is being turned away at your edge, a prompt-tracking tool will faithfully report that you are not being cited and offer no clue why. You would spend the quarter improving content that the system responsible for citing you cannot fetch.
The only instrument that answers this is your own access log, which records every request including the refused ones with their status codes. That is the one AI-visibility question answerable with facts rather than sampling, and the method is in log file analysis.
Where the Mismatch Comes From
Four common origins, none of which involve anyone deciding to block a crawler they wanted. The pattern is drift rather than intent, which is why it survives so long undetected.
| Origin | What happens | Typical response |
|---|---|---|
| Blanket bot-fighting turned on | Verified, well-behaved crawlers refused alongside scrapers | 403, or a challenge page a bot cannot pass |
| Rate limiting tuned for humans | A crawler working through a large site trips the limit | 429, then reduced crawling or abandonment |
| Platform defaults changed | Provider updates its stance on AI crawler categories | Categories blocked you never explicitly chose |
| Security response left in place | Rules added during an incident, never reviewed | Permanent block nobody remembers adding |
| Country or ASN blocking | Crawler infrastructure sits in a blocked range | Silent refusal from one region |
The third row has become the most common recently. Providers have moved toward classifying AI crawlers by purpose and applying defaults, so a category you never made a decision about can become blocked because the platform's default changed. That is a reasonable thing for a provider to do and a very easy thing for a site owner to miss.
Checking Your Own Site
The audit is short. You are looking for crawlers you intended to allow that are receiving anything other than a successful response, and for crawlers that have stopped appearing at all.
- Write down the intended policy first.
Which crawlers you allow, which you refuse, and why. Without this you cannot identify a mismatch, only observe traffic. The decision framework is in the crawler decision guide.
- Group log requests by user agent and status code.
Any allowed crawler with a meaningful share of non-200 responses is the finding. A crawler receiving only 403s is being refused regardless of what your robots file says.
- Check for absence, not just errors.
A crawler that has stopped requesting entirely may have given up after repeated refusals. Compare against an earlier period rather than looking only at the current one.
- Read your bot management settings directly.
Do not infer them. Open the configuration, list what is enabled, and check whether any category-level default covers crawlers you meant to permit.
- Verify identity before concluding.
Some refused requests are scrapers impersonating real crawlers, and blocking those is correct. Verify by reverse DNS or published IP ranges before treating a block as a mistake.
Fixing It Properly
The durable fix is one policy expressed in both layers rather than two configurations you hope agree. Where your provider offers it, deriving firewall rules from robots.txt removes the drift by construction.
Cloudflare offers a control that translates the AI bot rules in your robots.txt into firewall rules, which is the right shape: one source of truth, mechanically applied. Where no such feature exists, the practical equivalent is a documented policy and a scheduled check, because manual agreement between two systems degrades silently.
Whatever the mechanism, keep the distinction that makes the policy coherent in the first place. Training crawlers, retrieval crawlers, and user-triggered fetchers are three different decisions, and a blanket rule at the edge collapses them into one in a way that costs you the visibility while achieving none of the objection. That separation is the whole argument in should you block GPTBot.
Questions About Bot Blocking
- Can my firewall block a crawler I allowed in robots.txt?
Yes, and it is common. robots.txt is a request that well-behaved crawlers honour voluntarily. Bot management at your CDN or origin is enforcement that happens before the request reaches your application. The two are configured in different places, frequently by different people, and nothing reconciles them.
- How would I know it is happening?
Only from server or CDN logs. A crawler that receives a 403 never reaches your site, so it appears in no analytics, no Search Console report, and no prompt-tracking tool. The absence of a bot in your logs and the absence of citations look identical from every other surface.
- Does Cloudflare block AI crawlers by default?
Its bot products can, depending on which are enabled. Bot Fight Mode, Super Bot Fight Mode, WAF rules, and rate limiting can all turn away bots that respect robots.txt and identify themselves honestly. Cloudflare has also moved toward blocking training and agent-purpose crawlers by default in some configurations, particularly on pages showing ads.
- What does a blocked crawler actually cost me?
The cost depends entirely on which crawler you block. Blocking a training crawler costs nothing in visibility, which may be exactly what you intended. Blocking a search or retrieval crawler removes you from the surface it feeds, so a 403 to a crawler that produces citations is forfeited visibility you did not choose.
- How do I fix a mismatch once I find one?
Decide the policy once, then make both layers express it. Cloudflare offers a control that translates your robots.txt rules into firewall rules, which removes the drift between the two by construction. Whatever the mechanism, the fix is a single source of truth rather than two configurations you hope agree.
