Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.
Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.
What Was Announced, and When It Applies
Cloudflare announced on 1 July 2026 that AI companies have until 15 September to separate crawlers used for search from those used for training and agents. After that date, on pages carrying advertising, search crawlers remain allowed by default while training and agent crawlers are blocked by default.
Two groups get the defaults applied rather than chosen: sites on the free tier, and every domain onboarding after the date. Existing paying customers retain the ability to override from the dashboard. If you are in the first group and have never thought about crawler policy, a policy is about to be selected for you.
This is a continuation rather than a reversal. Cloudflare has been moving toward purpose-based crawler classification for over a year, having shipped controls that sort AI crawlers into search, agent, and training categories. The September date is when the defaults change for sites that have not configured anything.
The Part That Deserves a Check Rather Than a Shrug
Crawlers that serve more than one purpose are the complication. Reporting on the change describes multi-purpose crawlers as facing the strictest applicable rule, which means a rule blocking training can take the entire crawler with it, and mixed-use crawlers that do not declare their purpose per request can be blocked outright on ad-supported pages.
Googlebot is the obvious case. It serves ordinary search results and Google's AI features from the same crawl, which is exactly why no robots.txt rule has ever let you appear in Search while opting out of AI Overviews. A blunt block on training-purpose crawling applied to a crawler that cannot separate its purposes is not a small configuration decision.
Treat this as something to verify rather than something to panic about. The precise behaviour depends on your plan, your rules, and how the categories resolve on your specific configuration, and that is knowable today by reading your settings and your logs. What is not advisable is assuming a default is safe because it sounds conservative.
The Check, Before the Date
Four questions, answerable in under an hour, and worth answering whether or not you use Cloudflare, because the same categories are appearing across providers.
- Is your site behind Cloudflare, and on which plan?
This determines whether defaults apply to you automatically or whether you keep an override. Free-tier sites and new domains are the ones with a decision being made for them.
- Do the affected pages carry advertising?
The stricter defaults are described as applying to ad-carrying pages. A site with no advertising is in a different position from a publisher, and knowing which you are changes the urgency.
- Which categories does your configuration currently permit?
Read the settings rather than inferring them from robots.txt. These are separate systems, and the gap between them is a documented failure mode covered in when your firewall refuses crawlers you permitted.
- Decide the policy, per category, deliberately.
Training, search, and agent are three separate decisions with different consequences. Refusing training costs no visibility; refusing search or retrieval crawlers costs citations and clicks, per the crawler decision guide.
- Verify in logs afterwards, not in the dashboard.
A configuration that says allow and a log that shows 403s is the situation that costs visibility silently, and the log is the only place it is visible, per log file analysis.
What Each Category Is Actually Worth to You
The decision is only difficult if the categories are treated as one thing. Separated, most businesses find they have a clear preference for each, and it is rarely the blanket position.
| Category | What it does for you | Cost of blocking | |
|---|---|---|---|
| Search | Indexes you so answers and results can link to you | Direct loss of visibility and clicks | |
| Agent | Fetches on behalf of a person acting right now | Refusing a customer mid-task, per agentic browsers | |
| Training | Nothing directly; corpus collection | None in visibility terms; a values and licensing call | |
| Mixed-use, undeclared | Depends entirely on the request | Unknown, which is the problem; may include search |
The bottom row is where the September change bites. A crawler that will not say what it is doing on a given request cannot be selectively permitted, so it gets an all-or-nothing treatment. That is a reasonable rule for an infrastructure provider to write and an uncomfortable one for a site owner who benefits from some of what that crawler does.
The Wider Shift This Belongs To
The web is moving from a single robots.txt request toward purpose-declared, enforced, and increasingly priced access. This date is one visible step in that, and the direction has been consistent for two years.
For most small businesses the practical implication is modest and worth stating plainly: you probably want search and agent crawlers allowed, you may or may not care about training, and the risk is accidental over-blocking rather than deliberate under-blocking. The publishers with a genuine licensing interest are a different population with a different calculation.
It also raises the value of knowing what actually reaches your site. As enforcement moves to the edge and defaults change without your involvement, the only reliable record of which crawlers were served and which were refused is your own access log, which is the argument made throughout log file analysis and the reason a quarterly check is worth scheduling rather than reacting to announcements.
Questions About the September Change
- What is changing on 15 September 2026?
Cloudflare announced on 1 July 2026 that AI companies have until 15 September to separate the crawlers they use for search from those used for training and agents. After that date, on pages carrying advertising, search crawlers stay allowed by default while training and agent crawlers are blocked by default.
- Does this affect my site if I am not a publisher?
It affects you if your site sits behind Cloudflare and shows advertising, and it affects all new domains onboarding after the date. Free-tier sites and new sign-ups get the defaults applied automatically. Existing paying customers can override them from the dashboard, which is the difference worth knowing.
- Could this accidentally block Googlebot?
A blunt training rule can accidentally block Googlebot, so verify the effect on your own configuration. Reporting on the change describes multi-purpose crawlers as facing the strictest applicable rule, meaning a rule that blocks training can take the whole crawler with it. Googlebot serves ordinary search and AI features from the same crawl.
- What should I actually do before the date?
Establish three things: whether your site is behind Cloudflare, whether the pages in question carry advertising, and which crawler categories your current configuration permits. Then decide your policy deliberately rather than letting a default decide it, and verify the result in your logs rather than assuming the dashboard reflects reality.
- Is blocking training crawlers a bad idea?
It is a legitimate position and it costs you nothing in visibility, because training crawlers do not send traffic. The mistake is not blocking training; it is applying a rule so broad that it also refuses the crawlers that produce citations and clicks, which is a different decision people frequently make by accident.
