Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.
Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.
Most Sites Do Not Have a Crawl-Budget Problem
Google’s current advanced guidance is aimed primarily at sites with roughly one million or more frequently changing URLs, sites with 10,000 or more URLs changing daily, or properties with a large share of URLs stuck at Discovered: currently not indexed.
Those are rough indicators, not eligibility thresholds. A 600-page service site whose new pages are crawled the day they publish should not spend a quarter tuning crawl allocation. An inventory marketplace that creates millions of filter states while new listings wait a week has a credible problem.
Begin with the symptom: priority URLs are not fetched or refreshed within the business window, and evidence suggests Google is spending substantial work elsewhere or the server cannot accept more crawling. “Not indexed” alone is not proof; crawling and index selection are separate systems.
Crawl Budget Is Capacity Plus Demand
| Element | What it means | What changes it |
|---|---|---|
| Crawl capacity limit | How much concurrent crawler work the host can tolerate | Latency, response stability, 5xx/429 rates, Google resources |
| Crawl demand | How much Google wants to fetch or refresh | Inventory, updates, popularity, staleness, quality, relevance |
| Crawl budget | The set of URLs Google can and wants to crawl | The lower practical boundary across capacity and demand |
Google calculates this in the context of a hostname. Subdomains such as www.example.com and docs.example.com are treated as separate sites for budget purposes, while different Google crawlers share the hostname’s capacity. Shopping or Ads demand can therefore coexist with Search crawling on the same infrastructure.
Prove the Constraint With Four Evidence Sets
- Canonical inventory: the URLs the business expects Google to crawl, their template, update rate, and priority.
- Server logs: verified crawler requests, response status, latency, bytes, template, and timestamp.
- Search Console: Crawl Stats host status, response codes, file types, purpose, and Page indexing examples.
- Publication timing: created, internally linked, sitemap lastmod, first crawl, recrawl, and indexed-state dates.
Compare the desired inventory with actual requests. Calculate crawl concentration by template and parameter signature, not just total Googlebot hits. A rising total can hide the fact that most work is spent on duplicate facets, calendars, or tracking variants.
Verify that log requests are genuinely from Google before making infrastructure decisions. Then sample priority and waste classes to confirm what the aggregate suggests.
Control the URL Inventory Before Tuning the Server
Google identifies perceived inventory as the factor site owners can influence most directly. If the crawler knows about endless duplicates, it must decide whether and when to revisit them. Stop generating unnecessary URL forms, consolidate duplicates, and prevent internal modules from continuously rediscovering waste.
| Waste class | Primary control |
|---|---|
| Equivalent duplicate URLs | Normalize generation, redirect alternatives, align canonicals |
| Unwanted facet and sort spaces | Contain link generation and use durable robots rules |
| Permanently removed URLs | Return 404 or 410 and remove internal references |
| Soft 404s | Return the truthful status or build a substantive destination |
| Redirect chains | Point every legacy URL directly to the final target |
Use the faceted-navigation playbook for combinatorial filters. Blocking with robots.txt is appropriate for URL sets that should not be crawled at all; noindex still requires a request and is not a crawl-efficiency substitute.
Keep the Wanted Inventory Easy to Refresh
A clean inventory needs positive discovery paths. Link new and updated pages from stable hubs, keep XML sitemaps restricted to canonical intended URLs, and publish accurate lastmodvalues only for meaningful changes. Google reads sitemaps regularly and uses them as a discovery signal.
Do not update every lastmod on every deployment. That destroys the change signal. Segment sitemaps by useful template or freshness cohort when it improves monitoring, then compare submitted URLs with crawl timing and index outcomes using the XML sitemap guide.
Raise Capacity With Healthy, Cacheable Responses
Google adjusts capacity based partly on stable response times and availability. Rising latency, timeouts, 5xx errors, and 429 rate limits reduce the amount of concurrent crawl work the host receives. Faster, reliable pages can allow more useful content to be fetched.
Support conditional requests and 304 Not Modified responses when content has not changed. Google can reuse the cached version, saving server bandwidth and processing. Cache public assets and expensive page fragments appropriately, and make deploys resilient instead of producing waves of intermittent errors.
Use the HTTP status-code guide to distinguish a removed page from overload. Do not serve errors for days to force a lower rate; persistent server failures can eventually remove URLs from Search.
Change One Waste Class at a Time
Crawl behavior adapts, so a sitewide release that simultaneously rewrites parameters, robots rules, canonicals, sitemaps, and internal links becomes difficult to interpret. Start with the largest verified waste class on one controlled directory. Preserve baseline log volume, response time, priority-URL recrawl lag, and indexing states, then compare the same measures after Google has revisited the system.
Watch for product regressions as closely as crawl changes. A filter containment rule can remove useful browsing paths, a redirect can collapse distinct inventory, and an inaccurate lastmod can train crawlers to ignore sitemap updates. Keep a rollback condition for user behavior, search discovery, and server health.
If total crawling falls while important discovery improves, the release worked. If waste falls but priority behavior is unchanged, the hostname may not have been capacity constrained; move to the next business problem instead of chasing a higher request count.
Measure the Outcome, Not the Number of Crawls
More requests are not automatically better. A well-controlled site can receive fewer total requests while discovering and refreshing important content sooner. Track coverage of the intended inventory, time to first crawl, time to meaningful recrawl, response health, and concentration on useful templates.
- Did crawler share move away from waste classes?
- Did first-crawl latency improve for new priority URLs?
- Did updated pages receive timely recrawls?
- Did server error and high-latency rates fall?
- Did eligible pages progress beyond discovery into useful visibility?
If crawling improved but indexing did not, stop optimizing capacity and investigate canonicalization, rendering, duplication, and selection using the technical indexing diagnostics manual.
Questions About Crawl Budget SEO
- Does a small website need to optimize crawl budget?
Usually not. Google’s July 2026 guidance says sites without many rapidly changing pages generally need only an accurate sitemap and regular Page indexing checks. If important pages are discovered and recrawled promptly, work on content, internal paths, and user value instead of manufacturing a crawl-budget project.
- Can I ask Google for more crawl budget?
There is no request for an increase. Google says capacity can improve when the server has more resources and remains healthy, while demand reflects inventory, popularity, staleness, quality, relevance, and other product-specific factors. Make the site efficient and the useful inventory clear.
- Will blocking low-value URLs move crawling to important pages?
Only when the hostname is already constrained by its crawl capacity limit. Google explicitly warns that newly available capacity is not automatically reassigned otherwise. Block permanent unwanted crawl spaces because they are wasteful, not because every disallowed URL buys one crawl somewhere else.
- Should I use noindex to save crawl budget?
Not for a URL set you never want crawled. Google must request a page to see noindex, so that continues consuming crawl work. For permanent unwanted crawl spaces, manage URL generation or use robots.txt where appropriate. Use noindex when Google may crawl the page but it must not appear in Search.
Primary Sources Reviewed
- Google Crawling Infrastructure: Crawl-budget management
- Google Search Console: Crawl Stats report
- Google Crawling Infrastructure: Faceted-navigation crawling
- Google Crawling Infrastructure: HTTP status-code handling
- Google Search Central: Sitemap overview
Reviewed July 28, 2026, including Google’s July 22 crawl-budget revision. Thresholds are rough and crawler behavior is adaptive; use production logs and business freshness requirements.
