Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.
Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.
Declare an Indexing Incident Before You Touch the Site
An indexing incident needs a start time, an affected URL set, a known-good comparison set, an observable symptom, and a business consequence. “Google is not indexing us” is too broad to diagnose; “new service pages from one template are absent after the 22 July deploy while older service pages remain indexed” is workable.
Begin with the blast radius. Group URLs by template, directory, publication window, canonical pattern, sitemap, response code, and internal-link source. A single missing page calls for URL Inspection. A sharp change across one template calls for a release diff. A property-wide indexed-page drop calls for server, robots, noindex, canonical, and migration checks before individual content review.
Record the last known healthy date and every material change around it: CMS release, framework upgrade, domain or protocol migration, navigation change, CDN rule, robots edit, sitemap generation change, bulk publish, or content consolidation. Search Console tells you what Google observed. Your release history tells you what the site did.
Symptom · first observed · affected patterns · known-good patterns · recent changes · business journeys at risk · incident owner · next update time.
The Six-State Indexing Pipeline
Diagnose the earliest state that fails. Later optimization cannot compensate for an earlier broken dependency.
| State | Required condition | Primary evidence |
|---|---|---|
| 1 · Discovery | Google knows a crawlable URL exists | Internal href, submitted sitemap, discovered URL record |
| 2 · Fetch | Googlebot receives a usable response | Status, headers, robots decision, server/CDN logs |
| 3 · Render | Primary content and links are available to processing | URL Inspection screenshot and rendered HTML |
| 4 · Eligibility | No noindex or blocking state prevents selection | Meta robots, X-Robots-Tag, authentication, policy reports |
| 5 · Canonicalization | The intended representative wins duplicate selection | User-declared and Google-selected canonical |
| 6 · Index selection | Google retains the canonical page in its index | Indexed URL Inspection and Page indexing reason |
Reporting is a seventh operational state, not part of index eligibility. Search Console reports can lag behind a live fix or a newly processed URL. Do not reopen code because an aggregate chart has not caught up with a current inspected URL.
Collect a Small Evidence Pack
- Sample by pattern, not convenience.
Inspect two or three affected URLs and at least one healthy URL from the same template or directory. A homepage-only test proves almost nothing about a product or article template.
- Save both indexed and live inspection states.
Record last crawl, fetch result, robots eligibility, rendered output, user-declared canonical, Google-selected canonical, and referring sitemap.
- Fetch the URL outside the application session.
Confirm status, redirect chain, final URL, response headers, and primary HTML without cookies, authentication, or a warm client router. Use the HTTP status-code field note to interpret live, moved, missing, and overloaded states.
- Compare navigation and sitemap signals.
The intended canonical should be linked in context and listed accurately. A sitemap is a hint; it does not repair an orphaned or contradictory page.
- Diff the responsible template.
Check recent changes to head metadata, HTTP headers, rendering, pagination, URL generation, redirects, and navigation. When the response and rendered DOM differ, use the JavaScript SEO test matrix to isolate the failing boundary.
Google’s Page indexing documentation recommends fixing only states that conflict with your intent. “Not indexed” includes legitimate duplicates, redirects, deliberate noindex pages, and filter combinations as well as errors. The evidence pack exists to distinguish them.
When delayed discovery affects a large, rapidly changing inventory, test capacity and demand with the crawl-budget field note. Do not turn a single excluded page into a sitewide crawl project.
Read Common Search Console States Literally
Discovered · currently not indexed
Google knows the URL but has not recorded a crawl that led to indexing. Verify the URL is not being generated at unbounded scale, is linked from a useful path, appears in the intended sitemap, and is served reliably. On a new site or large URL set, time and crawl prioritization can be part of the state.
Crawled · currently not indexed
The fetch occurred. Stop treating crawl access as the only problem. Compare the page’s visible job with indexed pages, near-duplicates, soft-404 behavior, canonical signals, and the useful information available in the rendered result. This status does not expose a private “quality score,” so make bounded, evidence-backed changes rather than rewriting every affected page.
Soft 404
Google believes the response behaves like a missing or empty page despite a non-404 status. A genuinely missing page should return an actual 404 or 410. A valid page should provide the substantial primary content its title, template, and navigation promise. Empty category states, unavailable inventory, and placeholder location pages are frequent triggers.
Page with redirect
The source URL is not intended for indexing; the target may or may not be indexed. Inspect the final destination, remove redirect loops or long chains, and update important internal links and sitemap entries to the final canonical URL.
Blocked by robots.txt or marked noindex
Decide intent first. If exclusion is correct, close the finding. If the page should be indexed, remove the responsible rule and verify the live response. Remember that blocking crawling can prevent Google from seeing a page-level noindex directive. Use the robots.txt, noindex, and canonical comparison when the intended control is unclear.
Resolve Canonical Conflicts as a Signal System
A canonical link element is a strong hint, not a command. Google selects a representative from pages it considers duplicates. Make the preferred URL the obvious choice across redirects, internal links, sitemap membership, page content, hreflang, and canonical markup.
When Google chooses a different canonical, inspect three pages together: the tested URL, your declared canonical, and Google’s selected canonical. If the declared target is not substantially similar, it is not a valid duplicate relationship. If two pages have the same job, consolidate. If they have distinct jobs, strengthen the difference in primary content, titles, internal context, and navigation.
- Use one canonical form for protocol, host, case, slash, and parameters.
- Link internally to canonical destinations rather than alternates.
- List canonical, indexable 200 URLs in the sitemap.
- Do not canonicalize unrelated thin pages to a stronger page.
- Use permanent redirects when a URL has actually moved or merged.
For a full migration workflow, use the website redesign SEO prevention playbook. Migration incidents need inventory and redirect validation beyond a single-page canonical check.
Fix the Earliest Broken State, Then Validate in Two Phases
- Make one bounded change to the affected template or URL set.
- Test status, headers, directives, canonical, rendered content, and internal links in the deployed environment.
- Run live URL Inspection on representative affected and control URLs.
- Request indexing only for a small urgent sample after the fix; do not submit thousands of unchanged URLs.
- Wait for Google to recrawl and process the set. Track last-crawl dates so reporting lag is not mistaken for continued failure.
- Confirm recovery across the pattern, not just the URL used for the request.
| Validation phase | What it can prove | What it cannot prove |
|---|---|---|
| Live technical validation | The current response is fetchable, renderable, and eligible | That Google has selected or indexed it |
| Processed index validation | Google’s latest canonical and indexing decision | Future ranking, traffic, or permanence |
| Pattern monitoring | Whether the incident class is recovering | That every URL should be indexed |
Build Indexing Safety Into Releases
- Add automated checks for unexpected noindex, robots headers, canonical host changes, redirect chains, and sitemap drift.
- Test representative templates in preview and immediately after production deployment.
- Keep a URL inventory and redirect map for redesigns, platform moves, and consolidation projects.
- Make navigation links real anchors with href attributes and ensure important content is present in rendered HTML.
- Alert on sustained 5xx responses, crawler-specific 403s, and sudden indexed-page changes by directory.
- Record intentional exclusion policies so future operators do not “fix” healthy duplicate or private states.
If the entire site is simply absent rather than experiencing a pattern-level incident, start with the simpler ten-check missing-site diagnostic. This manual is for the point where the problem needs technical classification and release ownership.
Questions About Indexing Diagnostics
- Why is a page crawled but not indexed?
The status means Google fetched the URL but did not currently retain it in the index. It does not identify one technical fault. Compare the page with Google’s selected canonical, check whether its job is distinct, verify useful visible content and internal paths, and inspect whether a recent template or site change affected the URL set.
- Should every page on a website be indexed?
No. Google’s Page indexing documentation explicitly says 100% coverage is not the goal. Canonical versions of important pages should be indexed; duplicates, redirects, filter combinations, private pages, and deliberately excluded states often should not be.
- Does requesting indexing fix an indexing problem?
No. It can ask Google to recrawl an important changed URL, but it does not override robots rules, noindex, server failures, canonicalization, duplication, or index-selection decisions. Fix the observed cause first, then request indexing for a small urgent set if appropriate.
- Why does URL Inspection disagree with the Page indexing report?
The reports can reflect different crawl times. URL Inspection’s live test shows the current fetchable page, while the indexed view and Page indexing report describe Google’s last processed state. The live test also does not test every condition, including final duplicate and canonical selection.
