Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.
Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.
Start With the Outcome, Not the Tag
Robots.txt answers “may this crawler fetch the URL?” Noindex answers “may this resource appear in search?” Canonical answers “which URL should represent these duplicates?” They operate at different stages, and none of them is a privacy control.
Search incidents often begin with a reasonable goal implemented at the wrong layer. A team blocks a staging site in robots.txt, then discovers indexed URL-only results. A filter page is both disallowed and noindexed, so the crawler cannot read the noindex. A discontinued product canonicals to an unrelated category when it should have been retained, redirected, or removed.
Write the desired end state before touching configuration: accessible but absent from search, inaccessible to everyone except authorized users, consolidated with a duplicate, permanently moved, temporarily unavailable, or gone. The correct mechanism follows from that sentence.
The Decision Table
| Desired outcome | Primary mechanism | Do not substitute |
|---|---|---|
| Reduce crawling of low-value URL spaces | Robots.txt plus controlled link/URL generation | Noindex alone if crawl volume is the problem |
| Keep a public, accessible page out of results | Robots meta noindex or X-Robots-Tag | Robots.txt, which can leave a URL-only result |
| Choose a representative duplicate URL | Rel=canonical, consistent links, sitemap, or redirect | Noindex as a canonicalization shortcut |
| Protect confidential content | Authentication and authorization | Any crawler directive |
| Move a page permanently | Server-side permanent redirect | Canonical alone while users remain on the old URL |
| Remove content with no successor | 404 or 410 plus removal of internal references | Redirecting everything to the home page |
Robots.txt Controls Fetching, Not Secrecy
A robots.txt file lives at the root of a host and tells compliant crawlers which paths they may access. Google describes its main use as managing crawler traffic and avoiding crawling of unimportant or similar URLs. A disallowed web page can still appear as a URL if Google learns about it from a link.
The file is public. Do not list confidential paths there expecting them to become hidden; the list can advertise their existence. Protect customer records, staging tools, invoices, internal search, and private documents with authentication and correct server authorization.
Scope rules by crawler only when the policy truly differs. Test exact path matching, capitalization, parameters, host, and subdomain. Each subdomain has its own robots.txt scope. A rule onwww.example.com does not automatically governshop.example.com.
Noindex Requires the Crawler to See It
Place noindex in a robots meta tag for HTML or anX-Robots-Tag response header for HTML and non-HTML resources such as PDFs. The URL must remain crawlable long enough for the search engine to fetch and process the directive. If robots.txt blocks the fetch, Google cannot see the noindex instruction.
Noindex is appropriate for public utilities that people can use by direct link but that should not become search destinations: certain account states, internal result pages, thin confirmation pages, or files intentionally excluded from results. It is not a fix for a page that should be deleted, merged, or secured.
Other robots directives control presentation, not basic inclusion.nosnippet, max-snippet, max-image-preview, and data-nosnippet can affect displayed content. Google documents that nosnippet also limits direct use in its AI Search features. Choose these controls only after understanding the visibility tradeoff.
Canonical Is a Preference Among Duplicates
Canonicalization is Google’s process for selecting the representative URL from a duplicate or very similar set. A rel=canonical annotation is a strong signal, not an absolute rule. Redirects are also strong; sitemap inclusion is weaker. Google can choose a different canonical when the content and site signals contradict the annotation.
Use canonical for parameter variants, printer versions, equivalent campaign URLs, syndication cases, and other true duplicates that must remain accessible. Point every member of the set to one crawlable, indexable, successful destination. Put a self-reference on the preferred page, link internally to it, and list it, not its duplicates, in the XML sitemap.
Do not canonicalize substantially different pages because they share a topic. A canonical is not a way to transfer generic “authority” from expired products, old events, or thin city pages. When a page moved, redirect it. When it remains distinct, let it be distinct. When its job ended, use the consolidation decision framework.
Five Conflicts That Break the Intended Outcome
| Conflict | Why it fails | Repair |
|---|---|---|
| Robots disallow + noindex | The crawler cannot read the noindex | Allow crawling until the directive is processed |
| Noindex + canonical to another URL | Exclusion and consolidation goals compete | Choose the actual outcome; usually canonical for duplicates |
| Canonical points to a redirect or error | The representative is not a stable indexable page | Point directly to the final successful destination |
| Sitemap lists duplicate, canonical points elsewhere | Weak and strong signals disagree | List only the preferred canonical URL |
| Internal links favor noncanonical variants | The live graph keeps reinforcing duplicates | Update templates and contextual links to the canonical |
Audit Controls as a URL-State Matrix
Export representative URLs by template and record response status, robots.txt access, meta/X-Robots directives, declared canonical, Google-selected canonical, sitemap membership, and internal-link destination. Add the intended state. The mismatch between intention and live signals is the work queue.
- Sample indexable, noindexed, disallowed, redirected, and error URLs.
- Inspect raw response headers and initial HTML before relying on a crawler label.
- Test the rendered page where JavaScript can alter metadata.
- Use URL Inspection to compare declared and selected canonical state.
- Validate templates after deployment, then wait for recrawl before interpreting results.
For a broader incident workflow, use the technical indexing diagnostics guide. It separates discovery, fetch, rendering, eligibility, canonicalization, and reporting lag.
Questions About Search Controls
- Does robots.txt prevent a page from appearing in Google?
Not reliably. Robots.txt primarily controls whether a crawler may fetch a URL. Google can still know and index a disallowed URL from links or other sources without crawling its content. Use noindex on a crawlable page, authentication, or removal of the URL when the requirement is to keep it out of search.
- Can I use noindex and canonical together?
Avoid using them as competing instructions for duplicate consolidation. Noindex says the page must not appear in Search; canonical expresses a preference to consolidate a duplicate with another URL. Google recommends rel=canonical rather than noindex when choosing a canonical within one site.
- Should canonical URLs also have self-referencing canonicals?
Google recommends a self-referencing canonical on the preferred page. It makes the intended URL explicit and helps protect against accidental parameter, protocol, host, and campaign variants. Internal links and sitemap entries should point to the same preferred URL.
- How do I noindex a PDF?
Use an X-Robots-Tag HTTP response header with a noindex directive because a PDF has no HTML head for a robots meta tag. Google must be allowed to fetch the PDF response to see that header. If the document is private, require authentication instead of relying on a search directive.
Primary Sources Reviewed
- Google Search Central: Introduction to robots.txt
- Google Search Central: Robots meta and X-Robots-Tag specifications
- Google Search Central: What canonicalization means
- Google Search Central: Canonical methods and best practices
- Google Search Central: Redirects and Google Search
Reviewed July 28, 2026. Crawler support differs across search engines; verify directives against each platform that matters to the site.
