Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.
Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.
Everything Else Is a Simulation or a Summary
A third-party crawler tells you what a crawler would find if it behaved like Googlebot. Search Console crawl stats tell you aggregated totals. Neither tells you that Googlebot requested a specific parameterised URL nine hundred times last month while never requesting your new service page. Only the access log does.
That distinction used to matter mainly for large sites with crawl budget problems. It now matters more broadly, because the population of bots hitting a typical site has fractured. Googlebot and Bingbot crawl for their indexes. Training crawlers collect content. A separate class of retrieval bots fetches pages live to answer questions inside assistants. Agent traffic arrives looking like an ordinary browser.
Those categories have completely different value to you, and every reporting surface except your own logs either ignores them or lumps them together. If you have ever tried to decide which AI crawlers to allow, you have run into this: the decision requires knowing what each one costs you and what it returns, and that data exists in exactly one place.
You can name the top five bots by request volume on your site and say what each one is for.
Getting Logs You Can Trust
Ask for raw access logs covering thirty days minimum, from every layer that serves traffic. The most common analysis failure is not technique, it is analysing a partial log and drawing confident conclusions from it.
- Ask for the CDN logs too.
If a CDN sits in front of your origin, requests it serves from cache may never reach your server. Origin-only logs will show a fraction of crawler activity and make you think bots visit far less than they do.
- Confirm every server is included.
Behind a load balancer, one server's logs are a sample of unknown size. Either aggregate all of them or know the sampling factor, and state it wherever you report a number.
- Check what fields you have.
You need at minimum timestamp, requested URL, status code, user agent, and client IP. Without the IP you cannot verify bot identity, which makes the whole exercise unreliable.
- Handle it as personal data.
Access logs contain IP addresses, which are personal data in many jurisdictions. Keep them in an appropriate place, apply your retention policy, and do not email a raw log to a consultant without thinking about it.
Verify Identity Before You Believe Anything
A user agent string is self-declared. Anyone can send requests claiming to be Googlebot, and plenty of scrapers do. Google documents reverse DNS verification and publishes IP ranges, and skipping that step means your crawl analysis includes traffic that was never Google.
This is not a theoretical concern. Impersonation is common enough that unverified analysis routinely overstates Googlebot activity, and the overstatement is not evenly distributed: scrapers tend to hammer specific sections, which is exactly the pattern you would otherwise interpret as Google finding those sections important.
The verification is mechanical. Reverse DNS lookup on the IP, confirm it resolves to the expected Google domain, then forward lookup to confirm it maps back to the same IP. Google also publishes its crawler IP ranges as a file you can match against, which is faster at volume. Do the same for other major crawlers, all of which document equivalent verification methods.
The Six Questions Worth Asking
Logs will happily consume a week. These six produce decisions, and everything else is curiosity. Answer them in order and stop when you have found something worth acting on.
| Question | What you are looking for | Action if it is bad |
|---|---|---|
| Which URLs get the most requests? | Whether crawl concentrates on pages that matter | Internal linking and structure, per site architecture |
| What is never requested? | Pages the crawler has not reached | Discovery: sitemap, internal links, orphan pages |
| What status codes are returned? | Errors and redirect chains consuming requests | Fix at source, per status codes |
| How much goes to parameters and filters? | Combinatorial URL explosion | Faceted navigation controls |
| How quickly are new pages requested? | Discovery latency | Sitemap freshness and linking from crawled pages |
| Which bots, in what proportion? | The population and its cost | The access policy in the crawler decision guide |
The single most useful output is a crawl frequency distribution: every URL sorted by how often it was requested over the window. It shows immediately whether attention is going to your commercial pages or to pagination, and it is a two-line command on a raw log before any tool is involved.
The AI Bot Ledger
Group requests by verified bot, then set that against referrals from the same source. This produces the crawl-to-refer ratio: how many times a bot took content for every visitor it sent back. It is the only evidence-based way to decide crawler access.
The pattern that usually emerges is stark. Training crawlers appear in volume and return nothing by design, because they are not a referral mechanism. Retrieval bots appear less and correspond to citations that can send people. Googlebot and Bingbot underpin everything and are not a decision at all. Seeing those three groups separated on your own site converts an ideological argument into an arithmetic one.
Pair it with referral data that has been corrected for the attribution gap, or your ratios will overstate the case for blocking. A large share of assistant referrals arrive without a referrer and land in Direct, so uncorrected analytics undercounts exactly the side of the ratio that argues for allowing a bot. The fix is in why your AI traffic says Direct, and it should be done before, not after, you draw conclusions.
Turning Findings Into Work
Logs produce observations, not priorities. Convert each finding into a claim about business impact before it enters a queue, because most log findings are technically interesting and commercially irrelevant.
The test is simple: if this were fixed, what would change? Googlebot spending a third of its requests on filter combinations matters if new products take weeks to appear, and does not matter if everything is indexed within a day. A five hundred error appearing occasionally matters if it hits a template, and does not if it hit one URL during a deploy.
Then record what you changed and when, so the next log pull is a comparison rather than a fresh survey. That change log is the same discipline argued for in SEO split testing, and logs are one of the few places where a before-and-after is genuinely informative, because crawler behaviour responds to structural changes much faster than rankings do.
Questions People Ask About Log File Analysis
- What is log file analysis in SEO?
Reading your web server's access logs to see exactly which URLs each bot requested, when, and what status code it received. It is the only method that reports what crawlers actually did rather than what a third-party crawler simulates or what a summary report aggregates.
- Why not just use Search Console's crawl stats?
Crawl stats are useful and aggregated. They tell you roughly how much Googlebot crawled and what it encountered, not which specific URLs consumed the budget, and they say nothing at all about non-Google bots. When the question is which URLs are wasting crawl or how a particular AI crawler behaves, only raw logs answer it.
- How do I know a bot claiming to be Googlebot really is?
Verify it rather than trusting the user agent string, which anyone can set. Google documents reverse DNS verification and publishes IP ranges you can match against. Skipping verification means your analysis includes scrapers impersonating Googlebot, which is common enough to distort conclusions about crawl behaviour.
- How much log data do I need?
Thirty days is usually enough to see patterns, and ninety is better for crawl frequency distribution on a large site. What matters more than duration is completeness: logs filtered by a CDN, or covering only one server in a pool, will produce confident conclusions from a partial picture.
- Is this worth doing on a small site?
For crawl budget, no. Crawl budget is not a constraint on a site of a few hundred pages. There is one exception worth the hour: logs are the only way to see which AI crawlers are hitting you and how often, which is the input to deciding what to allow, and that question applies at any size.
