Check this page with an assistantOpens a chat asking it to summarise this article and name the evidence behind each claim.
Claude opens with the prompt on your clipboard: Anthropic does not support prefilled prompts on the web, and we would rather copy it than ship a button that drops it.
The Direct Answer: No Evidence It Does Anything Yet
llms.txt is a proposed standard, not an adopted one. No major AI provider has committed to reading it, Google's representatives have said its AI systems do not use it, and crawler-log studies show AI bots overwhelmingly never request the file. Adding it is harmless; expecting visibility from it is unsupported by every measurable signal as of mid-2026.
That answer deserves precision, because the question usually arrives with pressure attached. A plugin added the file automatically, a consultant listed it as a deliverable, or a competitor's site has one and yours does not. The honest framing is that llms.txt sits in the same category as hundreds of past web proposals: a reasonable idea waiting for the platforms it depends on to say yes, and so far none of them have.
This review walks the evidence in order: where the file came from, what the engines have actually said, what server logs show, and how it compares to the files that verifiably matter. At the end is the part vendors selling llms.txt services skip: what to do with the same hour that has evidence behind it.
Where the File Came From, and What It Was Meant to Do
llms.txt was proposed on September 3, 2024 by Jeremy Howard, co-founder of Answer.AI, as a markdown file served at your site's root that summarizes your most important content and links to clean markdown versions of key pages. The pitch: language models have limited context windows, and parsing HTML full of navigation, scripts, and ads is wasteful, so give them a curated map instead.
The idea is not silly. Howard's original proposal, published on answer.ai and llmstxt.org, was aimed largely at developer documentation, where an AI coding assistant genuinely benefits from a clean index of API references. Documentation platform Mintlify rolled out llms.txt support across its hosted docs sites in November 2024, which put the file on thousands of sites, including well-known developer tools, almost overnight.
From there the file migrated out of its niche. SEO plugins began generating it for ordinary business sites, and agencies began listing it as an AI-readiness deliverable. The gap worth understanding is not in the format, which is fine. It is between proposal and adoption. robots.txt works because every major crawler agreed to honor it. llms.txt has the spec, the plugins, and the invoices, but not the agreement: the consumers it was designed for never signed on.
What the Engines Actually Say and Do
Google's John Mueller has compared llms.txt to the keywords meta tag, a signal search engines abandoned because it was self-declared and unreliable, and noted that no AI service has said it uses the file, with server logs showing they do not even check for it. Google's documentation states plainly that no special machine-readable files are needed for its search or AI features.
Those statements, reported by Search Engine Journal and Search Engine Land in 2025, are the closest thing to an official verdict this file has received. On the other side of the ledger, no major provider has made a commitment: OpenAI, Anthropic, Google, and Perplexity all document their crawlers' behavior, and none has announced that its systems read llms.txt. Some AI companies publish the file on their own documentation sites, which says their docs platform supports emitting it, not that their crawlers consume it.
The log data settles what the statements imply. An Ahrefs study published in June 2026, covering server logs from 137,000 domains, found that 97% of llms.txt files received zero requests during May 2026. Of the requests the remaining files did get, AI retrieval bots accounted for just 1.1%, while SEO audit tools were the single largest requester at 21.7%. The file's main confirmed audience, in other words, is the tools that recommend creating it. An earlier SE Ranking analysis of 300,000 domains found no connection between having an llms.txt file and how often a site was cited by AI tools.
None of this reflects a lack of supply. Adoption tracking by Originality.ai counted roughly 36,000 live llms.txt files by May 2026, an 8.8x increase in twelve months, and SE Ranking's 300,000-domain sample put adoption at just over 10%. Tens of thousands of site owners have shipped the file; the missing half of the market is anyone reading it. That split, growing publication against flat consumption, is the clearest single picture of where llms.txt stands: a standard can survive low adoption by websites, but not indifference from the crawlers it was written for.
So why do vendors keep pushing it? Because it is a perfect deliverable: easy to ship, easy to invoice, and impossible for the customer to measure. A file no crawler requests can never visibly fail, which makes it safer to sell than work whose results show up in a report. It slots neatly into the AEO and GEO packaging wave, where old fundamentals get new acronyms and one new artifact to justify the line item.
llms.txt vs the Files That Do Matter
| File | Who reads it | Verdict |
|---|---|---|
| robots.txt | Every major crawler, including AI bots | Essential; the working control over crawler access |
| XML sitemap | Search engines | Essential for discovery on sites of any real size |
| IndexNow pushes | Bing and partner engines | Useful; near-instant indexing on the index ChatGPT reads from |
| Structured data in pages | Search and AI systems, via the rendered page | Useful where it labels real, visible content |
| llms.txt | No confirmed major consumer as of mid-2026 | Harmless, unproven; never worth paying for |
The pattern in that table is not about format elegance. The boring files won because engines publicly committed to reading them and their logs prove they do. llms.txt has a commitment problem, not a format problem: markdown is a perfectly good way to describe a site, but a map only matters if someone agrees to open it.
What to Do Instead With the Same Hour
The levers with evidence behind them are extractable page structure, entity consistency, and third-party corroboration: the things AI engines actually retrieve and cross-check when deciding whom to cite. An hour spent putting a direct two-sentence answer under each question-shaped heading does more than any file no crawler requests.
The practical order: first confirm your pages are indexed in both Google and Bing, since assistants can only quote what retrieval finds. Then restructure your highest-value pages so each buyer question gets a heading and an immediate answer, with numbers in real HTML tables. Then check that your business name and one-sentence description match everywhere they appear, because assistants cross-reference sources before naming anyone confidently. The full sequence is in our guide to getting recommended by ChatGPT, and the measurement behind each step is collected in our AI SEO statistics roundup, every figure with a named source.
If a plugin already added llms.txt to your site, leave it. It costs nothing to keep, and deleting it buys nothing either. The decision that matters is where the next hour goes, and the evidence says it goes into the pages themselves.
Questions People Ask About llms.txt
- What is llms.txt?
llms.txt is a proposed standard from September 2024, authored by Jeremy Howard of Answer.AI, for a markdown file at your site's root that lists your most important pages in a clean, LLM-friendly format. The idea is to give context-limited AI models a curated map instead of forcing them to parse cluttered HTML. It is a proposal, not an adopted standard: no major AI provider has committed to reading it.
- Does Google use llms.txt?
No. Google's John Mueller has said that no AI service has announced it is using llms.txt and that server logs show they do not even check for it, comparing the file to the long-dead keywords meta tag. Google's own documentation states that no special machine-readable files or markup are needed to appear in Google Search, including its AI features.
- Should I add llms.txt to my website?
If it takes ten minutes and costs nothing, it is harmless, and it doubles as a tidy human-readable index of your best pages. As a visibility tactic it is unsupported: Ahrefs found 97% of llms.txt files received zero requests, and SE Ranking found no link between having one and being cited by AI tools. Never pay for llms.txt creation or optimization as a service.
- What actually helps AI tools find and cite my site?
The evidenced levers are the boring ones: pages that are indexed in Google and Bing, question-shaped headings with short extractable answers beneath them, facts in visible HTML tables, one consistent business description across the web, and third-party corroboration on sites assistants retrieve. Engines cite what they can retrieve, verify, and quote, and no single file substitutes for that.
