Does llms.txt Work for AI Search? What the Evidence Shows

Mostly no. Ahrefs analysed 137,210 domains in May 2026 and found that 97% of published llms.txt files received zero requests — no bots, no humans, nothing. Google Search Central states you do not need machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its generative AI features. The genuine use case is agent and coding tools, not search visibility.

At a glance

• The claim: publishing an /llms.txt file makes ChatGPT, Perplexity, Claude and Google AI Overviews understand and cite your site.
• The origin: a proposal by Jeremy Howard of Answer.AI, published 3 September 2024. It has never been ratified by any search or AI company.
• Google’s position: the file is not used by Google Search and “will neither harm nor help your site’s visibility or rankings in Google Search.”
• The log evidence: 97% of valid llms.txt files got zero requests in May 2026 across 137,210 domains (Ahrefs).
• Who does read it: SEO audit tools, tech profiling crawlers, and some coding and browser agents. Documentation sites are the one legitimate beneficiary.
• The verdict: cheap, low-risk, and almost entirely irrelevant to whether an AI engine cites you today. Treat it as a five-minute chore, never as a strategy.

What llms.txt claims to do

The proposal is, on its own terms, reasonable. Jeremy Howard’s llmstxt.org describes it as “a proposal to standardise on using an /llms.txt file to provide information to help agents use a website.” The argument: a language model has a finite context window, and a modern HTML page is mostly navigation, cookie banners and JavaScript. A single Markdown file at the site root, listing the pages that matter with one-line descriptions, saves an agent the work of crawling and stripping all of that.

Note the word “proposal”. It is not a W3C standard, not an IETF RFC, and not something OpenAI, Anthropic, Google or Perplexity have agreed to honour. That distinction gets lost fast. Within about a year of publication, llms.txt had been rewritten by a large part of the AEO and GEO vendor market as a ranking factor for AI search — which is a claim the original proposal never made. The proposal is about agents reading documentation efficiently; the sales pitch is about getting cited in ChatGPT. Only one of those has evidence behind it.

What Google actually says

Google’s guide to optimising for generative AI features on Google Search, last updated 10 July 2026, is unambiguous: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” The same page says such files “will neither harm nor help your site’s visibility or rankings in Google Search.”

Google’s John Mueller has been blunter. In an r/TechSEO thread in April 2025, reported by Search Engine Journal, he wrote: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it).” He added that it is “comparable to the keywords meta tag”, being “what a site-owner claims their site is about”. He returned to it in June 2026, again via Search Engine Journal, calling the question “purely speculative for now” because “the file has existed for years, yet none of the AI systems use it”.

The keywords-meta-tag comparison is the sharpest part of it. A meta keywords tag was an unverifiable self-description, and engines stopped trusting it the moment it became a lever. An llms.txt file is structurally the same object: an assertion about your site, sitting next to the site itself, which the engine can simply read instead.

There is a wrinkle, and it is the reason the honest answer is “mostly no” rather than a flat no: Google Search says skip it, and Google Chrome does not.

Lighthouse 13.3, released in May 2026, moved an Agentic Browsing category into the default configuration, and it includes an llms.txt audit. The Chrome for Developers documentation says that “without this file, agents may spend more time crawling the site to understand its high-level structure and primary content.” It also says that if the file is absent and the server returns a 404, “the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.”

Read together, those positions are coherent rather than contradictory. Search visibility and agentic browsing are separate problems. The Search team is saying the file does nothing for citations; the Chrome team is assessing whether a site is convenient for an agent that has already been sent there. If your revenue depends on an agent completing a task on your site — booking, checking stock, comparing plans — the Lighthouse view matters to you. If your goal is being quoted in an answer, it does not.

What the server logs show

The strongest available evidence is server-log data rather than opinion. In June 2026 Ahrefs published a study by Louise Linehan covering “all 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026.” Roughly 38,000 of those domains — about 28% — served a valid llms.txt file at the root. Of those, the study reports: “Of the ~38,000 domains with a valid file, 97% saw no requests for it whatsoever in May. No bots. No humans. Nothing.”

The composition of the remaining 3% is more instructive than the headline. That 3% is about 1,100 domains and roughly 22,000 requests, so this is a small pool, and Ahrefs is explicit that a fetch is not proof anything was read: every figure in the study is “a ceiling on actual llms.txt consumption.”

Of the requests those files did receive, no single AI bot category made the top four. SEO audit tools led at 21.7%, ahead of other and unidentified bots at 14.9%, general web crawlers at 13.1% and tech profiling tools at 11.6%, and Ahrefs notes those four “all send more requests than any one AI bot.” The 21.7% needs one disclosure the study makes itself: Ahrefs’ own crawlers account for 10.6 of those 21.7 points, leaving third-party SEO audit tools on 11.1%. Group the four AI categories together and AI bots are the largest single bucket at 19.5% — but that is mostly agents (10.5%) and training crawlers (5.3%). AI retrieval bots, the class that actually fetches a page to answer a live user query in an assistant, made up 1.1% of all requests to these files: 233 fetches across the entire study.

Ahrefs’ own framing of that: “A whole ecosystem has formed around auditing, scoring, validating, and studying the llms.txt standard, before we’ve even established whether any major AI platform actually reads it.” The tools that catalogue and score the file — GEO/AEO scanners, llms.txt validators and research crawlers — sent 12.1% of requests between them, more than three times the combined traffic from AI retrieval bots and AI assistants.

Set the vendor pitch against the record and the gap is easy to see:

Who actually reads llms.txt

Developer documentation is the one context where llms.txt has real traction and a real payoff. Anthropic, Cursor and Cloudflare all serve one on their docs domains, and Mintlify, which hosts documentation for much of the developer-tools market, generates them for hosted sites. That is not a coincidence: when a developer asks a coding agent to integrate your API, the agent needs a compact, accurate map of your docs, and a Markdown index is genuinely cheaper to consume than 400 rendered HTML pages.

The pattern: llms.txt helps an agent that has already decided to work with your site. It does not help an engine decide to mention your site to someone who has never heard of you. If you sell software with an API, publish one and keep it current. If you run a services business, a clinic, a brokerage or a franchise network, the file will be read by audit crawlers and almost nothing else.

What actually correlates with being cited

The follow-up question is what to do instead, and the evidence points at unglamorous work. Ahrefs’ January 2026 analysis of AI Overview citations found “near-zero correlation (Spearman ~0.04) between word count and AI citations” — longer pages are not better pages. It also reports that pages ranking across a query’s fan-out variants are “161% more likely to be cited in the final AI Overview than pages that only rank for the primary search term.” The strongest measured signals sit off-site: a separate Ahrefs study of 75,000 brands put brand web mentions top on Spearman correlation with AI Overview brand visibility at 0.664, brand anchors at 0.527 and branded search volume at 0.392, against 0.218 for backlinks. Ahrefs is careful to say correlation is not causation and that every factor it measured was moderate to weak, so treat those as a direction of travel, not a formula. We covered the broader dataset picture in our summary of what actually drives AI citations, and the decoupling of citations from classic rankings in AI Overview citations no longer tracking rankings.

On structured data, Google is measured rather than dismissive: “Structured data isn’t required for generative AI search, and there’s no special schema.org markup you need to add. However, it’s a good idea to continue using it as part of your overall SEO strategy.” That is the correct weighting. Ship FAQPage and Article schema because it makes extraction unambiguous and earns rich results, not because a vendor called it a citation lever.

What we do on our own pages, and what we would do on yours, is narrower than most AEO packages and considerably more boring:

• An answer paragraph in the first 50 words, with a hard number and no warm-up sentence. An engine that reads 300 words before finding a quotable claim usually quotes someone else.
• A comparison table. Tables survive extraction intact. A paragraph comparing four options does not.
• Named-entity specificity. Real studies, real dates, real place names. Generic copy gives a model nothing to disambiguate against, which is the argument behind entity consistency work.
• FAQ schema that matches the visible page verbatim. Mismatched schema is worse than none.
• Something quotable nobody else will write. Google’s guidance asks for “unique expert or experienced takes that go beyond common knowledge” rather than commodity content. A page that states a limitation the rest of the market is selling around is, mechanically, the most citable page on the topic. That is why this one exists.

To check any of this against your own results rather than taking it on faith, see how to track your own AI search visibility without buying a tool, and what separates pages ChatGPT cites from pages it ignores.

Should you publish one anyway?

Yes, if it costs you fifteen minutes. No, if anyone is charging you for it as an AEO deliverable.

The file is close to harmless, with one caveat. Google has said explicitly that it will not hurt your rankings, and it takes one text file, a handful of Markdown links, and no upkeep beyond accuracy. We publish one at leadsnow.ai/llms.txt for exactly that reason. The caveat is that agents are built to trust the file: Ahrefs found a crawler identifying itself as prompt-injection-survey systematically probing llms.txt files, so treat yours like code — version-controlled, access-restricted, plain links and descriptions only, nothing instruction-shaped. What we do not do is tell clients it is why an engine cited them, because our own logs say what Ahrefs’ logs say.

The real risk is opportunity cost. Ahrefs’ verdict was: “the cons outweigh the pros right now. If you want to show up in AI search, there are more reliable ways to improve your visibility than this file.” A stale llms.txt pointing at deleted pages is worse than none, and an hour spent generating and validating one is an hour not spent on the answer paragraph, the comparison table, or the earned mention the correlation data actually supports.

Why the industry keeps selling it

The commercial mechanics explain the noise. AEO and GEO services are sold into a buying cycle where the client cannot verify the deliverable: nobody can point at a citation in ChatGPT and prove which action caused it, the way you can point at a ranking and a backlink. That creates demand for deliverables that are visible, fast and checkable. llms.txt fits perfectly. It is trivially produced, obviously present, and it sits on audit-tool checklists — which is exactly what the Ahrefs bot data shows, with the tools scoring and cataloguing the file sending several times more traffic to it than the AI search and assistant bots it was written for. A checklist item that costs nothing to make and cannot be falsified by the buyer is a durable product regardless of whether it works.

We sit on the other side of that problem. LeadsNow AI runs on a pay-per-result model — you pay on booked, qualified appointments, not on retainers, seats or deliverables — which removes the incentive to ship checklist items. Across 50,769+ AI-booked sales appointments since 2017 and 1M+ leads generated, none of it traces to a text file at a domain root. If a page of ours gets cited, it is because it said something specific, early, that was worth quoting. To talk through what that looks like for your site, book a call.

Frequently asked questions

Does llms.txt help you rank in Google AI Overviews?

No. Google Search Central states that you do not need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search including its generative AI capabilities, because Google Search does not use them. The same guidance, last updated on 10 July 2026, says such files will neither harm nor help your visibility or rankings.

Do ChatGPT and Perplexity read llms.txt?

Almost never, on the available log evidence. Ahrefs studied 137,210 domains and found that 97% of valid llms.txt files received no requests at all during May 2026. Among the requests that did arrive, AI retrieval bots, the class that fetches pages to answer a live user query, made up 1.1% of all requests, or 233 fetches. AI bots of every kind combined made up 19.5%, most of it coding agents and training crawlers rather than search.

If it does nothing, why does Chrome Lighthouse check for it?

Because Lighthouse is measuring a different thing. The llms.txt audit sits in the Agentic Browsing category, which assesses how easily an automated agent can operate a site it has already been sent to, rather than whether the site gets cited. Chrome for Developers notes that a missing file returns Not Applicable rather than a failure, because providing it is optional at the moment.

Is llms.txt an official web standard?

No. llmstxt.org describes it as a proposal to standardise on using an /llms.txt file to provide information to help agents use a website. It was published by Jeremy Howard of Answer.AI on 3 September 2024 and has not been adopted by any standards body, search engine or major AI provider.

Should I delete the llms.txt file I already have?

No. Google has confirmed it causes no harm, and coding and browser agents do occasionally read it. Keep it, keep it accurate, and stop treating it as a growth lever. A file that points at pages you have since deleted is the only version that actively costs you anything.

What should we do instead if we want AI engines to cite us?

Put a quotable, factual answer in the first 50 words, use tables for anything comparative, name real entities and dates rather than writing generically, keep FAQ schema matching the visible text, and invest in earned mentions off your own site. Ahrefs measured near-zero correlation between word count and AI citations, and in a separate study of 75,000 brands found brand web mentions to be the top correlating factor with AI Overview brand visibility at 0.664, ahead of brand anchors at 0.527. Correlation is not causation, but length is clearly not the lever and off-site reputation may well be.