llms.txt: who publishes it and who reads it

llms.txt is a Markdown file at the root of a website that tells AI agents what the site is and where its useful pages are. Every page ranking for “llms txt” explains what it is. The question people actually ask — Google’s second “People also ask” box on that search is Is llms.txt actually used? — splits three ways, and the three answers are different. Below: what the file is, what Google says about it in two documents from two teams, how many sites publish it, and who, as far as anyone has measured, reads it.

What the file is, in five lines

The proposal is Jeremy Howard’s, published 3 September 2024 and revised to version 2 on 10 August 2026. Its own one-line description is now:

A proposal to standardise on using an /llms.txt file to provide information to help agents use a website.

The file is Markdown. An H1 with the site’s name is the only required part; a blockquote summary, then H2 sections of links with one-line notes, are the convention. Version 2 adds a way for a page to point at the file — a <link> with rel="describedby", or the same relation in an HTTP Link header — and says plainly how the file is meant to be consumed:

Agents are expected to view or search llms.txt to find the information they need, then follow the relevant links.

The proposal says nothing about search rankings or AI citations, anywhere. The claim that the file helps a site get cited is made on the proposal’s behalf by pages selling something, not by the proposal.

The examples worth reading are the ones the spec itself points to. On 17 September 2026, OpenAI (developers.openai.com/llms.txt), Anthropic (docs.anthropic.com/llms.txt) and Google’s Gemini API (ai.google.dev/gemini-api/docs/llms.txt) all serve one. All three are developer documentation. None is the company’s marketing site. Ours is generated from this site’s content collection on every build, so it cannot drift from the pages that exist.

What Google says, twice

Two Google documents, two months apart, two answers — and they do not contradict each other.

Chrome’s Lighthouse added an llms.txt check on 5 May 2026, in a new audit category called Agentic Browsing, beside checks on the accessibility tree, layout stability and WebMCP tool registration:

The llms.txt file is an emerging convention used to provide a machine-readable summary of a website’s content, specifically designed for LLMs and AI agents.

Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content.

If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.

Google Search Central’s guide to generative AI features, last updated 10 July 2026, lists the file first under the heading “Here are a few things you can ignore for Google Search”:

You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.

It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.

And in the guide’s closing list, under “Prioritize effective SEO strategies over ‘AEO/GEO hacks’”:

For Google Search, you can ignore tactics like “chunking” content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions.

Read together, the two documents settle what the file is for. Lighthouse measures how well a site works for an agent browsing it; Search Central describes what ranks and what gets cited. llms.txt belongs to the first and not the second, and Google says both halves itself.

Who publishes it — counted

On 14 September 2026 we fetched /llms.txt from every domain that ranked in the top ten or was cited by an AI Overview across 75 US searches about SEO and GEO — the corpus behind our structured-data census. 276 domains; 248 answered. A file counted only if it opened with a Markdown H1, because sixteen domains — Salesforce, Algolia and WooCommerce among them — return their ordinary HTML page with a 200 status for any path, including this one. A checker that counts status codes reads 51%. One that opens the file reads 45%.

Three things in that table.

The file does not separate cited domains from ranking ones. 43% against 46%. Whatever an AI Overview is selecting for, the presence of this file is not visible in it.

Publishing it and blocking AI crawlers are different populations. Of the 125 domains with a file, one blocks a training crawler in its robots.txt and none blocks a retrieval crawler. The file is not a stance on AI; it is something sites that welcome crawlers add.

It travels with structured data. 115 of the 125 also carry JSON-LD, 92%, against 78% of the domains without a file. llms.txt is something the already thorough do in addition, not instead.

Now the caveat that changes the reading. These are SEO agencies, SEO tools, Shopify and its help centre, and marketing publishers — the population most aware this file exists. SE Ranking’s analysis of almost 300,000 domains, published 7 September 2026, found the file on 10.13%; the same article reports Trakkr’s 13.3% across 37,894 AI-cited domains in March and Casey Burridge’s 5.61% across the HTTP Archive top 10,000 in June. The industry that recommends the file publishes it at three to eight times the rate of the web it recommends it to — and inside that industry, it still makes no difference to who gets cited.

The search that recommends it

We captured the US results for “llms txt” on 17 September 2026 and fetched /llms.txt from every domain on the page. Seven pages rank: the proposal and its GitHub repository, Google’s Lighthouse documentation, Semrush, two documentation platforms that generate the file for their customers, and one page that now returns a 404. No ads on a 5,000-search-a-month term.

Five of the seven ranking domains have one. The two that do not are Google’s own developer site — the page documenting the audit that checks for the file — and a dead page. Of the domains the AI Overview cites without ranking, the one selling AI visibility software has none. Lighthouse’s page is fair about its own absence: a 404 is Not Applicable, because the file is optional.

The AI Overview on that search ends with its verdict, and its source is a forum: “Most users on Reddit SEO agree that major AI models and crawlers do not heavily rely on llms.txt yet.” Google’s own Lighthouse page ranks second on the same search, and the proposal that ranks first cites it. Asked the same question the same day, Perplexity cited thirteen sources, and not one of them appears in Google’s seven results or in the overview’s eight citations.

Who reads it

Publishing is measured above. Reading is measured by almost nobody, and the one number on the ranking page belongs to Semrush, from its own server logs:

From mid-August to late October 2025, the llms.txt page received zero visits from Google-Extended bot (Google’s AI crawler), GPTbot (OpenAI’s crawler), PerplexityBot, or ClaudeBot.

One site, ten weeks, a year ago — and consistent with Google’s statement. What the proposal claims, and what its examples support, is narrower: coding agents reading developer documentation. Version 2 says the file is “used most heavily for software documentation, where coding agents follow them to find API references and tutorials,” and the three labs’ files are exactly that. We have not measured crawler requests to our own file, and neither has any page on the search; a claim about who reads llms.txt that does not come from a server log is a guess.

Mandatory, and where to submit it

Not mandatory, and nowhere. The file is hosted, not submitted: an agent fetches /llms.txt directly, and no engine has a form for it. Version 2 adds the one piece of discoverability there is — <link rel="describedby" href="/llms.txt"> in a page’s head, or the same relation as an HTTP Link header. Every page on this site carries that link element as of 17 September 2026, added after reading the revision; it took one line.

What to actually do

1. Publish one if agents will read your site — developer documentation, a product with an API, anything a coding assistant gets pointed at. That is where the proposal’s own evidence is.
2. Generate it; never hand-maintain it. Ours is a route that reads the content collection, and the build fails if the file and the site disagree. A hand-written llms.txt is stale the week after it ships. A Shopify store already has a generated one: on 26 September 2026 we found a real /llms.txt on 28 of the 33 Shopify stores we checked, in our GEO for ecommerce measurement.
3. Add the link relation. One rel="describedby" element per page is the whole of version 2’s discoverability, and it costs nothing.
4. Do not count it as SEO. Google Search ignores the file and says so. It is not structured data, not a sitemap and not a robots.txt; it changes nothing about what ranks or what an AI Overview cites.
5. Do not pay for a score on it. A tool that marks a site down for a missing llms.txt is scoring against Lighthouse’s Not Applicable and Google’s ignore. The check is a fetch and a first line; anyone can run it.

What this does not show

• Not the web’s adoption rate. 45% is adoption among domains that rank or get cited for SEO and GEO queries. The 5.6–13.3% figures are other people’s, reported here through SE Ranking’s article.
• Not who reads the file. We measured who publishes it. Semrush’s logs are one site in 2025; ours have not been read for this.
• Not why cited and ranking domains adopt at the same rate. A non-difference was measured, not a mechanism.
• Not why the labs publish theirs. They publish for their developer docs; their reasons are not on record and are not inferred here.
• One locale variant of Google’s guide. On 17 September 2026 the guide served with ?hl=en omitted the “ignores them” sentences while the same URL without the parameter carried them, under the same “last updated” date. The quotes above are from the bare URL.

Why we wrote it

Every explainer on this search was written before May 2026, so none of them has the two Google documents or the revised proposal — the three texts that answered the question this year, from both sides. The proposal ranks first and cites Lighthouse; Lighthouse ranks second and says nothing about Search. The one page that holds all three, SE Ranking’s, does not rank on Google for the term and is cited by Perplexity instead. And nobody had counted the file on the domains that recommend it. The file is real, cheap, and for agents. Treating it as a lever for search or citations is the mistake the numbers rule out.