llms.txt is a plain markdown file served at the root of a website that lists the pages an AI model should read first, with a short note on each. It was proposed by Jeremy Howard in September 2024 and revised in August 2026, and the measured answer to "who reads it" as of mid 2026 is: coding agents and documentation tools sometimes, the major answer engines almost never. Most sites therefore do not need one; a developer documentation site is the exception worth making.
What is in an llms.txt file?
A short, strictly shaped markdown document. The specification requires an H1 with the name of the project or site, which is the only required section, followed by a blockquote with a short summary, then zero or more sections delimited by H2 headers containing file lists of URLs, where each entry is a markdown hyperlink, then optionally a colon and notes. A section titled Optional holds secondary information by convention.
The proposal also asks sites to serve a clean markdown version of each page at the same URL with .md appended or with the extension replaced by .md. The reasoning is stated plainly: context windows are still too small for most websites in their entirety, and every wasted token costs time and money, and an HTML page wraps its information in navigation, ads and JavaScript, and converting it back into clean text is difficult and imprecise.
A minimal file for a small company looks like this:
# Example Co
> Example Co makes invoicing software for law firms. Pricing, docs and contact are linked below.
## Docs
- [Getting started](https://example.com/docs/start.md): setup in ten minutes
- [API reference](https://example.com/docs/api.md): every endpoint with examples
## Optional
- [Blog](https://example.com/blog/): articles on billing and compliance
The August 2026 revision changed how the file is meant to be found and used. Version 2 adds standard link relations, rel="alternate" with type text/markdown for markdown versions and rel="describedby" for the applicable llms.txt, and reframes consumption: agents view or search the llms.txt to find what they need, then follow the relevant links, rather than tools stuffing the whole thing into context.
Who actually reads llms.txt?
Almost nobody, according to every measurement published so far. In June 2025 Google's John Mueller wrote on Bluesky, "FWIW no AI system currently uses llms.txt", adding that the consumer chatbots will fetch your pages for training and grounding, but none of them fetch the llms.txt file.
A year later the logs agreed. Ahrefs analyzed 137,000 domains in May 2026 and found that 28% published an llms.txt file and 97% of those files received zero traffic that month. Of the requests that did arrive, 96% came from bots, led by SEO audit tools at 21.7%, unidentified bots at 14.9%, general web crawlers at 13.1% and tech profiling tools at 11.6%. The retrieval bots behind the answer engines barely registered: OAI-SearchBot, PerplexityBot and Claude's search crawler combined made only a couple of hundred fetches, and AI retrieval services accounted for 1.1% of requests.
The proposal's own authors describe the file's use in the same terms. They write that their expectation was that llms.txt would mainly be useful for inference rather than training, and that is how it has been used, and they point to adoption where it exists: thousands of sites publish one, documentation platforms generate one automatically, and Chrome's Lighthouse audits sites for one as part of its agentic browsing checks. That is a picture of a developer tooling convention, not a search channel.
Does llms.txt help you rank in Google, ChatGPT or Perplexity?
No, on the evidence available in September 2026. Google's documentation on AI features states that you don't need to create new machine readable files, AI text files, or markup to appear in AI Overviews or AI Mode; eligibility is being indexed and eligible for a snippet. The Ahrefs figures above show the ChatGPT and Perplexity search crawlers effectively not requesting the file at all.
There is one Google statement that does concern the file directly. In July 2025 Mueller said that using noindex for it could make sense, as sites might link to it and it could otherwise become indexed, which would be weird for users. So the only documented interaction between llms.txt and Google Search is a reason to keep it out of the results.
If your goal is being cited by answer engines, the levers are elsewhere: crawlable pages, text that answers the question near the top, specific figures with sources, and coverage on the pages engines already retrieve. That is the work described on Qubba Marketing, and none of it runs through a text file.
When does a site need one?
When its readers are coding agents. Developer documentation is read by tools that follow links and fetch markdown, which is exactly what the file was built for. The proposal notes that OpenAI, Anthropic and Gemini publish llms.txt files for their own developer docs; Anthropic's, for instance, is a markdown index of its entire developer documentation with an H1, a summary blockquote and H2 sections of links. Documentation hosts have made it a default: Mintlify automatically hosts an llms.txt at the root of every project and an llms-full.txt that combines the entire documentation site into a single file.
| Site type | Publish llms.txt? | Why |
|---|---|---|
| API or developer documentation | Yes | Coding agents and doc tools fetch markdown and follow link lists |
| Open source project site | Yes | Same readers, same benefit, near zero cost |
| SaaS marketing site with a docs section | Only for the docs section | The marketing pages gain nothing measurable |
| Local business, ecommerce, portfolio | No | No documented consumer, and the file can leak into search results |
| Publisher or blog | No | Answer engines retrieve your articles through search, not through the index |
For everything outside that first column, the honest cost benefit is a few minutes of work against no measured return, plus a maintenance duty: an index that lists deleted pages or stale prices is worse than no index.
How do you publish one properly?
If you fall into the documentation case, do it fully rather than symbolically. Serve the file at /llms.txt in the required shape, with real one line descriptions rather than page titles repeated. Provide the markdown versions the spec asks for, at the page URL with .md appended or the extension replaced, and add the version 2 link relations so agents can discover the markdown version and the applicable llms.txt from the HTML page or HTTP headers. Send an X-Robots-Tag noindex header on the file, following Mueller's reasoning. Then check your access logs for /llms.txt after thirty days; if the only visitors are audit tools, you have your answer for that site.
Frequently asked questions
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may not fetch, and the search and training crawlers of Google, OpenAI, Perplexity and Anthropic honor it. llms.txt is a curated markdown index of pages a model should read, with no permission semantics, and the Ahrefs logs show the answer engine crawlers rarely request it.
Should llms.txt be noindexed?
Google's John Mueller thinks it can make sense, because sites might link to it and it could otherwise become indexed, which would be weird for users.
A noindex HTTP header keeps the file out of search results while leaving it fetchable by any tool that wants it.
What is llms-full.txt?
A companion file that inlines everything instead of linking to it.
Mintlify describes its generated llms-full.txt as combining the entire documentation site into a single file, with each page's title, source URL, description and full markdown content.
It suits tools that load a whole documentation set into context at once.
Will the answer engines start using llms.txt later?
Nobody can promise that, and the trend since 2025 has not moved that way.
Google's guidance still says no AI text files are needed, and the 2026 Ahrefs data shows retrieval bots ignoring the file.
Build for what is measured today and revisit if a vendor documents support.