llms.txt: What It Is, and Whether It Is Doing Anything for You

A client forwarded us a newsletter recently with a subject line to the effect that every website now needs an llms.txt file or it will vanish from AI search. The file is real, the proposal behind it is sensible, and that sentence is marketing. Both things can be true, and knowing which part is which is worth the ten minutes it takes to read this.

What the File Actually Is

llms.txt is a proposed convention: a plain-text file at the root of your domain, written in Markdown, that gives a language model a curated map of your site. Not a crawl blocklist — a reading list. It typically opens with a short description of what the organisation does, then lists the pages that matter with a line of context for each, so that a model arriving with a question has a shortcut to the authoritative pages instead of having to infer structure from your navigation.

The reasoning behind it is genuinely good. A modern site delivers its content wrapped in navigation, cookie banners, scripts and layout. A human eye discards all of that instantly. A model has to spend its limited attention budget on it. A clean Markdown index of what is worth reading is a real improvement over making something parse your homepage.

What It Is Not

It is not robots.txt, and confusing the two causes real damage. robots.txt is an instruction about access that established crawlers honour. llms.txt is a suggestion about reading order that carries no access control whatsoever. Putting a page in llms.txt does not permit anything, and leaving it out does not block anything. If your intent is to stop a specific AI crawler from taking your content, llms.txt is the wrong tool entirely — that conversation belongs in robots.txt and your terms.

It is also not a ranking factor, in AI search or anywhere else. There is no evidence that adding the file moves you up in anything. The honest framing is that it may make your content easier to use correctly for any system that chooses to read it.

The Part the Newsletters Skip

Here is the uncomfortable bit: this is a community proposal, not a standard any major AI vendor has publicly committed to honouring. Adoption on the publishing side has run well ahead of any confirmed adoption on the consuming side. It is entirely possible to add the file, do it well, and have nothing read it.

We are saying this because the alternative is letting a client believe they have bought a result they have not bought. If somebody sells you llms.txt implementation as an AI visibility service with an outcome attached, ask them which specific system reads it and how they intend to verify that. The answer is instructive.

So Should You Add One?

Probably yes, for reasons that have nothing to do with the hype:

• It is cheap. A competent version of this file is an afternoon of work, most of it deciding what belongs on the list.
• It has no downside. It cannot deindex you, it cannot leak anything you did not already publish, and no engine penalises its presence.
• The exercise itself is useful. Writing a one-line description of every page that matters forces you to notice which pages do not, and that is a content audit arriving through the side door.
• If adoption does arrive, you are already there, and you are there with a curated file rather than a generated one.

That is a considerably smaller claim than the one in the newsletter, and it is the one we are willing to stand behind.

How to Write One That Is Worth Reading

Most llms.txt files we look at are a sitemap with the XML stripped off — every URL, no context, no hierarchy. That is a wasted opportunity, because the entire value of the format is the editorial judgement it lets you express.

A file worth having does the following:

• Opens with a plain description of what the organisation does and who it serves, in language a stranger would understand. No slogans.
• Lists a small number of genuinely important pages, not everything. If a page would not be a good answer to a question about you, it does not belong.
• Gives every entry a line of context explaining what the page answers, not just what it is titled.
• Separates the essential from the optional, so that a system reading only the first section still gets an accurate picture.
• Gets updated when the site changes. A stale file confidently describing pages that no longer exist is worse than no file.

Length is not the goal. A short file listing the ten pages that actually matter will serve you better than one listing four hundred, for the same reason a good index beats a photocopy of the whole book.

What Actually Moves AI Visibility

If the reason you are reading about llms.txt is that you want to be cited by AI answers, it is worth being clear that this file is nowhere near the top of the list. The things that have consistently mattered are duller: content that answers a specific question directly rather than circling it, pages that state facts a model can lift and attribute, structured data that is accurate, and being mentioned somewhere other than your own website.

Add llms.txt. It costs an afternoon and it might help. Then go do the work that definitely does.

If you want a read on how your site currently looks to an answer engine, our services page is at https://netwebmedia.com/services.html, and WhatsApp is at https://netwebmedia.com/whatsapp.html.

← Back to all articles