The complete guide to llms.txt

Every so often, a single file gets famous. Right now it is llms.txt.

You have probably seen the claims. It is the robots.txt for AI. Every website needs one immediately. It might become one of the most important files on the web. The energy around it is real, and so is the confusion.

Here is the calmer version. llms.txt solves a genuine problem. It is also an early proposal, not an established standard, and there is no evidence yet that adding one will lift your AI visibility on its own. Both of those things are true at the same time. That is worth sitting with before you rush to publish one, or rush to dismiss it.

So let's walk through what it actually is, what it does not do, and whether it deserves a place on your list.

What llms.txt actually is

An llms.txt file is a plain text document you place at the root of your site, usually at https://example.com/llms.txt.

Its job is simple. It gives large language models a short, curated map of the pages you consider most useful. Instead of hoping a system works its way through thousands of URLs to find the parts that matter, you point it straight at your documentation, product pages, API reference, pricing, research, whatever explains your business best.

Think of it as a map, not a rulebook. It cannot tell an AI what to do. It can make the useful things easier to find.

Why anyone proposed it

Most websites are bigger than they need to be.

Before a crawler reaches the documentation that answers a real question, it might wade through campaign pages, duplicate URLs, old announcements, filtered category views, and internal search results. Search engines spent decades learning to cut through that noise, with canonical tags, XML sitemaps, structured data, and a hundred other signals.

Language models have slightly different needs. When they pull information together to answer a question, it helps to know which pages a site itself treats as authoritative. That is the small gap llms.txt is trying to fill.

The idea came from Jeremy Howard in 2024, as a lightweight convention that sites and AI platforms could choose to adopt. Since then, a growing number of CMS plugins and developer tools have started generating these files automatically.

It is not a standard, and that matters

This is the part the excited takes tend to skip.

Unlike robots.txt, there is no recognized standard that requires AI systems to read or respect llms.txt. It is a voluntary convention. Whether it becomes genuinely useful depends entirely on whether the large AI platforms decide to support it, and right now nobody can tell you how far that will go.

That does not make it worthless. It does mean you should treat confident claims about its impact with a raised eyebrow.

How it differs from robots.txt

The comparison is understandable. Both files live at the root of your site, and both are written for machines. But they do almost opposite things.

robots.txt is about access. It tells crawlers where they may and may not go. llms.txt blocks nothing and permits nothing. It is closer to a recommended reading list. It says, in effect, if you are trying to understand this site, start with these pages.

One is a fence. The other is a signpost.

One is a fence, the other a signpost: robots.txt controls access, llms.txt just points to your most useful pages.

What goes inside

Part of the appeal is how little there is to learn. A typical llms.txt includes:

• the company or project name
• a short description of the site
• links to the pages that matter most, such as product, documentation, API reference, help center, pricing, and research
• optional notes explaining what each link is

No new syntax, no tooling required. Most sites can write one in a few minutes and keep it current without much thought.

Will ChatGPT actually use it?

This is the first question everyone asks, and the honest answer is that we do not know.

The picture is mixed, and it keeps moving. Google has been the most direct about it: its own guidance says llms.txt is not needed for AI Overviews, AI Mode, or any of its generative search features. Others have warmed to the idea, recommending it or fetching it in places. But no major system has clearly documented that it systematically reads the llms.txt of sites it does not control, and the confident claims tend to run ahead of what the platforms have actually confirmed. Adoption across the web is still small. Until the people building these systems say more, anything more specific is guesswork dressed up as insight.

Does it improve AI citations?

There is no public evidence that adding an llms.txt file, by itself, gets you cited more often or ranked higher inside AI answers.

Which should not surprise anyone. A text file cannot make thin content authoritative. It cannot manufacture expertise or independent recognition. If your pages do not answer questions well, making them easier to find will not change whether they deserve to be cited. The content still carries the weight. The file just points at it.

Where it genuinely helps

None of that means it is pointless.

Picture two SaaS companies. The first has fifteen thousand URLs built up over years of launches, campaigns, blog posts, and abandoned microsites. The second offers a clean llms.txt pointing straight at its docs, pricing, API reference, and security information.

Which one is easier to understand?

The second, clearly. And the advantage has nothing to do with the file being magic. It comes from removing ambiguity. Instead of asking a retrieval system to guess which pages represent you, you tell it. That is a small thing that can quietly matter.

Should you make one?

For most sites, probably yes.

Not because it is a proven ranking factor, but because the cost is close to nothing. It takes a few minutes, needs almost no upkeep, and leaves you ready if support grows. If adoption stalls, you have lost very little. This is housekeeping, not a growth hack.

If you do write one, point it at the pages that actually represent you: homepage, product, documentation, API docs, help center, pricing, security, privacy, research, company information. For SaaS especially, documentation tends to be the most valuable section, because it explains what you do clearly and in enough detail for a machine to follow.

And leave out the clutter. Internal search results, temporary campaign pages, duplicate URLs, thin pages, expired promotions, old announcements nobody needs. The goal is not a complete index. It is an honest highlight reel.

It is not an SEO shortcut

Every few years a new technical feature arrives wrapped in outsized expectations. Structured data was going to transform rankings. Core Web Vitals were going to decide everything. llms.txt is getting the same treatment now, and it is getting a little too much credit.

Technical improvements are worth doing. They just tend to matter only when there is something worth finding underneath them. Original research, real expertise, strong documentation, and mentions from places you do not control still do far more for you than any single file in your root directory.

The question worth asking instead

Rather than debating whether llms.txt is a ranking factor, ask something you can actually measure.

Is your documentation showing up more often in AI answers? Are your product pages being cited more? Has your AI visibility moved at all? Which of your pages are actually shaping what assistants say about you?

Those are answerable questions, and they are the ones we built Voris to answer. Whether an llms.txt file moves any of them is something you can test with real data instead of taking it on faith. For what it is worth, we keep our own llms.txt, and we treat it exactly this way: useful housekeeping we measure, not a lever we oversell.

The bigger picture

It is easy to fixate on the technical details, because they are the easy part. Adding a file is simple. Earning the attention it points to is not.

Think of your site as a library. An llms.txt file does not write better books. It hands the librarian a map to the best shelves. If the books are good, that map saves everyone time. If the books are weak, the map changes almost nothing.

That is really the whole story. The companies earning AI citations are not winning because of one file. They are publishing original work, explaining hard things clearly, building a reputation beyond their own domain, and making all of it easy to reach. A small text file can support that effort.

It just cannot replace it.