The complete guide to llms.txt

Every so often a single file gets famous, and then gets torn down. Right now llms.txt is managing both at once.

For a while the pitch was everywhere. It is the robots.txt for AI. Every site needs one immediately. Then the backlash arrived, and it brought receipts. Independent analyses have struggled to find any link between having an llms.txt file and getting mentioned more often in AI answers. A large share of the files people publish are never fetched by an assistant at all. Google has said in plain language that it does not need the file for its AI search surfaces. In some corners it is already being written off as a dud, or the keywords meta tag reborn.

So here is the calmer read. The skeptics have the hype dead to rights, and the file is still worth understanding, because for the right kind of site, built with a little care, it is cheap, harmless, and quietly useful. Both things are true at the same time. That is worth sitting with before you rush to publish one, or write it off.

Let's walk through what it actually is, why the doubt is fair, and when it earns a place on your list.

What llms.txt actually is

An llms.txt file is a plain text document you place at the root of your site, usually at https://example.com/llms.txt.

Its job is simple. It gives large language models a short, curated map of the pages you consider most useful. Instead of hoping a system works its way through thousands of URLs to find the parts that matter, you point it straight at your documentation, product pages, API reference, pricing, research, whatever explains your business best.

Think of it as a map. It cannot tell an AI what to do. What it can do is make the useful things easier to find.

Why anyone proposed it

Most websites are bigger than they need to be.

Before a crawler reaches the documentation that answers a real question, it might wade through campaign pages, duplicate URLs, old announcements, filtered category views, and internal search results. Search engines spent decades learning to cut through that noise, with canonical tags, XML sitemaps, structured data, and a hundred other signals.

Language models have slightly different needs. When they pull information together to answer a question, it helps to know which pages a site itself treats as authoritative. That is the small gap llms.txt is trying to fill.

The idea came from Jeremy Howard in 2024, as a lightweight convention that sites and AI platforms could choose to adopt. Since then, a growing number of CMS plugins and developer tools have started generating these files automatically.

It is only a convention

This is the part the excited takes tend to skip.

Unlike robots.txt, no recognized standard requires AI systems to read or respect llms.txt. It is a voluntary convention, and whether it becomes genuinely useful depends on whether the large AI platforms decide to support it. Right now, nobody can tell you how far that will go. Treat confident claims about its impact with a raised eyebrow.

How it differs from robots.txt

The comparison is understandable. Both files live at the root of your site, and both are written for machines. Their jobs, though, point in almost opposite directions.

robots.txt is about access. It tells crawlers where they may and may not go. llms.txt blocks nothing and permits nothing. It works more like a recommended reading list, a way of saying: if you are trying to understand this site, start with these pages.

One is a fence. The other is a signpost.

One is a fence, the other a signpost: robots.txt controls access, llms.txt just points to your most useful pages.

What goes inside

Part of the appeal is how little there is to learn. A typical llms.txt includes:

• the company or project name
• a short description of the site
• links to the pages that matter most, such as product, documentation, API reference, help center, pricing, and research
• optional notes explaining what each link is

There is no new syntax and no tooling required. Most sites can write one in a few minutes and keep it current without much thought.

Will ChatGPT actually use it?

This is the first question everyone asks, and the honest answer is that we do not fully know. The early signals, though, are underwhelming.

Google has been the most direct about it. Its own guidance says llms.txt is not needed for AI Overviews, AI Mode, or any of its generative search features. Beyond that, no major AI system has clearly documented that it systematically reads the llms.txt of sites it does not control. Adoption across the web is still in the low single digits, and when researchers have gone looking, most published files sit untouched, fetched by no assistant at all. Until the people building these systems say more, confident claims about what llms.txt does for your visibility are guesswork dressed up as insight.

Does it improve AI citations?

There is no public evidence that adding an llms.txt file, by itself, gets you cited more often or ranked higher inside AI answers. Analyses across hundreds of thousands of domains have come up empty on that link.

Which should surprise no one. A text file cannot make thin content authoritative. It cannot manufacture expertise or independent recognition. If your pages do not answer questions well, making them easier to find will not change whether they deserve to be cited. The content still carries the weight. The file just points at it.

Where it genuinely helps

None of that makes it pointless.

Picture two SaaS companies. The first has fifteen thousand URLs built up over years of launches, campaigns, blog posts, and abandoned microsites. The second offers a clean llms.txt pointing straight at its docs, pricing, API reference, and security information.

Which one is easier to understand?

The second, clearly. The advantage has nothing to do with the file being magic. It comes from removing ambiguity. Instead of leaving a retrieval system to guess which pages represent you, you simply tell it. That is a small thing that can quietly matter, and it is exactly why documentation-heavy products, developer platforms, and knowledge bases get more out of llms.txt than a thin brochure site ever will.

Should you make one?

For most content-heavy sites, probably yes.

The reasoning is unglamorous: the cost is close to nothing. It takes a few minutes, needs almost no upkeep, and leaves you ready if support grows. If adoption stalls, you have lost very little. Think of it as sensible housekeeping, and keep your expectations modest.

If you do write one, point it at the pages that actually represent you: homepage, product, documentation, API docs, help center, pricing, security, privacy, research, company information. For SaaS especially, documentation tends to be the most valuable section, because it explains what you do clearly and in enough detail for a machine to follow.

And leave out the clutter. Internal search results, temporary campaign pages, duplicate URLs, thin pages, expired promotions, old announcements nobody needs. Aim for an honest highlight reel of the pages that represent you.

An llms.txt is a curated highlight reel: point at the pages that represent you, and strip the clutter.

You do not have to build it by hand. The Voris llms.txt generator reads your site, drafts a clean file with your key pages and one-line descriptions, and hands it back for you to edit before you publish. Generate your llms.txt.

It is not an SEO shortcut

Every few years a new technical feature arrives wrapped in outsized expectations. Structured data was going to transform rankings. Core Web Vitals were going to decide everything. llms.txt is getting the same treatment now, and it is getting a little too much credit.

Technical improvements are worth doing. They just tend to matter only when there is something worth finding underneath them. Original research, real expertise, strong documentation, and mentions from places you do not control still do far more for you than any single file in your root directory.

The question worth asking instead

Rather than debating whether llms.txt is a ranking factor, ask something you can actually measure.

Is your documentation showing up more often in AI answers? Are your product pages being cited more? Has your AI visibility moved at all? Which of your pages are actually shaping what assistants say about you?

Those are answerable questions, and they are the ones we built Voris to answer. Whether an llms.txt file moves any of them is something you can test with real data instead of taking it on faith. For what it is worth, we keep our own llms.txt, and we treat it exactly this way: useful housekeeping we measure, and never a lever we oversell.

The bigger picture

It is easy to fixate on the technical details, because they are the easy part. Adding a file is simple. Earning the attention it points to is not.

Think of your site as a library. An llms.txt file does not write better books. It hands the librarian a map to the best shelves. If the books are good, that map saves everyone time. If the books are weak, the map changes almost nothing.

That is really the whole story. The companies earning AI citations are not winning because of one file. They publish original work, explain hard things clearly, build a reputation beyond their own domain, and make all of it easy to reach. A clean llms.txt can help that work travel. Ask it to do the work itself, and you will be disappointed.