The short answer: a curated map of your site, written for machines
llms.txt is a plain markdown file you put at the root of your domain, at /llms.txt, listing the pages you most want an AI model to read. It was proposed in late 2024 as a way to help large language models find the useful parts of a website quickly, instead of crawling navigation, cookie banners and footers to piece together what you do.
The reasoning behind it is sound. A model working on a question has a limited amount of room for context. Feeding it your entire site is wasteful, and most of a modern web page is markup rather than meaning. A short curated index solves that: here is what we do, here are the twelve pages that matter, in clean text.
It is a file written for readers who may never turn up. Marketing has done stranger things for less.
How it differs from robots.txt and your sitemap
The name causes most of the confusion, because it looks like a cousin of robots.txt. It is not doing the same job.
- robots.txt controls access. It tells crawlers where they may and may not go, including the AI crawlers that publish a user agent. Google’s robots.txt documentation covers the syntax. This is the file that actually has enforcement behind it.
- sitemap.xml lists everything. Every indexable URL on your site, for search engines that want completeness. It is a full inventory, not a recommendation.
- llms.txt lists what matters. A short, human-curated set of links with one-line descriptions, in markdown, aimed at a model that has room for a handful of pages rather than a thousand.
There is a companion convention, llms-full.txt, which holds the actual content of those pages in one file rather than links to them. That one gets big quickly and is mostly used by documentation sites.
Worth being clear on one thing: llms.txt does not stop anyone training on your content, and it is not a permission system. If that is your concern, robots.txt and your terms of use are the tools, not this.
What actually goes in the file
The format is deliberately simple, which is the best thing about it. A heading with your business name, a one-line summary in a blockquote, then sections of markdown links with short descriptions.
- An H1 with the name of your business or product. Required. This is the only mandatory element.
- A blockquote summarizing what you do. One or two sentences, plainly written. If a model only reads this line, it should still get you right.
- Optional context paragraphs. Anything a reader needs to interpret the rest: who you serve, where you operate, what you do not do.
- H2 sections with lists of links. Each item is a markdown link plus a short note: what the page covers and who it is for. Group them sensibly, for example Services, Guides, About.
- An "Optional" section at the end. The convention is that anything under this heading can be skipped when context is tight. Put your secondary material here.
Keep it short. The whole point is curation. A file listing two hundred URLs has recreated your sitemap and thrown away the only advantage it had.
If you cannot write a one-sentence blockquote that describes your business accurately, the file is not your problem. Your positioning is.
Does anything actually read it?
This is the part most articles on the subject skip, so here it is plainly: as things stand, no major search or AI provider has confirmed that it uses llms.txt to decide what to retrieve or cite. Google’s search advocates have said publicly that Google does not use it. The large assistant providers have not committed to it either.
What has adopted it is a slice of the developer world: documentation sites, developer tools, and platforms that generate the file automatically for every site they host. In that context it makes obvious sense, because the audience genuinely is a coding assistant pulling in reference material.
For a plumbing business in Adelaide or a fashion store on Shopify, the honest position is that this is a low-cost bet with no confirmed payoff yet. That is not the same as useless. Standards get adopted, and this one costs almost nothing to support. It is just not the thing that will get you cited in an AI answer this quarter.
If you can’t measure it properly, don’t celebrate it yet. Anyone selling llms.txt as an AI visibility service should be able to explain how they will show you it made a difference. So far, nobody can.
So should you add one?
Add one if it costs you an hour and you are the kind of business an AI assistant would be asked to summarize: software, documentation, professional services with a lot of explanatory content, anything with an API. It is cheap insurance against the standard being adopted, and writing it forces a useful conversation about which twelve pages actually matter.
Do not add one if any of this is true:
- Someone wants to charge you a monthly fee for it. It is a static text file. It should be written once, reviewed when your services change, and otherwise left alone.
- It would come out of the budget for pages people actually read. A well-written service page earns its keep in search, in AI answers and with human beings. An llms.txt file currently earns its keep in exactly none of those, on the public evidence.
- Your site has bigger problems. If key pages are not indexed, or your content does not answer the questions people ask, a file pointing at them changes nothing. Our guide to what a technical SEO audit checks covers the order to work in.
- You would auto-generate it from your sitemap. That produces a long list with no curation, which is the one thing the format was designed to avoid.
What actually gets you cited by AI assistants
The mechanisms that decide whether an assistant mentions your business are, for now, the same ones that decide whether Google shows you. Google says as much in its guidance on AI features in Search and its guide to optimizing for them: there is no separate lever, just content that is findable, clear and worth quoting.
In practice that means five unglamorous things:
- Answer the question in the first paragraph. Models quote passages that state an answer. Copy that circles for three paragraphs before getting to the point does not get pulled.
- Use headings that match real questions. If somebody asks it, make it a heading and answer it underneath.
- Put facts in text, not in images or PDFs. Prices, specifications, service areas, hours. If it only exists in a picture, it does not exist.
- Keep structured data accurate. It is how machines confirm what a page is about without guessing.
- Be citable. Specific, dated, sourced statements get referenced. Vague claims get skipped.
There is a longer version of this in how to rank in AI Overviews and a practical checklist in how to do AEO. If you want the strategy rather than the file, that is what our answer engine optimization work covers.
Frequently asked questions
What is llms.txt used for?
It is a markdown file at the root of a domain that gives large language models a short, curated list of the pages that matter most on your site, each with a one-line description. The aim is to save a model from crawling navigation and boilerplate to work out what you do, at a point where its available context is limited.
Is llms.txt the same as robots.txt?
No. robots.txt controls which crawlers may access which parts of your site and is the file with real enforcement behind it. llms.txt makes no access rules at all. It is a recommendation about which pages are worth reading, and it does nothing to prevent training or scraping.
Does Google use llms.txt?
No. Google has said publicly that it does not use the file. Adoption so far is concentrated in developer documentation and platforms that generate it automatically. Treat it as a low-cost bet on a possible future standard rather than as something that affects your visibility today.
How do I create an llms.txt file?
Write a markdown file with an H1 naming your business, a blockquote summarizing what you do, then H2 sections listing your most important pages as markdown links with short descriptions. Put lower-priority links under a final heading called Optional. Save it as llms.txt at the root of your domain. It takes about an hour.
Will llms.txt help me appear in ChatGPT or AI Overviews?
There is no evidence that it does. Being mentioned in AI answers currently depends on the same things as ranking well: content that is crawlable, states answers plainly, matches real questions, and is specific enough to quote. Build that first, and add llms.txt afterwards if you want the belt as well as the braces.