Short answer: sitemaps and robots.txt are the machinery that governs discovery, and they have been for years. llms.txt is a reasonable proposal that no major platform has committed to reading. Add it if you want to test something; do not build on it.
The three files, and which ones do the work
| File | What it is | Status | Worth your time |
|---|---|---|---|
sitemap.xml | A list of URLs you want indexed, with dates | Standard, documented, working | Yes |
robots.txt | Per-path crawl instructions per user agent | Convention, widely honoured | Yes, deliberately |
llms.txt | Prose summary of your important pages for language models | Proposal, not adopted | Optional experiment |
Only the first two are infrastructure you should depend on.
sitemaps.xml
The format is deliberately boring: XML, a urlset, and inside it one url element per address with an optional lastmod. The protocol documentation is short and worth reading once, because most sitemap problems are misunderstandings of it rather than misconfiguration. [3]
Google’s guide to building and submitting one is the operational reference, and it is the right place to confirm the format before generating anything by hand. The detail that catches people out is lastmod: Google says to set it only when content meaningfully changed, and notes that setting it on every build trains crawlers to distrust the signal. An accurate lastmod is worth having. A wrong one is worse than none. [2]
For a site with a few dozen pages, this is a two-minute-per-month job, and most of it is noticing when a URL has moved.
robots.txt
Robots.txt is a convention, not a control. There is no enforcement; a crawler that ignores it is not violating anything, it is simply ignoring a file it has to fetch first. That matters for how you think about AI crawlers specifically, because they are not a single entity with one policy.
Google publishes its crawler user agents, including the ones it uses for AI features, and documents how to check what a given user agent has actually fetched through its crawl statistics. [4]
If you want to be found by systems that read these files, allow the relevant user agents. The blunt warning is that a Disallow: / aimed at all crawlers also removes you from anything built on those systems — and, for search, entirely. Ahrefs’ guide covers the practical side of writing directives that behave the way you intended. [5]
llms.txt, stated honestly
llms.txt is a proposal for a plain-text file at the root of a domain, containing a title, a short summary, and a set of links with descriptions, intended to give language models an efficient way to learn what a site contains without parsing its HTML. It reached v2 while I was writing this.
What is true about it:
- The proposal itself is published openly, and the current version is documented in enough detail to generate from — an H1, a blockquote summary, and a list of links each with a description. It is cheap to produce, and the format is legible by a human being. [1]
- It removes a real inefficiency — parsing navigation, boilerplate and cookie banners to find the actual pages.
- It is not adopted by any major platform as a requirement.
What is not yet true, and is worth being clear about rather than repeating:
- That it improves citation rates. There is no published measurement of this.
- That it is read by ChatGPT, Perplexity or Claude. Some may. None have said so.
- That having one and not having one is a ranking distinction.
Treating an unadopted proposal as load-bearing infrastructure is the error here. Treating it as a twenty-minute experiment you run alongside everything else is not.
The more useful version of the same question
If you want a system to summarise your page well, the levers you actually control are boring: one clear claim, stated early, in words that survive being lifted out of the page. An assistant picking your page has to be able to extract a sentence that means something on its own. That is a writing constraint, not a file-format one.
So the honest sequencing is: fix the page first, then decide whether llms.txt adds anything for that page. Not the other way round.
Doing this without an afternoon of yak-shaving
- Confirm your sitemap is accurate. Every URL resolves, returns 200, and no longer redirects.
- Keep
lastmodhonest. Change it when the content changes, not when you deploy. - Read your robots.txt and decide deliberately what each directive is for. Most blanket blocks are cargo-culted.
- Check it against reality, not intention — Google’s crawl stats will tell you whether the agents you allow are the ones arriving.
- Optionally generate llms.txt from your existing page frontmatter. If you already have a title and description on each page, this is a build step rather than a writing project.
An objection worth taking seriously
The strongest argument against llms.txt is that it optimises for machines reading your site and not for people reading it. There is something to that. The counter is that it costs almost nothing if generated and never costs readers anything, because no reader will be harmed by a file they never see.
That asymmetry is the whole decision. Cheap, reversible, unreported: worth trying. Load-bearing: no.
Which is also why the version of this article that told you the 2026 discovery loop now required llms.txt was worth rewriting. A dependency on an unadopted file is a dependency on someone else’s roadmap, and it is the kind of thing that looks like diligence right up until it becomes the reason nothing else got done.
How this connects
- Modern search sets out where discovery fits in a wider foundation.
- What social signals do covers the same problem in a different channel.
- Why reputation beats links covers what a crawler finds once it arrives.
- Work smarter, not harder covers making discovery a build step instead of a recurring task.
