How llms.txt works
The idea was proposed in 2024 to solve a practical problem: web pages are full of navigation, scripts and adverts, and AI tools working within limited context struggle to find the substance. An llms.txt file offers a clean map instead, with a title, a short description and grouped lists of links, sometimes pointing to Markdown versions of key pages. A companion llms-full.txt can hold the full text.
It differs from robots.txt, which tells crawlers what they may access, and from an XML sitemap, which lists URLs for indexing. llms.txt is guidance about what matters, written for language models. Some documentation platforms generate one automatically, and some AI coding tools read them when pointed at a site.
The misconception is that llms.txt makes a site visible in AI answers. No major AI search product has confirmed relying on it for ranking or citation, and Google has given no indication that it does. Treat it as a cheap, low-risk addition, not a substitute for crawlable pages and clear content.
How to improve it
- 01Fix crawl access, indexing and page clarity first, since these matter far more.
- 02If you add the file, start with a one-paragraph summary of who you are and what you offer.
- 03Group links to your most useful pages under short headings, with a one-line note on each.
- 04Keep it accurate as pages change, because an outdated map is worse than none.
- 05Do not use it to block or allow crawlers; that remains the job of robots.txt.
A payroll software company with hundreds of help articles adds an llms.txt listing its setup guides, integration pages and pricing explanation in plain sections. Its developers already point AI coding tools at the file. The marketing team records it as an optional addition, not a visibility tactic.