llms.txt

Also known as: LLM-readable index

llms.txt is a proposed plain-text file at the root of your site that gives AI models a clean, curated map of your most important content in Markdown. Like robots.txt for crawlers or sitemap.xml for search, it is a convention meant to help LLMs find and understand your key pages.

llms.txt is a plain-text file you place at the root of your domain to hand AI models a tidy, curated index of your most important content. It follows the spirit of robots.txt and sitemap.xml: a simple convention at a predictable location that helps machines work with your site. Where a sitemap lists everything, llms.txt is meant to be curated and high-signal, written in Markdown so models can read it easily.

What it is for

Large sites bury their best pages under navigation, scripts, and clutter. llms.txt is a chance to say, plainly, “here are the pages that actually matter and here is what each one covers.” A model that respects the file gets a clean shortcut to your key content, which can help it understand your entity and find the right passage to cite.

An honest note on adoption

llms.txt is an emerging convention, not a guaranteed input. Not every engine reads it, and it will not rescue a site that AI crawlers cannot otherwise parse. The right way to treat it: a cheap, forward-looking addition on top of the fundamentals, being genuinely crawlable, well-structured, and written answer-first. It may help where supported and costs almost nothing to add. For a step-by-step, see what llms.txt is and how to add it, and pair it with a sensible AI crawler policy.

Frequently asked questions

Do AI models actually use llms.txt?

Adoption is still early and not guaranteed across engines. It is a low-cost, forward-looking convention: it may help models that support it and does no harm if they do not. Treat it as a nice-to-have on top of a genuinely crawlable, well-structured site.

What goes in an llms.txt file?

A short description of your site and a curated, linked list of your most important pages, written in Markdown, often with brief notes on each. The point is a clean, high-signal index rather than a dump of every URL.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may access. llms.txt offers models a curated guide to your best content. They serve different purposes and can coexist.