llms txt generator

llms.txt is a proposed Markdown convention for describing a site and linking to canonical documentation. Support and crawler behaviour vary by tool and can change, so treat the file as a maintained navigation aid rather than a ranking signal. This generator creates a concise starting draft for human review.

Why llms.txt exists

A clear index can point a reader or an automated tool to the pages that define your product, policy or technical documentation. It does not control what a model fetches, how it ranks sources or how it answers a question. The strongest foundation remains accurate, accessible and internally consistent canonical content.

How llms.txt differs from robots.txt and sitemap.xml

• robots.txt is permission — what crawlers may and may not access
• sitemap.xml is inventory — every URL you want indexed, in machine-readable form
• llms.txt is an optional editorial index that some tools may use as an additional navigation hint

All three can coexist. Keep any optional index concise enough for a person to review and keep its links current.

The spec at a glance

An llms.txt-style file is Markdown commonly organised around a title, short description and links to core documents. There is no universal enforcement mechanism, so prioritise clarity and accurate links over a rigid format:

1. H1 with the project or site name (required)
2. Blockquote with the canonical URL (recommended)
3. Free-form paragraph describing what the site is for (recommended)
4. `## Docs` H2 with a Markdown bullet list of [Title](URL) links to core docs (recommended)
5. `## Optional` H2 with secondary resources (blog, changelog, examples) (optional)

Use headings and link labels that make sense to a human reader. Do not claim that a particular file structure guarantees ingestion or citation.

What to put in the Docs section

Think "if a model could only read 5 pages of my site to answer questions about it, which 5?" That's the Docs list. For most sites this is: getting-started, core concepts, the API reference (or feature reference), pricing, and a FAQ. Resist the urge to dump your entire sitemap — that defeats the purpose.

Optional links: what to add and what to leave out

Use the Optional section for context-broadening resources: the changelog (helps LLMs answer "what's new" questions accurately), the blog index (gives editorial voice), a public roadmap, public case studies. Leave out anything gated, anything outdated, and anything that contradicts your Docs section — LLMs penalise contradictions by citing less.

Where to publish it

Upload the generated file to https://yourdomain.com/llms.txt — the root of your site, same convention as robots.txt. Some platforms also publish llms-full.txt (a single concatenated document version of all the linked docs) for models that want the full corpus in one fetch. The generator above produces the index file; the full version is a content task you do later.

Adoption today

Publishing an optional content index may be useful when it is accurate and maintained, but it is not a proven shortcut to visibility or citation. Assess it alongside the quality of the underlying content, crawl accessibility and the needs of your users.

Common mistakes

• Skipping the canonical URL blockquote — LLMs use it to disambiguate when your site has many subdomains
• Too many links in Docs — keep it under 10, ideally 5–8
• Pasting full content into Optional — link to it, don't inline it; the file is meant to be small
• Forgetting to update after a re-architecture — broken links here erode trust like 404s in your sitemap