What Is llms.txt? Specification, Examples, and SEO Reality

As developers and search marketers adapt websites for large language models (LLMs), a new root-level file has appeared across documentation sites and content platforms: /llms.txt (often paired with /llms-full.txt). Proposed in September 2024 by Jeremy Howard at Answer.AI and documented at llmstxt.org, llms.txt is a lightweight Markdown index designed to give AI inference engines and coding assistants a clean, concise map of a site's canonical resources.

Before adding llms.txt to your site, it is essential to separate what the file actually does from common SEO myths: llms.txt is not a crawler access control file like robots.txt, it is not a replacement for sitemap.xml, and Google Search does not use it to rank pages or generate AI Overviews.

Why llms.txt was proposed: context windows vs HTML boilerplate#

Traditional search engine crawlers like Googlebot index millions of pages ahead of time into an inverted index and knowledge graph. Large language models used at inference time—such as IDE coding assistants (Cursor, Windsurf, Claude Code), RAG agents, or user-prompted web readers—face a different constraint: finite context windows and token noise.

When an inference agent fetches a raw HTML page, 60% to 85% of the bytes often consist of navigation menus, inline scripts, CSS class names, SVG icons, cookie banners, and footer links. Parsing that markup wastes context tokens and can obscure API signatures or factual definitions.

The llms.txt proposal solves two problems for inference-time tools:

1. Curated prioritization (/llms.txt): Unlike sitemap.xml, which is a machine-readable URL inventory for search crawlers (often spanning thousands of canonical URLs or segmented sitemaps with zero prose context), /llms.txt is a concise index of only the essential pages, grouped by topic, with a one-sentence description of what each link contains.
2. Clean Markdown delivery (per-page .md alternates and optional /llms-full.txt): The llmstxt.org specification recommends serving a clean Markdown version of individual pages when .md is appended to the URL (or index.html.md for directory URLs) so agents can fetch details selectively. As an unofficial ecosystem convention outside the core llmstxt.org specification, some documentation portals also publish /llms-full.txt to concatenate core documentation into a single file.

Official llms.txt file structure and syntax#

According to the llmstxt.org specification, an llms.txt file sits at the root path /llms.txt (or optionally a subpath for a specific subproject) and uses strict, deterministic Markdown formatting so both humans and simple parsers can read it:

1. An H1 heading (# Name) — The only mandatory element, stating the name of the project, brand, or site.
2. A blockquote (> Summary) — A concise summary of the site or project and what it provides.
3. Zero or more descriptive body paragraphs or lists — Important operational notes, conventions, or how to interpret the site's content (no headings allowed in this block).
4. Zero or more H2 sections (## Section Name) — Each optional H2 section contains a Markdown bulleted list of links in the format - [Link Title](https://example.com/url/): Optional concise description.
5. An optional ## Optional section — A specially named H2 section whose links may be skipped by an LLM loader when assembling a smaller, token-constrained context window.

Production /llms.txt example#

# Perfect SEO Tools

> Free browser-based technical SEO utilities, schema generators, and primary-sourced guides on crawling, indexing, Core Web Vitals, and structured data.

All tools run client-side in the browser without storing private site data. Guides cite official Google Search Central, W3C, and IETF RFC specifications.

## Core Technical SEO Guides

- [Technical SEO Audit Guide](https://www.perfectseotools.com/blog/technical-seo-audit-guide/): 10-step workflow for auditing HTTP responses, robots.txt, sitemaps, canonicals, and Core Web Vitals.
- [Robots.txt Guide](https://www.perfectseotools.com/blog/robots-txt/): RFC 9309 rule precedence, wildcards, HTTP status codes, and AI crawler user-agent tokens.
- [Canonical Tag Guide](https://www.perfectseotools.com/blog/canonical-tag/): How Google selects canonical URLs and common mistakes that split indexing signals.
- [XML Sitemap Best Practices](https://www.perfectseotools.com/blog/xml-sitemap-best-practices/): Sitemaps.org 0.9 limits, lastmod accuracy, and sitemap index architecture.

## Interactive SEO Tools

- [Robots.txt Generator](https://www.perfectseotools.com/tools/robots-manager/): Build user-agent groups for search engines and AI crawlers.
- [JSON-LD Schema Generator](https://www.perfectseotools.com/tools/jsonld-generator/): Generate Article, FAQPage, LocalBusiness, and Product JSON-LD markup.
- [Canonical Tag Manager](https://www.perfectseotools.com/tools/canonical-manager/): Draft and validate self-referencing and cross-domain canonical link tags.

## Optional

- [AI Crawler Policy](https://www.perfectseotools.com/ai-crawler-policy/): Details on our crawler access rules and attribution standards.
- [About Perfect SEO Tools](https://www.perfectseotools.com/about/): Source verification, fact-checking, and correction standards.

How llms.txt compares with robots.txt and sitemap.xml#

Site owners frequently confuse the roles of these three root files. They serve completely separate functions:

If your goal is to communicate crawl or training opt-out requests to compliant AI model crawlers (GPTBot, ClaudeBot, Google-Extended) or AI search crawlers (OAI-SearchBot, Claude-SearchBot), you must configure /robots.txt (paired with server/WAF rules if you require hard technical blocking)—never /llms.txt. Use our robots.txt generator and read our robots.txt guide to configure AI crawler tokens properly.

Does llms.txt help you rank in Google AI Overviews or ChatGPT Search?#

In Google Search Central's documentation on AI features and your website, Google states that no special files, custom text formats, or non-standard schemas are required to appear in Google AI Overviews or AI Mode; standard Google Search crawling, indexing, and helpful content fundamentals apply. Similarly, ChatGPT Search (OAI-SearchBot) and Perplexity retrieve live web pages through standard search indices and HTML crawls governed by robots.txt.

Where /llms.txt and /llms-full.txt provide genuine, immediate utility today is for:

• Developer documentation, SDKs, and APIs (adopted by platforms such as Cloudflare, Stripe, Vercel, and Anthropic) where users paste /llms-full.txt into coding assistants or RAG pipelines.
• Structured editorial and research sites that want an automatically generated, zero-fluff Markdown catalog of their published guides and tools.

If you want to improve your visibility in Google AI Overviews and citation-based answer engines, focus first on crawlable HTML, clear direct answers backed by primary sources, accurate <lastmod> dates in your XML sitemaps, and our AI Overviews citation strategy guide.

Best practices if you publish /llms.txt#

1. Generate /llms.txt automatically at build time: Never maintain /llms.txt by hand, or it will quickly accumulate broken 404 links and outdated descriptions. Wire it to the same content index that builds your sitemap so draft or excluded articles never leak into /llms.txt.
2. Enforce strict canonical URL parity: Every URL listed in /llms.txt must match the exact protocol, host, and trailing-slash format of your rel="canonical" tags and sitemap.xml entries. Test your canonicals with our canonical tag manager and organize your sitemaps with our sitemap index generator.
3. Keep descriptions factual and concise: Write one clear sentence per link stating the scope and facts covered on that page rather than promotional slogans.

Frequently asked questions#

Can I put Disallow or Allow rules inside /llms.txt?#

No. The llms.txt format has no directive syntax for blocking or allowing crawlers, and automated bots do not check /llms.txt for permission before crawling your site. Always place crawler access directives in /robots.txt per RFC 9309.

What is the difference between /llms.txt and /llms-full.txt?#

/llms.txt is the only file formally defined by the llmstxt.org specification: a compact index containing your site title, summary, and annotated Markdown links (which the spec recommends pairing with per-page .md alternates so AI clients can fetch individual pages selectively). /llms-full.txt is an unofficial community convention—not part of the llmstxt.org standard—used by some developer documentation sites to concatenate their pages into one large Markdown file for manual copy-pasting or bulk RAG ingestion.

Will missing an llms.txt file hurt my SEO?#

No. Missing /llms.txt has zero negative impact on Google Search rankings, Google AI Overviews, Bing, or standard web indexing. Treat it as a helpful progressive enhancement for documentation and AI-assisted research rather than a mandatory SEO requirement.

Sources#

• llmstxt.org: The /llms.txt file specification by Jeremy Howard (Answer.AI)
• Google Search Central: AI features and your website
• RFC 9309: Robots Exclusion Protocol