What Is llms.txt? Format, Real Examples and How to Create One

What is llms.txt? It's a Markdown file at /llms.txt on a website that gives AI agents a short, curated map of the site: one heading with the site's name, a one-paragraph summary, and lists of links to the pages worth reading, ideally as clean Markdown. Instead of crawling every HTML page, an agent reads one small file and fetches only what it needs.

The idea comes from a proposal by Jeremy Howard at llmstxt.org, first published on 3 September 2024 and revised as v2 in August 2026. This guide covers the exact format rules, real files we fetched on 23 October 2026, how llms.txt differs from robots.txt and sitemap.xml, what llms-full.txt and Markdown twins are, and how to create your own file, with a real run that built one in 11 seconds for $0.026.

llms.txt at a glance

What is llms.txt for, and what changed in v2?

Web pages are built for people. An HTML page wraps its information in navigation, scripts, cookie banners and ads, and an agent with a limited context window pays for every one of those tokens. The proposal's answer is a small file, written once, that says what the site is and where the useful material lives. The detail stays behind the links and is fetched only when needed.

The v2 revision describes what two years of adoption taught. According to the changes page, it:

• Adds discovery through link relations. rel="alternate" type="text/markdown" points from a page to its Markdown version, and rel="describedby" points to the llms.txt that covers it, either as HTML <link> elements or as an HTTP Link: header.
• Defines path scoping. A file covers the pages under its path, and when several apply, agents use the most specific one. So /docs/llms.txt can describe only your documentation.
• Accepts both Markdown URL forms: page.html.md (append .md) and page.md (replace the extension).
• Drops the context-expansion tooling from v1, and with it the special meaning of the Optional section. Optional sections are still allowed as a convention for secondary links.

The proposal is explicit about how the file is used: "on demand, when an agent needs information about a topic while assisting a user", mostly at inference time rather than for training. It names software documentation as the heaviest use, because coding agents follow these files to find API references.

The llms.txt format, section by section

A file that follows the spec contains these parts, in this order:

1. An optional byte-order mark.
2. An H1 with the name of the project or site. This is the only required section.
3. A blockquote with a short summary that holds the key information needed to understand the rest of the file.
4. Zero or more Markdown sections of any type except headings, such as paragraphs or lists with more detail.
5. Zero or more sections under H2 headings, each a "file list": a Markdown list where every item is a required link [name](url), optionally followed by : and notes.

The proposal's own skeleton:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

Because the structure is fixed, a parser or a regex can read the file as reliably as a model can. The proposal's writing tips are short: use concise, clear language, give each link a brief description, avoid unexplained jargon, and test the file by asking an agent questions about your site with only the llms.txt as a starting point.

A real llms.txt example: Scrape.ly

Here is an excerpt of Scrape.ly's llms.txt as it was served on 23 October 2026 ([...] marks lines we left out). The full file is 47 KB, because it lists every section of the site:

# Scrape.ly

> Scrape.ly is a web data platform powered by Evomi: ready-made scrapers (Actors) in the Scrape.ly Store, a cloud runtime for custom scrapers, an HTTP API with JavaScript and Python SDKs, residential and datacenter proxies, and a managed data service.

[...]

Every page on this site has a Markdown version: append `.md` to its path (`/index.md` for the home page) or request the page with `Accept: text/markdown`. Links inside the Markdown point at other Markdown pages, so an agent can crawl the whole site without parsing HTML.

AI agents: to connect the Scrape.ly MCP server for your user and run an Actor, read [agents.md](https://scrape.ly/agents.md).

[...]

## Products

Scrapers, proxies, APIs, and the template library.

- [Scrape.ly Store](https://scrape.ly/store.md): Browse ready-made scrapers.
[...]
- [MCP Server](https://scrape.ly/mcp.md): Connect AI agents to Scrape.ly.

Every part of the spec is there: the H1, the blockquote, plain paragraphs with instructions for agents, and H2 file lists whose links point at Markdown pages. The details paragraph is doing real work too. It tells an agent how to get the Markdown version of any page, which saves it from guessing.

llms.txt vs robots.txt vs sitemap.xml

The three files sit side by side at the site root, and they answer different questions.

The proposal spells out why a sitemap isn't a substitute: it often lacks the Markdown versions, never lists helpful external URLs, and usually covers far more content than fits in a context window. And llms.txt doesn't replace robots.txt. If you want to keep a bot out, robots.txt is still the file that says so.

llms-full.txt and Markdown twins

Two companions usually travel with llms.txt.

llms-full.txt is one file with the full content of the site as Markdown, so an agent can load everything in a single request. It isn't part of the llmstxt.org proposal; it's a convention that documentation platforms adopted. Mintlify's docs describe it as combining "your entire documentation site into a single file", and Mintlify generates and hosts both files automatically for the docs sites it builds. Sizes vary wildly. On 23 October 2026, Scrape.ly's llms-full.txt was 1.85 MB, and Cloudflare's was 69 MB, far beyond any context window. That's why the index file matters more: an agent reads the small map first and follows only the links it needs.

Markdown twins are Markdown versions of individual pages at a predictable URL, which v2 of the proposal recommends. Scrape.ly serves one for every page, three ways:

# 1. Append .md to the path
curl https://scrape.ly/pricing.md

# 2. Ask for Markdown on the normal URL
curl -H "Accept: text/markdown" https://scrape.ly/pricing

# 3. Discover it from the HTML page's response headers
curl -sI https://scrape.ly/pricing | grep -i '^link'
# link: </pricing.md>; rel="alternate"; type="text/markdown"

The third one is the v2 discovery mechanism in action: any agent that reads response headers learns where the clean version of the page lives, without guessing.

Who uses llms.txt

The proposal says thousands of sites publish one. We checked these on 23 October 2026 with a plain curl, and each returned an llms.txt with an H1 and link lists:

The pattern is clear: developer documentation leads, and several publishers scope the file to their docs path instead of the whole domain.

Who actually reads llms.txt

Publishing a file is not the same as having it read, and it's worth being precise here. The proposal positions llms.txt as something agents fetch while helping a user, and Chrome's Lighthouse now checks sites for one as part of its agentic browsing audits.

The largest measurement so far is an Ahrefs study of 137,000 domains: 28% published an llms.txt, but 97% of those files got no requests at all in May 2026. Of the fetches that did happen, 19.5% came from named AI tools, with GPTBot first and Claude-Code second. The same article reports that Google's guidance on generative AI search says files like llms.txt aren't needed to appear there.

So treat llms.txt as what it was designed to be: a map for agents doing real work on your site, such as coding agents reading your docs and assistants answering questions about your product. It's not a search ranking trick. It takes minutes to publish, costs nothing to serve, and pays off every time an agent lands on your docs and gets the right page on the first try.

How to create an llms.txt file

You have three options.

Write it by hand. For a small site, the skeleton above plus ten well-described links is a complete file. Write the blockquote as if it were the only thing the agent reads.

Let your docs platform do it. The proposal lists platforms that generate one automatically, including Mintlify, GitBook, Yoast SEO, AIOSEO and Wix, plus plugins for VitePress, Docusaurus and Drupal.

Generate it from your live site. Scrape.ly's llms.txt Generator Actor finds your pages through the sitemap and navigation, writes a title and note for each, groups them into sections, and saves both llms.txt and llms-full.txt with download links. It works on any site, including ones you don't control, which is handy for building a context file for a library you use.

On 23 October 2026 we ran it on the HTTPX documentation, which had no llms.txt of its own, with a 5-page limit:

{
  "startUrls": ["https://www.python-httpx.org/"],
  "maxPagesPerSite": 5,
  "includeFullText": true
}

The run finished in 11 seconds. It found the sitemap (23 URLs), picked 5 pages across the site's sections, and produced this llms.txt, verbatim:

# HTTPX

> A next-generation HTTP client for Python.

## Main pages

- [QuickStart](https://www.python-httpx.org/quickstart/)

## Advanced

- [Clients](https://www.python-httpx.org/advanced/clients/)
- [Authentication](https://www.python-httpx.org/advanced/authentication/)
- [SSL](https://www.python-httpx.org/advanced/ssl/)

The home page became the H1 and the blockquote (from its meta description), and the four other pages became two sections. The links have no notes because those pages have no meta description of their own. That's the one edit worth making by hand before you upload: add a short note after each link. The llms-full.txt from the same run held the main content of all 5 pages as Markdown, 42 KB, with menus and footers removed. The log:

2026-10-23T03:03:16.161Z [scrapely.actor] INFO  Generating llms.txt for 1 site(s), up to 5 page(s) each, via datacenter proxy.
2026-10-23T03:03:17.831Z [scrapely.actor] INFO  Download links are signed.
2026-10-23T03:03:18.220Z [scrapely.actor] INFO  python-httpx.org: llms.txt with 5 page(s) in 2 section(s)
2026-10-23T03:03:18.221Z [scrapely.actor] INFO  Generated llms.txt for 1 site(s) from 5 page(s). 0.3 MB downloaded.

It was charged $0.026: $0.01 for the generated site, 5 × $0.003 for the processed pages, and $0.001 of compute, storage and proxy traffic. A 30-page file for your own site would cost about $0.10 in Actor fees, well inside the free plan's $5 of monthly credit. For a documentation-only file, set Only URLs matching to /docs/, as the Crawlee Python docs preset does.

To regenerate the file from a script or a deploy pipeline, call the Actor through the API. This run takes seconds, so the synchronous endpoint returns the dataset directly; the site row carries the full text in llmsTxt:

curl -X POST "https://api.scrape.ly/v2/actors/querystone~llms-txt-generator/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://example.com"], "maxPagesPerSite": 30}'

Or in Python, saving both files from the run's key-value store:

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

run = client.actor("querystone/llms-txt-generator").call(
    run_input={"startUrls": ["https://example.com"], "maxPagesPerSite": 30}
)
store = client.key_value_store(run.default_key_value_store_id)
for key in ["llms.txt", "llms-full.txt"]:
    record = store.get_record(key)
    if record:
        open(key, "w").write(record["value"])

Schedule it weekly and your llms.txt never drifts from the site. An AI agent can run it too: with the Scrape.ly MCP server connected, ask Claude or Cursor to "use scrapely to run querystone/llms-txt-generator on https://example.com with 30 pages and show me the llms.txt".

Where to put it and how to check it

• Location. Serve it at https://yoursite.com/llms.txt. For docs on a shared host, /docs/llms.txt works too and covers everything under /docs/.
• Content type. Plain text or Markdown are both common: Stripe and Crawlee serve text/markdown, Anthropic and Vercel text/plain.
• Discovery. Add Link: </llms.txt>; rel="describedby" to your responses, as v2 recommends, so agents find the file from any page.
• Links. Point them at Markdown versions of your pages where you have them.
• Freshness. Regenerate it when your navigation changes. A stale map sends agents to dead pages.
• Test. Give an agent only the llms.txt URL and ask three questions a customer would ask. If it can't answer, improve the notes.

How Scrape.ly does it

Scrape.ly is built for agents from the ground up, and its site follows the v2 proposal end to end: an llms.txt with sections and notes, an llms-full.txt, a Markdown twin for every page (by .md URL or Accept: text/markdown), rel="alternate" headers that point to them, and an agents.md that tells an agent how to connect the MCP server and run a scraper.

The same thinking runs through the product. The MCP server lets an agent search the Store of 568 ready-made Actors, run one with a dollar cap, and read the results page by page. When an agent needs a whole site as clean Markdown rather than an index, the Website Content Crawler does that at $0.50 per 1,000 pages. For how the hosted servers compare, see our guide to the best MCP servers for web scraping.

FAQ

Does llms.txt help SEO?

There's no evidence that it affects search rankings, and Google's guidance on generative AI search, as reported by Ahrefs, says such files aren't needed to appear there. Its value is for agents that read your site while helping a user, such as coding assistants reading your docs.

Is llms.txt an official standard?

No. It's an open proposal by Jeremy Howard, maintained at llmstxt.org with a public GitHub repository, now in its second version. It's widely implemented by documentation platforms and large developer sites, but no standards body has adopted it.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a short index with links. llms-full.txt is a single file with the full content of the site as Markdown. The first fits in a context window and lets an agent choose what to read; the second can run to megabytes. llms-full.txt is a platform convention, not part of the proposal.

Can llms.txt block AI crawlers?

No. It controls nothing. To tell crawlers, including AI crawlers, what they may access, use robots.txt.

Where should I put my llms.txt file?

At the root, https://yoursite.com/llms.txt, for a site-wide map. If you only control a path, or want a file just for your docs, put it at that path, for example /docs/llms.txt. It then covers the pages under that path.

How do I create an llms.txt file quickly?

Write the H1, a one-paragraph blockquote and a few H2 link lists by hand, or generate one from your live site with a crawler that reads your sitemap. Our run built a 5-page file in 11 seconds for $0.026; check the notes on each link before you publish it.