--- title: "What Is llms.txt? Format and Real Examples | Scrape.ly" description: "What is llms.txt? The format rules, real examples from Anthropic, Stripe and Cloudflare, llms.txt vs robots.txt, llms-full.txt, and how to create one." url: https://scrape.ly/blog/what-is-llms-txt markdown: https://scrape.ly/blog/what-is-llms-txt.md --- [AI agents](https://scrape.ly/blog/category/ai-agents.md) # What Is llms.txt? Format, Real Examples and How to Create One llms.txt is a Markdown file at the root of a website that gives AI agents a short, curated map of its content. The format rules from the v2 proposal, real files from Anthropic, OpenAI, Stripe and Cloudflare, llms-full.txt, Markdown twins, and how to create one in 11 seconds. [Querystone Actor publisher on the Scrape.ly Store](https://scrape.ly/blog/author/querystone.md) - Published Oct 23, 2026 - Reading time 11 min read ![Pixel-art robot with an orange >\_ on its terminal head, an AI sparkle in a speech bubble above it, reaches out to a standing text file with a folded corner whose dark plate reads LLMS, next to a web page](https://scrape.ly/media/what-is-llms-txt.png?w=1080) What is llms.txt? It's a Markdown file at `/llms.txt` on a website that gives AI agents a short, curated map of the site: one heading with the site's name, a one-paragraph summary, and lists of links to the pages worth reading, ideally as clean Markdown. Instead of crawling every HTML page, an agent reads one small file and fetches only what it needs. The idea comes from a proposal by Jeremy Howard at [llmstxt.org](https://llmstxt.org/), first published on 3 September 2024 and revised as v2 in August 2026. This guide covers the exact format rules, real files we fetched on 23 October 2026, how llms.txt differs from robots.txt and sitemap.xml, what llms-full.txt and Markdown twins are, and how to create your own file, with a real run that built one in 11 seconds for $0.026. ## llms.txt at a glance | | | | ------------------ | ---------------------------------------------------------------------------------------- | | What it is | A Markdown index of a site (or part of it) for language models and agents | | Where it lives | `/llms.txt` at the site root, or at a subpath such as `/docs/llms.txt` | | Required content | Only an H1 with the site or project name | | Usual content | A blockquote summary, a few notes, then H2 sections with link lists | | Who reads it | Agents at the moment they need information, for example a coding agent looking up an API | | What it doesn't do | Control access or block crawlers (that's robots.txt) | | Status | An open proposal, not a formal standard; v2 published August 2026 | ## What is llms.txt for, and what changed in v2? Web pages are built for people. An HTML page wraps its information in navigation, scripts, cookie banners and ads, and an agent with a limited context window pays for every one of those tokens. The proposal's answer is a small file, written once, that says what the site is and where the useful material lives. The detail stays behind the links and is fetched only when needed. The v2 revision describes what two years of adoption taught. According to the [changes page](https://llmstxt.org/changes.html), it: - **Adds discovery through link relations.** `rel="alternate" type="text/markdown"` points from a page to its Markdown version, and `rel="describedby"` points to the llms.txt that covers it, either as HTML `` elements or as an HTTP `Link:` header. - **Defines path scoping.** A file covers the pages under its path, and when several apply, agents use the most specific one. So `/docs/llms.txt` can describe only your documentation. - **Accepts both Markdown URL forms**: `page.html.md` (append `.md`) and `page.md` (replace the extension). - **Drops the context-expansion tooling** from v1, and with it the special meaning of the `Optional` section. Optional sections are still allowed as a convention for secondary links. The proposal is explicit about how the file is used: "on demand, when an agent needs information about a topic while assisting a user", mostly at inference time rather than for training. It names software documentation as the heaviest use, because coding agents follow these files to find API references. ## The llms.txt format, section by section A file that follows the spec contains these parts, in this order: 1. An optional byte-order mark. 2. **An H1 with the name of the project or site.** This is the only required section. 3. A blockquote with a short summary that holds the key information needed to understand the rest of the file. 4. Zero or more Markdown sections of any type except headings, such as paragraphs or lists with more detail. 5. Zero or more sections under H2 headings, each a "file list": a Markdown list where every item is a required link `[name](url)`, optionally followed by `:` and notes. The proposal's own skeleton: ```plaintext # Title > Optional description goes here Optional details go here ## Section name [Link title](https://link_url): Optional link details ## Optional [Link title](https://link_url) ``` Because the structure is fixed, a parser or a regex can read the file as reliably as a model can. The proposal's writing tips are short: use concise, clear language, give each link a brief description, avoid unexplained jargon, and test the file by asking an agent questions about your site with only the llms.txt as a starting point. ## A real llms.txt example: Scrape.ly Here is an excerpt of [Scrape.ly's llms.txt](https://scrape.ly/llms.txt) as it was served on 23 October 2026 (`[...]` marks lines we left out). The full file is 47 KB, because it lists every section of the site: ```plaintext # Scrape.ly > Scrape.ly is a web data platform powered by Evomi: readymade scrapers (Actors) in the Scrape.ly Store, a cloud runtime for custom scrapers, an HTTP API with JavaScript and Python SDKs, residential and datacenter proxies, and a managed data service. [...] Every page on this site has a Markdown version: append `.md` to its path (`/index.md` for the home page) or request the page with `Accept: text/markdown`. Links inside the Markdown point at other Markdown pages, so an agent can crawl the whole site without parsing HTML. AI agents: to connect the Scrape.ly MCP server for your user and run an Actor, read [agents.md](https://scrape.ly/agents.md). [...] ## Products Scrapers, proxies, APIs, and the template library. [Scrape.ly Store](https://scrape.ly/store.md): Browse ready-made scrapers. [...] [MCP Server](https://scrape.ly/mcp.md): Connect AI agents to Scrape.ly. ``` Every part of the spec is there: the H1, the blockquote, plain paragraphs with instructions for agents, and H2 file lists whose links point at Markdown pages. The details paragraph is doing real work too. It tells an agent how to get the Markdown version of any page, which saves it from guessing. ## llms.txt vs robots.txt vs sitemap.xml The three files sit side by side at the site root, and they answer different questions. | | llms.txt | robots.txt | sitemap.xml | | -------------------------- | -------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------- | | Question it answers | What is this site, and which pages should an agent read? | What may automated clients access? | Which URLs exist for indexing? | | Audience | Language models and agents | Crawlers and bots | Search engines | | Format | Markdown | Plain-text directives (`User-Agent`, `Allow`, `Disallow`) | XML | | Coverage | Curated: the pages that matter, with notes | Rules, not pages | Every indexable page | | Links to Markdown versions | Yes, recommended | No | Usually not | | Can block anything | No | It states what's allowed; compliant bots follow it | No | | Location | `/llms.txt` or any subpath | Root only | Root, or wherever robots.txt points | The proposal spells out why a sitemap isn't a substitute: it often lacks the Markdown versions, never lists helpful external URLs, and usually covers far more content than fits in a context window. And llms.txt doesn't replace robots.txt. If you want to keep a bot out, robots.txt is still the file that says so. ## llms-full.txt and Markdown twins Two companions usually travel with llms.txt. **llms-full.txt** is one file with the full content of the site as Markdown, so an agent can load everything in a single request. It isn't part of the llmstxt.org proposal; it's a convention that documentation platforms adopted. [Mintlify's docs](https://www.mintlify.com/docs/ai/llmstxt) describe it as combining "your entire documentation site into a single file", and Mintlify generates and hosts both files automatically for the docs sites it builds. Sizes vary wildly. On 23 October 2026, Scrape.ly's `llms-full.txt` was 1.85 MB, and [Cloudflare's](https://developers.cloudflare.com/llms-full.txt) was 69 MB, far beyond any context window. That's why the index file matters more: an agent reads the small map first and follows only the links it needs. **Markdown twins** are Markdown versions of individual pages at a predictable URL, which v2 of the proposal recommends. Scrape.ly serves one for every page, three ways: ```shell # 1. Append .md to the path curl https://scrape.ly/pricing.md # 2. Ask for Markdown on the normal URL curl -H "Accept: text/markdown" https://scrape.ly/pricing # 3. Discover it from the HTML page's response headers curl -sI https://scrape.ly/pricing grep -i '^link' # link: ; rel="alternate"; type="text/markdown" ``` The third one is the v2 discovery mechanism in action: any agent that reads response headers learns where the clean version of the page lives, without guessing. ## Who uses llms.txt The proposal says thousands of sites publish one. We checked these on 23 October 2026 with a plain `curl`, and each returned an llms.txt with an H1 and link lists: | Publisher | File | Size | Note | | ------------------- | ---------------------------------------- | ----- | ------------------------------------------------------------- | | Anthropic | `docs.anthropic.com/llms.txt` | 90 KB | Redirects to `platform.claude.com/llms.txt` | | OpenAI | `developers.openai.com/llms.txt` | 6 KB | Developer docs hub | | Google (Gemini API) | `ai.google.dev/gemini-api/docs/llms.txt` | 38 KB | Path-scoped: covers the Gemini API docs only | | Stripe | `docs.stripe.com/llms.txt` | 98 KB | Opens with instructions for agents installing Stripe packages | | Cloudflare | `developers.cloudflare.com/llms.txt` | 17 KB | Plus a 69 MB llms-full.txt | | Vercel | `vercel.com/llms.txt` | 5 KB | Site-wide | | Perplexity | `docs.perplexity.ai/llms.txt` | 40 KB | API docs | | uv (Astral) | `docs.astral.sh/uv/llms.txt` | 5 KB | Path-scoped | | FastHTML | `fastht.ml/docs/llms.txt` | 5 KB | The proposal's own example | The pattern is clear: developer documentation leads, and several publishers scope the file to their docs path instead of the whole domain. ## Who actually reads llms.txt Publishing a file is not the same as having it read, and it's worth being precise here. The proposal positions llms.txt as something agents fetch while helping a user, and Chrome's Lighthouse now checks sites for one as part of its agentic browsing audits. The largest measurement so far is an [Ahrefs study](https://ahrefs.com/blog/llmstxt-study/) of 137,000 domains: 28% published an llms.txt, but 97% of those files got no requests at all in May 2026. Of the fetches that did happen, 19.5% came from named AI tools, with GPTBot first and Claude-Code second. The same article reports that Google's guidance on generative AI search says files like llms.txt aren't needed to appear there. So treat llms.txt as what it was designed to be: a map for agents doing real work on your site, such as coding agents reading your docs and assistants answering questions about your product. It's not a search ranking trick. It takes minutes to publish, costs nothing to serve, and pays off every time an agent lands on your docs and gets the right page on the first try. ## How to create an llms.txt file You have three options. **Write it by hand.** For a small site, the skeleton above plus ten well-described links is a complete file. Write the blockquote as if it were the only thing the agent reads. **Let your docs platform do it.** The proposal lists platforms that generate one automatically, including Mintlify, GitBook, Yoast SEO, AIOSEO and Wix, plus plugins for VitePress, Docusaurus and Drupal. **Generate it from your live site.** Scrape.ly's [llms.txt Generator](https://scrape.ly/store/querystone/llms-txt-generator.md) Actor finds your pages through the sitemap and navigation, writes a title and note for each, groups them into sections, and saves both llms.txt and llms-full.txt with download links. It works on any site, including ones you don't control, which is handy for building a context file for a library you use. On 23 October 2026 we ran it on the HTTPX documentation, which had no llms.txt of its own, with a 5-page limit: ```json { "startUrls": ["https://www.python-httpx.org/"], "maxPagesPerSite": 5, "includeFullText": true } ``` The run finished in 11 seconds. It found the sitemap (23 URLs), picked 5 pages across the site's sections, and produced this llms.txt, verbatim: ```plaintext # HTTPX > A nextgeneration HTTP client for Python. ## Main pages [QuickStart](https://www.python-httpx.org/quickstart/) ## Advanced [Clients](https://www.python-httpx.org/advanced/clients/) [Authentication](https://www.python-httpx.org/advanced/authentication/) [SSL](https://www.python-httpx.org/advanced/ssl/) ``` The home page became the H1 and the blockquote (from its meta description), and the four other pages became two sections. The links have no notes because those pages have no meta description of their own. That's the one edit worth making by hand before you upload: add a short note after each link. The llms-full.txt from the same run held the main content of all 5 pages as Markdown, 42 KB, with menus and footers removed. The log: ```plaintext 20261023T03:03:16.161Z [scrapely.actor] INFO Generating llms.txt for 1 site(s), up to 5 page(s) each, via datacenter proxy. 20261023T03:03:17.831Z [scrapely.actor] INFO Download links are signed. 20261023T03:03:18.220Z [scrapely.actor] INFO pythonhttpx.org: llms.txt with 5 page(s) in 2 section(s) 20261023T03:03:18.221Z [scrapely.actor] INFO Generated llms.txt for 1 site(s) from 5 page(s). 0.3 MB downloaded. ``` It was charged $0.026: $0.01 for the generated site, 5 × $0.003 for the processed pages, and $0.001 of compute, storage and proxy traffic. A 30-page file for your own site would cost about $0.10 in Actor fees, well inside the free plan's $5 of monthly credit. For a documentation-only file, set **Only URLs matching** to `/docs/`, as the [Crawlee Python docs preset](https://scrape.ly/store/querystone/llms-txt-generator/presets/docs-only-custom-header.md) does. Scrape.ly Store Create your llms.txt in one run Get llms.txt and llms-full.txt for any site, with sections and clean Markdown: $0.01 per site plus $0.003 per page. [Try it free](https://scrape.ly/store/querystone/llms-txt-generator.md) [Browse the Store](https://scrape.ly/store.md) To regenerate the file from a script or a deploy pipeline, call the Actor through the API. This run takes seconds, so the synchronous endpoint returns the dataset directly; the `site` row carries the full text in `llmsTxt`: ```shell curl -X POST "https://api.scrape.ly/v2/actors/querystone~llms-txt-generator/run-sync-get-dataset-items" -H "Authorization: Bearer $SCRAPELY_TOKEN" -H "Content-Type: application/json" -d '{"startUrls": ["https://example.com"], "maxPagesPerSite": 30}' ``` Or in Python, saving both files from the run's key-value store: ```python import os from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") run = client.actor("querystone/llms-txt-generator").call( run_input={"startUrls": ["https://example.com"], "maxPagesPerSite": 30} ) store = client.key_value_store(run.default_key_value_store_id) for key in ["llms.txt", "llms-full.txt"]: record = store.get_record(key) if record: open(key, "w").write(record["value"]) ``` Schedule it weekly and your llms.txt never drifts from the site. An AI agent can run it too: with the [Scrape.ly MCP server](https://scrape.ly/mcp.md) connected, ask Claude or Cursor to "use scrapely to run querystone/llms-txt-generator on https\://example.com with 30 pages and show me the llms.txt". ## Where to put it and how to check it - **Location.** Serve it at `https://yoursite.com/llms.txt`. For docs on a shared host, `/docs/llms.txt` works too and covers everything under `/docs/`. - **Content type.** Plain text or Markdown are both common: Stripe and Crawlee serve `text/markdown`, Anthropic and Vercel `text/plain`. - **Discovery.** Add `Link: ; rel="describedby"` to your responses, as v2 recommends, so agents find the file from any page. - **Links.** Point them at Markdown versions of your pages where you have them. - **Freshness.** Regenerate it when your navigation changes. A stale map sends agents to dead pages. - **Test.** Give an agent only the llms.txt URL and ask three questions a customer would ask. If it can't answer, improve the notes. ## How Scrape.ly does it Scrape.ly is built for agents from the ground up, and its site follows the v2 proposal end to end: an [llms.txt](https://scrape.ly/llms.txt) with sections and notes, an llms-full.txt, a Markdown twin for every page (by `.md` URL or `Accept: text/markdown`), `rel="alternate"` headers that point to them, and an [agents.md](https://scrape.ly/agents.md) that tells an agent how to connect the MCP server and run a scraper. The same thinking runs through the product. The [MCP server](https://scrape.ly/mcp.md) lets an agent search the [Store](https://scrape.ly/store.md) of 568 ready-made Actors, run one with a dollar cap, and read the results page by page. When an agent needs a whole site as clean Markdown rather than an index, the [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) does that at $0.50 per 1,000 pages. For how the hosted servers compare, see our guide to the [best MCP servers for web scraping](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md). ## FAQ ### Does llms.txt help SEO? There's no evidence that it affects search rankings, and Google's guidance on generative AI search, as reported by Ahrefs, says such files aren't needed to appear there. Its value is for agents that read your site while helping a user, such as coding assistants reading your docs. ### Is llms.txt an official standard? No. It's an open proposal by Jeremy Howard, maintained at llmstxt.org with a public GitHub repository, now in its second version. It's widely implemented by documentation platforms and large developer sites, but no standards body has adopted it. ### What is the difference between llms.txt and llms-full.txt? llms.txt is a short index with links. llms-full.txt is a single file with the full content of the site as Markdown. The first fits in a context window and lets an agent choose what to read; the second can run to megabytes. llms-full.txt is a platform convention, not part of the proposal. ### Can llms.txt block AI crawlers? No. It controls nothing. To tell crawlers, including AI crawlers, what they may access, use robots.txt. ### Where should I put my llms.txt file? At the root, `https://yoursite.com/llms.txt`, for a site-wide map. If you only control a path, or want a file just for your docs, put it at that path, for example `/docs/llms.txt`. It then covers the pages under that path. ### How do I create an llms.txt file quickly? Write the H1, a one-paragraph blockquote and a few H2 link lists by hand, or generate one from your live site with a crawler that reads your sitemap. Our run built a 5-page file in 11 seconds for $0.026; check the notes on each link before you publish it. About the author [Querystone](https://scrape.ly/blog/author/querystone.md) Actor publisher on the Scrape.ly Store Search results, AI answers, web pages and market feeds as structured data. For SEO teams, developers and AI builders, plus finance, crypto and sports data for analysts. A Scrapely studio. - [Website](https://scrape.ly/users/querystone.md) Keep reading ## More in AI agents - [![Pixel-art robot with an orange >\_ on its terminal head pulls a page of text out of a monitor showing the cyan Crawl4AI logo above lines of code, and drops it into a crate](https://scrape.ly/media/crawl4ai-guide.png?w=750)](https://scrape.ly/blog/crawl4ai-guide.md) ### [Crawl4AI Tutorial: Setup, Examples and a Hosted Alternative](https://scrape.ly/blog/crawl4ai-guide.md) [A tested Crawl4AI tutorial for v0.9.5: install it, crawl a page to Markdown, extract JSON with CSS selectors or an LLM, deep crawl a site, run it in Docker or over MCP, and see when a hosted crawler saves you the infrastructure.](https://scrape.ly/blog/crawl4ai-guide.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from an office tower with the blue LinkedIn logo on its sign into a server rack labelled MCP](https://scrape.ly/media/linkedin-mcp.png?w=750)](https://scrape.ly/blog/linkedin-mcp.md) ### [LinkedIn MCP: Give Claude, Cursor and ChatGPT LinkedIn Data Without Your Login](https://scrape.ly/blog/linkedin-mcp.md) [A LinkedIn MCP server lets your AI agent pull company pages, job listings, posts and ads into the chat. Set one up in Claude, Cursor or ChatGPT in a minute, without handing over your LinkedIn session, and see the exact tool calls and what came back.](https://scrape.ly/blog/linkedin-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a feed of posts headed by the orange Reddit logo into a server rack labelled MCP](https://scrape.ly/media/reddit-mcp.png?w=750)](https://scrape.ly/blog/reddit-mcp.md) ### [Reddit MCP: Connect Claude or Cursor to Reddit Data Without API Keys](https://scrape.ly/blog/reddit-mcp.md) [A Reddit MCP server lets Claude, Cursor or ChatGPT search Reddit, read subreddits and pull comment threads mid-conversation. Set one up in a minute without Reddit API credentials, and see the exact tool calls, a real run and what it cost.](https://scrape.ly/blog/reddit-mcp.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.