llms.txt Generator

🤖 llms.txt Generator

Create a spec-compliant llms.txt and llms-full.txt for any website in one run. The generator finds your key pages through the sitemap and navigation, writes a clean title and note for each one, groups them into sections and saves ready-to-upload files with download links. No login, no API keys, no code.

✨ Why use llms.txt Generator?

• 📄 Follows the llms.txt format: H1 site name, a blockquote summary, then ## Sections with - [Title](url): note link lists, plus an ## Optional section for secondary pages.
• 🧭 Smart page selection: pages are picked round-robin across your sections (docs, product, blog, help...), shallow pages first, so even a 30-page limit covers the whole site instead of 30 blog posts. Old doc versions, tag and author archives and pagination are left out.
• ✂️ Clean titles and notes: site-name suffixes such as "| Acme" are stripped from titles, generic site-wide descriptions are replaced by the page's own first sentences.
• 📚 llms-full.txt included: the main content of every page as clean Markdown (menus, footers and cookie banners removed), ideal for AI assistants, RAG and agents.
• 🔗 Download links: both files are saved to storage with direct download URLs, and the llms.txt text is also in the dataset row.
• 💸 Fair pricing: a small fee per site plus a fraction of a cent per page.

🎯 Use cases

• 🌐 Website owners and marketers: publish /llms.txt so ChatGPT, Perplexity, Gemini and other AI assistants understand your site and cite the right pages.
• 🧑‍💼 SEO and GEO agencies: generate llms.txt files for every client in bulk and keep them fresh with a weekly schedule.
• 🛠️ Developer docs teams: build an llms.txt and llms-full.txt for documentation (use "Only URLs matching" /docs/).
• 🧠 AI builders: get a site's key content as one Markdown file for prompts, RAG or agents.

🚀 How to use

1. Paste one or more home pages under Websites (for example https://example.com).
2. Set Max pages per site (30 is a good start; large docs sites may want 200+).
3. Optionally set the Site name and Summary, or limit pages with Only URLs matching.
4. Click Start.
5. Download llms.txt and llms-full.txt from the links in the Generated files view (or the Output tab) and upload them to your site root, e.g. https://example.com/llms.txt.

📥 Input

{
  "startUrls": ["https://crawlee.dev/"],
  "maxPagesPerSite": 30,
  "includeFullText": true
}

📤 Output

Files (key-value store): llms-<domain>.txt and llms-full-<domain>.txt for every site, plus llms.txt and llms-full.txt when you generate a single site.

Example llms.txt (trimmed):

# Crawlee

> Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.

## JavaScript: API

- [Crawlee for JavaScript](https://crawlee.dev/js/api)
- [@crawlee/basic](https://crawlee.dev/js/api/basic-crawler)
- [@crawlee/browser](https://crawlee.dev/js/api/browser-crawler)

## JavaScript: Docs

- [Deployment guides](https://crawlee.dev/js/docs/deployment): Here you can find guides on how to deploy your crawlers to various cloud providers.
- [Examples](https://crawlee.dev/js/docs/examples)
- [Experiments](https://crawlee.dev/js/docs/experiments)

## JavaScript

- [Crawlee for JavaScript](https://crawlee.dev/js)

Dataset: one site row per website and one page row per page.

Example site row (trimmed):

{
  "type": "site",
  "site": "crawlee.dev",
  "status": "ok",
  "siteName": "Crawlee",
  "summary": "Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.",
  "pageCount": 30,
  "sectionCount": 7,
  "sections": [
    {
      "name": "JavaScript: API",
      "pageCount": 6,
      "optional": false
    },
    {
      "name": "JavaScript: Docs",
      "pageCount": 5,
      "optional": false
    },
    {
      "name": "JavaScript",
      "pageCount": 1,
      "optional": false
    }
  ],
  "llmsTxtUrl": "https://api.scrape.ly/v2/key-value-stores/<store>/records/llms-crawlee.dev.txt?signature=...",
  "llmsFullTxtUrl": "https://api.scrape.ly/v2/key-value-stores/<store>/records/llms-full-crawlee.dev.txt?signature=...",
  "llmsTxtBytes": 4394,
  "llmsFullTxtBytes": 304676,
  "existingLlmsTxt": "https://crawlee.dev/llms.txt"
}

Example page row:

{
  "type": "page",
  "site": "crawlee.dev",
  "url": "https://crawlee.dev/js/api/browser-crawler",
  "title": "@crawlee/browser | API",
  "description": null,
  "descriptionSource": null,
  "section": "JavaScript: API",
  "optional": false,
  "wordCount": 525,
  "lang": "en",
  "foundVia": "sitemap",
  "depth": 1
}

💰 Pricing

• $0.01 per site for which files are generated, plus $0.003 per page read ($3 per 1,000 pages).
• A 30-page site costs about $0.10; a 200-page docs site about $0.61. Platform usage (compute and proxy traffic) is small, typically under $0.01 per site.
• Pages that fail to load, duplicates, noindex pages and files are not charged. A site that cannot be reached is not charged. Set a maximum charge per run and the run stops cleanly when it is reached.

❓ FAQ

What is llms.txt? A proposed standard (llmstxt.org): a Markdown file at /llms.txt that tells AI assistants what a site is about and links its most useful pages. llms-full.txt contains the full text of those pages.

Is it legal? The generator reads public pages like a search engine does and respects robots.txt by default. Use it for sites you own or manage.

Do I need a login or proxy? No. Datacenter proxies are used by default and blocked pages are retried once over a residential proxy.

How are sections chosen? From the URL structure (/docs/, /blog/...), named after your navigation labels when they match, with big sections split one level deeper. Legal, careers, contact and similar pages go to ## Optional.

Can I edit the result? Yes. The files are plain Markdown: tweak them before uploading, or set the site name, summary and details in the input.

Can I schedule it or use the API? Yes. Run it weekly so your llms.txt follows new pages, or start runs and download the files through the API.

⚠️ Limits

• Pages that only render with JavaScript may have little content in llms-full.txt.
• At most 5,000 pages per site, because every page of a site is kept in memory until its llms.txt is built.
• llms-full.txt is capped at 8 MB per site; the row says when it was cut.
• Sites behind a login or a strong anti-bot wall cannot be read.
• Download links are signed and valid for 30 days; the files stay in the run's storage for as long as your storage retention keeps them.