//llms.txt Generator example\\

llms.txt for Crawlee Python docs, 8 pages

Docs-only llms.txt with a custom site name and summary.

Updated October 21, 2026

Input

The input this example runs llms.txt Generator with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

WebsitesstartUrls: https://crawlee.dev/
Max pages per sitemaxPagesPerSite: 8
Site name (H1)siteName: Crawlee for Python
Summary (blockquote)siteSummary: Crawlee for Python documentation.
Only URLs matchingincludeUrlPatterns: /python/docs/

.json

{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxPagesPerSite": 8,
  "includeUrlPatterns": [
    "/python/docs/"
  ],
  "siteName": "Crawlee for Python",
  "siteSummary": "Crawlee for Python documentation."
}

The internet is full of data.
Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.

Get Started Free Book a Demo

No credit card. No setup calls.