Sitemap Extractor

🗺️ Sitemap Extractor – All URLs from XML Sitemaps & robots.txt

Get every URL a website publishes in its sitemaps, in seconds. Paste a domain and the actor finds the sitemaps through robots.txt and the usual locations, follows sitemap indexes to the end, reads gzip and text sitemaps, and falls back to RSS/Atom feeds when a site has no sitemap. One clean row per URL with lastmod, change frequency, priority and image, video and news data. No login, no browser, and a low price per URL.

✨ Why use Sitemap Extractor?

• ⚡ Fast: sitemaps are streamed and parsed on the fly. In our tests a 4,000-URL documentation site took about 4 seconds of work and 10,000 news URLs about 8 seconds, with memory flat on large sitemaps.
• 🔎 Finds sitemaps for you: reads Sitemap: lines in robots.txt, tries /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and more, and follows nested sitemap indexes.
• 📰 Rich data: lastmod, changefreq, priority, image sitemap entries, video details (title, duration, thumbnail, player URL), Google News data (publication, title, keywords) and hreflang alternates.
• 🎯 Filters built in: include/exclude URL patterns, a "modified since/until" date filter that skips whole child sitemaps that are too old, same-domain only, and per-site limits.
• 🧹 Deduplicated: a URL that appears in several sitemaps is saved and charged once.
• 🩺 Optional status check: one HEAD request per URL adds the HTTP status code and redirect target, so you can find 404s and redirects listed in your sitemap.

🎯 Use cases

• 🤖 Feed AI and RAG pipelines: get the complete list of a site's documentation or blog pages before you crawl them.
• 📈 SEO audits: list every indexable URL, check for broken pages and redirects, compare lastmod dates, and review hreflang alternates.
• 🆕 Monitor new content: run on a schedule with "Modified since: 1 day" to get only the pages a competitor or news site added or changed.
• 🛒 E-commerce catalog discovery: collect all product or category URLs with a pattern such as /product/, then pass them to a product scraper.
• 🧭 Site migrations: export the old site's URL inventory to build redirect maps.

🚀 How to use

1. Paste one or more websites (for example https://crawlee.dev or just example.com), or direct sitemap URLs.
2. Set Max URLs (0 = everything) and, if you want, URL patterns or a date filter.
3. Click Start. Most sites finish in a few seconds.
4. Download the results as JSON, CSV, Excel or HTML, or fetch them through the API.

📥 Input

{
  "startUrls": [{ "url": "https://crawlee.dev" }, { "url": "https://www.theguardian.com/sitemaps/news.xml" }],
  "maxUrls": 5000,
  "excludePatterns": ["/tag/", "*.pdf"],
  "lastmodSince": "30 days",
  "includeUrlsWithoutLastmod": true
}

📤 Output

Example rows (shape from a real run, values shortened):

[
  {
    "url": "https://crawlee.dev/js/docs/quick-start",
    "lastmod": null,
    "changefreq": "weekly",
    "priority": 0.5,
    "type": "page",
    "sitemapUrl": "https://crawlee.dev/sitemap.xml",
    "sitemapType": "sitemap",
    "discoveredVia": "robots.txt",
    "site": "crawlee.dev",
    "inputUrl": "https://crawlee.dev",
    "imageCount": 0,
    "videoCount": 0,
    "images": [],
    "videos": [],
    "news": null,
    "alternates": [],
    "title": null
  },
  {
    "url": "https://news.example.com/2026/oct/14/city-council-approves-new-park",
    "lastmod": "2026-10-15T01:46:00Z",
    "changefreq": "hourly",
    "priority": null,
    "type": "news",
    "sitemapUrl": "https://news.example.com/sitemaps/news.xml",
    "sitemapType": "sitemap",
    "discoveredVia": "input",
    "site": "news.example.com",
    "inputUrl": "https://news.example.com/sitemaps/news.xml",
    "imageCount": 1,
    "videoCount": 0,
    "images": [{ "url": "https://img.example.com/park.jpg" }],
    "videos": [],
    "news": {
      "publicationName": "Example News",
      "language": "en",
      "publicationDate": "2026-10-15T01:45:54Z",
      "title": "City council approves new park",
      "keywords": ["Local news", "Parks"]
    },
    "alternates": [],
    "title": "City council approves new park"
  }
]

💰 Pricing

You pay $0.30 per 1,000 unique URLs saved ($0.0003 per URL), plus the small platform usage of the run (compute and datacenter proxy traffic, usually well under a cent for a few thousand URLs). Duplicates, filtered URLs and failed sitemaps are never charged.

Example: extracting 1,000 URLs costs about $0.30 in URL fees plus roughly $0.001 of platform usage. Set a maximum total charge on the run and the actor stops cleanly when it is reached.

❓ FAQ

Is it legal to extract sitemaps? Sitemaps and robots.txt are public files that websites publish so that crawlers can find their pages. Respect each site's terms and use the URLs responsibly.

Do I need a login or my own proxy? No. The default datacenter proxy works for almost every site. Switch to residential only if a site blocks datacenter IPs.

How many URLs can I get? As many as the sitemaps list. Large sitemaps are streamed, so memory stays flat even for hundreds of thousands of URLs. Use Max URLs or Max URLs per site to cap a run.

Why is lastmod empty? Many sitemaps don't include it. Turn on Check page status to fill it from the page's Last-Modified header where the server sends one.

What if a site has no sitemap? The actor tries all common locations, then the site's RSS/Atom feeds. The status message lists sites where nothing was found.

Can I schedule it or use the API? Yes. Schedule a daily run with "Modified since: 1 day" to collect new pages, and read the results through the API or integrations.

⚠️ Limits

• Only URLs listed in sitemaps or feeds are returned. Pages that are not in any sitemap are not discovered (this is not a link crawler).
• RSS/Atom feeds usually list only the latest items, so feed-only sites return fewer URLs.
• Sites that block all automated access to their sitemaps (for example with a JavaScript challenge) cannot be read.
• lastmod, changefreq and priority are self-reported by the site and may be missing or inaccurate.
• Sitemap indexes are followed up to 8 levels deep and 5,000 sitemap files per site.