Sitemap URLs of crawlee.dev, 10 URLs

What this example returns

This example returns up to 10 URLs.

The dataset’s “URLs” view shows 11 columns: URL, Last modified, Type, Change freq, Priority, Title, Images, Videos, Status, Sitemap, and Site. See every output field.

What each setting does

The input this example runs Sitemap Extractor with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Websites or sitemap URLs startUrls

https://crawlee.dev

Domains, home pages, sitemap files (.xml, .xml.gz, .txt) or RSS/Atom feeds. For a domain or page URL the actor reads robots.txt and the common sitemap locations and follows sitemap indexes to the end. A direct sitemap URL extracts just that sitemap and its children.

Max URLs per site maxUrlsPerSite

10

Limit for each input site when you extract many sites in one run. 0 = no per-site limit.

Default: 0 → this example uses 10

{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrlsPerSite": 10
}

What it costs

About $0.003 for 10 URLs

An estimate: 10 × $0.0003 per url (One unique URL saved to the dataset). A run is charged for the events it reports, so one that finds fewer URLs costs less. Platform usage is included in the event price.

Full pricing

When to use this example

From the Sitemap Extractor readme:

Other ways to run Sitemap Extractor

• All sitemap URLs of crawlee.dev, 15 URLsFinds every sitemap of a site and lists its URLs.
• Docs pages changed since 2024, 10 URLsSitemap URLs filtered by path and last-modified date.
• URLs from a sitemap.xml file, 15 URLsReads a sitemap file directly.

All examples of Sitemap Extractor

About Sitemap Extractor

Extract every URL from a website's XML sitemaps: finds sitemaps via robots.txt and common paths, follows sitemap indexes, reads .xml.gz, text sitemaps and RSS/Atom feeds. Get lastmod, changefreq, priority, image, video and news data, with URL pattern and date filters. No login needed.

Price:

from $0.242 / 1,000 results

Developer:

Querystone

Categories:

SEO tools, Developer tools, AI

Sitemap Extractor pageOutput fieldsPricing