Docs pages changed since 2024, 10 URLs

Input

The input this example runs Sitemap Extractor with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Websites or sitemap URLsstartUrls

https://crawlee.dev

Max URLsmaxUrls

10

Include URLs matchingincludePatterns

/docs/

Modified sincelastmodSince

2024-01-01

Keep URLs without lastmod when a date filter is setincludeUrlsWithoutLastmod

Yes

Sitemap discoverydiscoveryMode

Thorough (robots.txt + all common paths)

{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 10,
  "discoveryMode": "thorough",
  "includeUrlsWithoutLastmod": true,
  "includePatterns": [
    "/docs/"
  ],
  "lastmodSince": "2024-01-01"
}