Python library docs with HTML, 3 pages

Input

The input this example runs Website Content Crawler with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Start URLsstartUrls

https://docs.python.org/3/library/json.html

Max pagesmaxCrawlPages

3

Include URL patterns (globs)includeUrlGlobs

{"glob":"https://docs.python.org/3/library/*"}

Max crawling depthmaxCrawlDepth

20

Max resultsmaxResults

3

URL #fragments identify unique pageskeepUrlFragments

No

Ignore canonical URLsignoreCanonicalUrl

No

Respect the robots.txt filerespectRobotsTxtFile

No

Load URLs from SitemapsuseSitemaps

No

Crawl /llms.txt and Markdown filesuseLlmsTxt

No

HTML transformerhtmlTransformer

Extractus

Remove HTML elements (CSS selector)removeElementsCssSelector

nav, footer, script, style, noscript, svg, img[src^='data:'], [role="alert"], [role="banner"], [role="dialog"], [role="alertdialog"], [role="region"][aria-label*="skip" i], [aria-modal="true"]

Remove duplicate text linesaggressivePrune

No

Remove cookie warningsremoveCookieWarnings

No

Expand iframe elementsexpandIframes

Yes

Wait for dynamic contentdynamicContentWaitSecs

0

Maximum scroll heightmaxScrollHeightPixels

0

Block loading of images and videosblockMedia

No

Save MarkdownsaveMarkdown

Yes

Save HTML to dataset (deprecated)saveHtml

Yes

Save HTML to key-value storesaveHtmlAsFile

No

Save filessaveFiles

No

Max concurrencymaxConcurrency

200

Initial concurrencyinitialConcurrency

0

Page request timeoutrequestTimeoutSecs

60

Minimum file download speedminFileDownloadSpeedKBps

128

Maximum number of retries on network / server errorsmaxRequestRetries

3

Maximum number of session rotationsmaxSessionRotations

5

Ignore HTTPS errorsignoreHttpsErrors

No

Custom HTTP headerscustomHttpHeaders

{}

Readable text extractor character thresholdreadableTextCharThreshold

100

Store skipped URLsstoreSkippedUrls

No

{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/library/json.html"
    }
  ],
  "maxCrawlPages": 3,
  "maxResults": 3,
  "includeUrlGlobs": [
    {
      "glob": "https://docs.python.org/3/library/*"
    }
  ],
  "excludeUrlGlobs": [],
  "maxCrawlDepth": 20,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "respectRobotsTxtFile": false,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "htmlTransformer": "extractus",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]",
  "keepElementsCssSelector": "",
  "clickElementsCssSelector": "",
  "aggressivePrune": false,
  "removeCookieWarnings": false,
  "expandIframes": true,
  "dynamicContentWaitSecs": 0,
  "waitForSelector": "",
  "softWaitForSelector": "",
  "maxScrollHeightPixels": 0,
  "blockMedia": false,
  "saveMarkdown": true,
  "saveHtml": true,
  "saveHtmlAsFile": false,
  "saveFiles": false,
  "maxConcurrency": 200,
  "initialConcurrency": 0,
  "requestTimeoutSecs": 60,
  "minFileDownloadSpeedKBps": 128,
  "maxRequestRetries": 3,
  "maxSessionRotations": 5,
  "ignoreHttpsErrors": false,
  "customHttpHeaders": {},
  "readableTextCharThreshold": 100,
  "storeSkippedUrls": false
}