Python library docs with HTML, 3 pages

What this example returns

This example returns up to 3 crawled pages.

What each setting does

The input this example runs Website Content Crawler with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Start URLs startUrls

https://docs.python.org/3/library/json.html

Pages to start crawling from. Put your own URL here. Sub-pages are crawled too, e.g. https://example.com/blog also covers https://example.com/blog/post. Include URL patterns can widen this.

Max pages maxCrawlPages

3

The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.

Default: 9999999 → this example uses 3

Include URL patterns (globs) includeUrlGlobs

{"glob":"https://docs.python.org/3/library/*"}

Glob patterns for links to crawl beyond the start URLs and their sub-pages, e.g. https://www.example.com/blog/** matches every blog page. Only applies to links found on pages; start URLs are always crawled.

Max crawling depth maxCrawlDepth

20

How many links away from the start URLs to follow. Start URLs are depth 0; 0 crawls only the start URLs.

Max results maxResults

3

The maximum number of web pages and files to store. This setting helps prevent an accidental crawler runaway by automatically stopping the crawl once this limit is reached. Note that the crawler skips pages whose canonical URL matches a page that has already been crawled, so it may crawl more pages than the number of…

Default: 9999999 → this example uses 3

URL #fragments identify unique pages keepUrlFragments

No

Indicates that URL fragments (e.g. http://example.com#fragment) should be included when checking whether a URL has already been visited or not. Typically, URL fragments are used for page navigation only and therefore they should be ignored, as they don't identify separate pages. However, some single-page websites use…

Ignore canonical URLs ignoreCanonicalUrl

No

If enabled, the Actor will ignore the canonical URL or the `ETag` header reported by the page, and use the actual URL instead. You can use this feature for websites that report invalid canonical URLs, which causes the Actor to skip those pages in results.

Respect the robots.txt file respectRobotsTxtFile

No

If enabled, the crawler will consult the robots.txt file for the target website before crawling each page. At the moment, the crawler does not use any specific user agent identifier. The crawl-delay directive is also not supported yet.

Load URLs from Sitemaps useSitemaps

No

Also queue matching URLs from the sitemaps of the start URL domains. You can add a sitemap.xml directly as a start URL. Finds pages that aren't linked, but large sitemaps can take a long time. Sitemap pages get depth 1.

Crawl /llms.txt and Markdown files useLlmsTxt

No

If enabled, the crawler will look for `/llms.txt` files at the root of the domains of the provided Start URLs (e.g., `https://example.com/llms.txt`) and enqueue them for crawling. Note that this also enables crawling other Markdown files and enqueueing links from them.

HTML transformer htmlTransformer

Main element

How to extract the main content after elements are removed or clicked. Readable text (default) keeps the article text and drops headers, footers and navigation. Readable text with fallback keeps the original HTML when the page isn't clearly an article. Main element keeps the page's article or main element (or its…

Default: Readable text → this example uses Main element

Remove HTML elements (CSS selector) removeElementsCssSelector

nav, footer, script, style, noscript, svg, img[src^='data:'], [role="alert"], [role="banner"], [role="dialog"], [role="alertdialog"], [role="region"][aria-label*="skip" i], [aria-modal="true"]

CSS selector for elements removed before text extraction. The default removes navigation, headers, footers, modals, scripts and inline images. Set it to a selector that matches nothing (e.g. dummy_keep_everything) to keep everything.

Remove duplicate text lines aggressivePrune

No

This is an experimental feature. If enabled, the crawler will prune content lines that are very similar to the ones already crawled on other pages, using the Count-Min Sketch algorithm. This is useful to strip repeating content in the scraped data like menus, headers, footers, etc. In some (not very likely) cases, it…

Remove cookie warnings removeCookieWarnings

No

When enabled, the page is opened in Chrome and short cookie, consent, or GDPR banners are removed before the text is saved. A banner is matched by its id, class, or aria-label. When disabled, this does not open Chrome.

Expand iframe elements expandIframes

Yes

By default, the Actor will extract content from `iframe` elements. If you want to specifically skip `iframe` processing, disable this option. Up to five same-host iframe documents are fetched and inlined.

Wait for dynamic content dynamicContentWaitSecs

0

How long, in seconds, Chrome waits before the text is saved. When Wait for selector is set, Chrome waits for that selector and stops early if it appears. `0` does not add this wait and does not open Chrome by itself. A value above `0` opens the page in Chrome, including a server-rendered page, so content that appears…

Maximum scroll height maxScrollHeightPixels

0

How far, in pixels, Chrome scrolls down the page before the text is saved. Scrolling stops at this distance or when the page no longer grows. `0` does not scroll and does not open Chrome by itself. A value above `0` opens the page in Chrome.

Block loading of images and videos blockMedia

No

When enabled, the page is opened in Chrome and stylesheets are not loaded either. Chrome never loads images, fonts, video or audio, because only text, Markdown and HTML are saved (image links stay in the page). Scripts still load. When disabled, this does not open Chrome.

Save Markdown saveMarkdown

Yes

If enabled, the crawler converts the transformed HTML of all pages found to Markdown, and stores it under the `markdown` field in the output dataset.

Save HTML to dataset (deprecated) saveHtml

Yes

If enabled, the crawler stores full transformed HTML of all pages found to the output dataset under the `html` field. This option has been deprecated in favor of the `saveHtmlAsFile` option, because the dataset records have a size of approximately 10MB and it's harder to review the HTML for debugging.

Default: No → this example uses Yes

Save HTML to key-value store saveHtmlAsFile

No

If enabled, the crawler stores full transformed HTML of all pages found to the default key-value store and saves links to the files as `htmlUrl` field in the output dataset. Storing HTML in key-value store is preferred to storing it into the dataset with the `saveHtml` option, because there's no size limit and it's…

Save files saveFiles

No

Deprecated in favor of the `saveContentTypes` option. Will be removed soon. If enabled, the crawler downloads files linked from the web pages, as long as their URL has one of the following file extensions: PDF, DOC, DOCX, XLS, XLSX, and CSV. Note that unlike web pages, the files are downloaded regardless if they are…

Max concurrency maxConcurrency

200

The maximum number of web browsers or HTTP clients running in parallel. This setting is useful to avoid overloading the target websites and to avoid getting blocked.

Initial concurrency initialConcurrency

0

The initial number of web browsers or HTTP clients running in parallel. The system scales the concurrency up and down based on the current CPU and memory load. If the value is set to 0 (default), the Actor uses the default setting for the specific crawler type. Note that if you set this value too high, the Actor will…

Page request timeout requestTimeoutSecs

60

Timeout in seconds for making the request and processing its response. Defaults to 60s.

Minimum file download speed minFileDownloadSpeedKBps

128

The minimum viable file download speed in kilobytes per seconds. If the file download speed is lower than this value for a prolonged duration, the crawler will consider the file download as failing, abort it, and retry it again (up to "Maximum number of retries" times). This is useful to avoid your crawls being stuck…

Maximum number of retries on network / server errors maxRequestRetries

3

The maximum number of times the crawler will retry the request on network, proxy or server errors. If the (n+1)-th request still fails, the crawler will mark this request as failed.

Maximum number of session rotations maxSessionRotations

5

The maximum number of times the crawler will rotate the session (IP address + browser configuration) on anti-scraping measures like CAPTCHAs. If the crawler rotates the session more than this number and the page is still blocked, it will finish with an error.

Default: 10 → this example uses 5

Ignore HTTPS errors ignoreHttpsErrors

No

If enabled, the scraper will ignore HTTPS certificate errors. Use at your own risk.

Custom HTTP headers customHttpHeaders

{}

HTTP headers that will be added to all requests made by the crawler. This is useful for setting custom authentication headers or other headers required by the target website. The value is expected to be a JSON object with `name` and `value` properties pairs. For example: `{ "name1": "value1", "Authorization": "Basic…

Readable text extractor character threshold readableTextCharThreshold

100

A configuration options for the "Readable text" HTML transformer. It contains the minimum number of characters an article must have in order to be considered relevant.

Store skipped URLs storeSkippedUrls

No

If enabled, the crawler will store all URLs that were skipped during the crawl in a Key-Value Store record named `SKIPPED_URLS`. The record will contain a JSON object with reasons for skipping and the URLs that were skipped for each reason. This is useful for debugging and understanding why certain pages were not…

{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/library/json.html"
    }
  ],
  "maxCrawlPages": 3,
  "maxResults": 3,
  "includeUrlGlobs": [
    {
      "glob": "https://docs.python.org/3/library/*"
    }
  ],
  "excludeUrlGlobs": [],
  "maxCrawlDepth": 20,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "respectRobotsTxtFile": false,
  "useSitemaps": false,
  "useLlmsTxt": false,
  "htmlTransformer": "mainElement",
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'],\n[role=\"alert\"],\n[role=\"banner\"],\n[role=\"dialog\"],\n[role=\"alertdialog\"],\n[role=\"region\"][aria-label*=\"skip\" i],\n[aria-modal=\"true\"]",
  "keepElementsCssSelector": "",
  "clickElementsCssSelector": "",
  "aggressivePrune": false,
  "removeCookieWarnings": false,
  "expandIframes": true,
  "dynamicContentWaitSecs": 0,
  "waitForSelector": "",
  "softWaitForSelector": "",
  "maxScrollHeightPixels": 0,
  "blockMedia": false,
  "saveMarkdown": true,
  "saveHtml": true,
  "saveHtmlAsFile": false,
  "saveFiles": false,
  "maxConcurrency": 200,
  "initialConcurrency": 0,
  "requestTimeoutSecs": 60,
  "minFileDownloadSpeedKBps": 128,
  "maxRequestRetries": 3,
  "maxSessionRotations": 5,
  "ignoreHttpsErrors": false,
  "customHttpHeaders": {},
  "readableTextCharThreshold": 100,
  "storeSkippedUrls": false
}

What it costs

From about $0.00027 for 3 crawled pages

An estimate at the lowest plan price: 3 × $0.0000884 per crawled page (One crawled page saved to the dataset with its content (text, Markdown or HTML)). A run is charged for the events it reports, so one that finds fewer crawled pages costs less. Platform usage is included, except when a run renders pages in a browser: such runs also pay their platform usage (compute, browser, proxy) at your plan's rates.

Full pricing

Other ways to run Website Content Crawler

• Python tutorial pages as text, 10 pages
• Python tutorial as markdown, 5 pagesClean readable text from a documentation crawl.
• Docusaurus docs via sitemap, 5 pagesPages found from the sitemap as clean text.

All examples of Website Content Crawler

About Website Content Crawler

Crawl whole websites and extract clean text and Markdown for AI models, vector databases, and RAG pipelines. Removes navigation and boilerplate and keeps titles, metadata, and links.

Price:

from $0.088 / 1,000 results

Developer:

Querystone

Categories:

AI, Developer tools, SEO tools

Website Content Crawler pageOutput fieldsPricing