Docusaurus docs via sitemap, 5 pages

What this example returns

This example returns up to 5 crawled pages.

What each setting does

The input this example runs Website Content Crawler with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Start URLs startUrls

https://docusaurus.io/

Pages to start crawling from. Put your own URL here. Sub-pages are crawled too, e.g. https://example.com/blog also covers https://example.com/blog/post. Include URL patterns can widen this.

Max pages maxCrawlPages

5

The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.

Default: 9999999 → this example uses 5

Max results maxResults

5

The maximum number of web pages and files to store. This setting helps prevent an accidental crawler runaway by automatically stopping the crawl once this limit is reached. Note that the crawler skips pages whose canonical URL matches a page that has already been crawled, so it may crawl more pages than the number of…

Default: 9999999 → this example uses 5

Load URLs from Sitemaps useSitemaps

Yes

Also queue matching URLs from the sitemaps of the start URL domains. You can add a sitemap.xml directly as a start URL. Finds pages that aren't linked, but large sitemaps can take a long time. Sitemap pages get depth 1.

Default: No → this example uses Yes

HTML transformer htmlTransformer

Main element, boilerplate removed

How to extract the main content after elements are removed or clicked. Readable text (default) keeps the article text and drops headers, footers and navigation. Readable text with fallback keeps the original HTML when the page isn't clearly an article. Main element keeps the page's article or main element (or its…

Default: Readable text → this example uses Main element, boilerplate removed

{
  "startUrls": [
    {
      "url": "https://docusaurus.io/"
    }
  ],
  "useSitemaps": true,
  "htmlTransformer": "mainElementClean",
  "maxCrawlPages": 5,
  "maxResults": 5
}

What it costs

From about $0.00044 for 5 crawled pages

An estimate at the lowest plan price: 5 × $0.0000884 per crawled page (One crawled page saved to the dataset with its content (text, Markdown or HTML)). A run is charged for the events it reports, so one that finds fewer crawled pages costs less. Platform usage is included, except when a run renders pages in a browser: such runs also pay their platform usage (compute, browser, proxy) at your plan's rates.

Full pricing

Other ways to run Website Content Crawler

• Python tutorial pages as text, 10 pages
• Python tutorial as markdown, 5 pagesClean readable text from a documentation crawl.
• Python library docs with HTML, 3 pagesOnly pages under one URL path, with saved HTML.

All examples of Website Content Crawler

About Website Content Crawler

Crawl whole websites and extract clean text and Markdown for AI models, vector databases, and RAG pipelines. Removes navigation and boilerplate and keeps titles, metadata, and links.

Price:

from $0.088 / 1,000 results

Developer:

Querystone

Categories:

AI, Developer tools, SEO tools

Website Content Crawler pageOutput fieldsPricing