Python tutorial pages as text, 10 pages

What this example returns

This example returns up to 10 crawled pages.

What each setting does

The input this example runs Website Content Crawler with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Start URLs startUrls

https://docs.python.org/3/tutorial/

Pages to start crawling from. Put your own URL here. Sub-pages are crawled too, e.g. https://example.com/blog also covers https://example.com/blog/post. Include URL patterns can widen this.

Max pages maxCrawlPages

10

The maximum number pages to crawl. It includes the start URLs, pagination pages, pages with no content, etc. The crawler will automatically finish after reaching this number. This setting is useful to prevent accidental crawler runaway.

Default: 9999999 → this example uses 10

Max results maxResults

10

The maximum number of web pages and files to store. This setting helps prevent an accidental crawler runaway by automatically stopping the crawl once this limit is reached. Note that the crawler skips pages whose canonical URL matches a page that has already been crawled, so it may crawl more pages than the number of…

Default: 9999999 → this example uses 10

Respect the robots.txt file respectRobotsTxtFile

Yes

If enabled, the crawler will consult the robots.txt file for the target website before crawling each page. At the moment, the crawler does not use any specific user agent identifier. The crawl-delay directive is also not supported yet.

Default: No → this example uses Yes

{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "maxCrawlPages": 10,
  "respectRobotsTxtFile": true,
  "maxResults": 10
}

What it costs

About $0.005 for 10 crawled pages

An estimate: 10 × $0.0005 per crawled page (One crawled page saved to the dataset with its content (text, Markdown or HTML)). A run is charged for the events it reports, so one that finds fewer crawled pages costs less. Platform usage is billed on top, typically about $0.102 per 1,000 results.

Full pricing

Other ways to run Website Content Crawler

• Python tutorial as markdown, 5 pagesClean readable text from a documentation crawl.
• Python library docs with HTML, 3 pagesOnly pages under one URL path, with saved HTML.
• Crawlee site via sitemap, 5 pagesPages found from the sitemap as clean text.

All examples of Website Content Crawler

About Website Content Crawler

Crawl whole websites and extract clean text and Markdown for AI models, vector databases, and RAG pipelines. Removes navigation and boilerplate and keeps titles, metadata, and links.

Price:

from $0.40 / 1,000 results

Developer:

Querystone

Categories:

AI, Developer tools, SEO tools

Website Content Crawler pageOutput fieldsPricing