The Python tutorial's start page as text

What each setting does

The input this example runs Web Scraper with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Start URLs startUrls

https://docs.python.org/3/tutorial/

Pages to scrape. Put your own URL here: the default page function returns its title and main text.

Max pages per run maxPagesPerCrawl

10

Stop after loading this many pages. The real count can be slightly higher. 0 means no limit.

Default: 0 → this example uses 10

Page function pageFunction

// Runs on every page and returns its title and main text. // To scrape another site, change the Start URL. To extract other data, edit this // function; the README lists everything on the "context" object. async function pageFunction(context) { // jQuery reads the page (needs "Inject jQuery", …

JavaScript (ES6) function run in the browser on every page. Use it to extract data, interact with the page or queue more URLs. See the README for the context object.

Exclude Glob Patterns excludes

{"glob":"/**/*.{png,jpg,jpeg,pdf}"}

Glob patterns to match links in the page that you want to exclude from being enqueued.

Respect the robots.txt file respectRobotsTxtFile

Yes

If enabled, the crawler will consult the robots.txt file for the target website before crawling each page. At the moment, the crawler does not use any specific user agent identifier. The crawl-delay directive is also not supported yet.

Default: No → this example uses Yes

Pre-navigation hooks preNavigationHooks

// We need to return array of (possibly async) functions here. // The functions accept two arguments: the "crawlingContext" object // and "gotoOptions". [ async (crawlingContext, gotoOptions) => { // ... }, ]

Async functions evaluated before navigation. Each function receives crawlingContext and gotoOptions. When Use Chrome is on, crawlingContext.page is the Chromium page before navigation. Otherwise page is null.

Post-navigation hooks postNavigationHooks

// We need to return array of (possibly async) functions here. // The functions accept a single argument: the "crawlingContext" object. [ async (crawlingContext) => { // ... }, ]

Async functions evaluated after navigation and before the page function. When Use Chrome is on, crawlingContext.page is the Chromium page. Otherwise page is null.

{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "pageFunction": "// Runs on every page and returns its title and main text.\n// To scrape another site, change the Start URL. To extract other data, edit this\n// function; the README lists everything on the \"context\" object.\nasync function pageFunction(context) {\n    // jQuery reads the page (needs \"Inject jQuery\", which is on by default).\n    const $ = context.jQuery;\n    const pageTitle = $('title').first().text().replace(/\\s+/g, ' ').trim();\n\n    // The main content: the page's <main> or role=\"main\" area, a single <article>,\n    // a common content container, or else the whole body.\n    let main = $('main, [role=\"main\"]').first();\n    if (!main.length && $('article').length === 1) main = $('article');\n    if (!main.length) main = $('#content, #main, #main-content, .main-content, .post-content, .entry-content').first();\n    if (!main.length) main = $('body');\n\n    // Work on a copy without menus, headers, footers, sidebars, forms and scripts.\n    main = main.clone();\n    main.find('nav, header, footer, aside, form, script, style, noscript, template, svg, iframe, button, '\n        + '[role=\"navigation\"], [role=\"banner\"], [role=\"contentinfo\"], [role=\"search\"], [aria-hidden=\"true\"], .headerlink').remove();\n    // Mark the end of each block element, so paragraphs become separate lines.\n    const BREAK = '\\uE000';\n    main.find('p, div, section, li, h1, h2, h3, h4, h5, h6, pre, blockquote, tr, dt, dd, br').after(BREAK);\n    const text = main.text()\n        .split(BREAK)\n        .map((line) => line.replace(/\\s+/g, ' ').trim())\n        .filter(Boolean)\n        .join('\\n');\n\n    context.log.info(`URL: ${context.request.url}, TITLE: ${pageTitle}`);\n\n    // Each returned object becomes one row of the dataset.\n    return {\n        url: context.request.url,\n        pageTitle,\n        text,\n    };\n}",
  "excludes": [
    {
      "glob": "/**/*.{png,jpg,jpeg,pdf}"
    }
  ],
  "respectRobotsTxtFile": true,
  "preNavigationHooks": "// We need to return array of (possibly async) functions here.\n// The functions accept two arguments: the \"crawlingContext\" object\n// and \"gotoOptions\".\n[\n    async (crawlingContext, gotoOptions) => {\n        // ...\n    },\n]\n",
  "postNavigationHooks": "// We need to return array of (possibly async) functions here.\n// The functions accept a single argument: the \"crawlingContext\" object.\n[\n    async (crawlingContext) => {\n        // ...\n    },\n]",
  "maxPagesPerCrawl": 10
}

What it costs

You pay for the events the actor reports, such as each result or page it produces. Platform usage is included in the event prices: compute, proxies, browser and storage cost nothing extra.

• Result$0.00072One record with data returned by your page function and saved to the dataset.

Full pricing

Other ways to run Web Scraper

• Crawl the Python tutorial, 5 pagesPage titles and data extracted with a page function.
• Python tutorial without media, 5 pagesSame crawl with images and CSS downloads turned off.

All examples of Web Scraper

About Web Scraper

Crawl websites over HTTP and run your own page function on each page to extract exactly the data you need. Follow links with globs and pseudo-URLs, set limits, and export structured results.

Price:

from $0.572 / 1,000 results

Developer:

Querystone

Categories:

Developer tools, Automation

Web Scraper pageOutput fieldsPricing