//Web Scraper example\\

Python tutorial without media, 5 pages

Same crawl with images and CSS downloads turned off.

Updated October 21, 2026

Input

The input this example runs Web Scraper with. “Try this input” opens it in the console, where you can run it as-is or change any field first.

Start URLsstartUrls: https://docs.python.org/3/tutorial/
Max pages per runmaxPagesPerCrawl: 5
Page functionpageFunction: // The function accepts a single argument: the "context" object. // For a complete list of its properties and functions, // see the Input section of this actor's README. async function pageFunction(context) { // jQuery is handy for finding DOM elements and extracting data from them. // To u…
Link selectorlinkSelector: a[href]
Glob Patternsglobs: {"glob":"https://docs.python.org/3/tutorial/*"}
Exclude Glob Patternsexcludes: {"glob":"/**/*.{png,jpg,jpeg,pdf}"}
Respect the robots.txt filerespectRobotsTxtFile: Yes
Navigation waits untilwaitUntil: networkidle2
Download media filesdownloadMedia: No
Download CSS filesdownloadCss: No
Pre-navigation hookspreNavigationHooks: // We need to return array of (possibly async) functions here. // The functions accept two arguments: the "crawlingContext" object // and "gotoOptions". [ async (crawlingContext, gotoOptions) => { // ... }, ]
Post-navigation hookspostNavigationHooks: // We need to return array of (possibly async) functions here. // The functions accept a single argument: the "crawlingContext" object. [ async (crawlingContext) => { // ... }, ]
Run moderunMode: PRODUCTION
Custom datacustomData: {}

.json

{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "pageFunction": "// The function accepts a single argument: the \"context\" object.\n// For a complete list of its properties and functions,\n// see the Input section of this actor's README.\nasync function pageFunction(context) {\n    // jQuery is handy for finding DOM elements and extracting data from them.\n    // To use it, make sure to enable the \"Inject jQuery\" option.\n    const $ = context.jQuery;\n    const pageTitle = $('title').first().text();\n    const h1 = $('h1').first().text();\n    const first_h2 = $('h2').first().text();\n    const random_text_from_the_page = $('p').first().text();\n\n\n    // Print some information to Actor log\n    context.log.info(`URL: ${context.request.url}, TITLE: ${pageTitle}`);\n\n    // Manually add a new page to the queue for scraping.\n   await context.enqueueRequest({ url: 'http://www.example.com' });\n\n    // Return an object with the data extracted from the page.\n    // It will be stored to the resulting dataset.\n    return {\n        url: context.request.url,\n        pageTitle,\n        h1,\n        first_h2,\n        random_text_from_the_page\n    };\n}",
  "linkSelector": "a[href]",
  "globs": [
    {
      "glob": "https://docs.python.org/3/tutorial/*"
    }
  ],
  "excludes": [
    {
      "glob": "/**/*.{png,jpg,jpeg,pdf}"
    }
  ],
  "pseudoUrls": [],
  "respectRobotsTxtFile": true,
  "waitUntil": [
    "networkidle2"
  ],
  "preNavigationHooks": "// We need to return array of (possibly async) functions here.\n// The functions accept two arguments: the \"crawlingContext\" object\n// and \"gotoOptions\".\n[\n    async (crawlingContext, gotoOptions) => {\n        // ...\n    },\n]\n",
  "postNavigationHooks": "// We need to return array of (possibly async) functions here.\n// The functions accept a single argument: the \"crawlingContext\" object.\n[\n    async (crawlingContext) => {\n        // ...\n    },\n]",
  "runMode": "PRODUCTION",
  "customData": {},
  "maxPagesPerCrawl": 5,
  "downloadMedia": false,
  "downloadCss": false
}

The internet is full of data.
Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever.

Get Started Free Book a Demo

No credit card. No setup calls.