Web Scraper

🕸️ Web Scraper

Turn public pages into JSON with the page function you already know how to write. The default fetch is HTTP. Turn on useChrome when the text only appears after scripts run.

Built for teams who collect public records from sites that put the data in the response, and for the pages that need a real document.

✨ Features

• 🌐 HTTP first. With useChrome off, each page is downloaded with curl_cffi impersonating Chrome 131. No extra User-Agent is added on top of that impersonation.
• 🧭 useChrome runs the document. That flag launches headless Chromium, waits for the navigation, and gives the page function the DOM after scripts. Other runs stay on HTTP.
• 🧩 Your page function. An async JavaScript function runs on that document. context.jQuery is real jQuery when Inject jQuery is on.
• 🖱️ Scroll and cookie dialogs, in Chromium. With useChrome on, maxScrollHeightPixels scrolls until new content stops or that distance is reached (0 skips scrolling). closeCookieModals clicks common consent controls so the dialog is dismissed before the page function runs.
• 🚫 Choose the subresources. With useChrome on, downloadMedia and downloadCss allow images, fonts, video, audio, and stylesheets, or block them. ignoreCorsAndCsp lets a navigation hook fetch another origin from the page.
• 🪝 Hooks can drive the page. With useChrome on, pre-navigation and post-navigation hooks receive the Chromium page. Without it, page is null.
• 🔗 Lists and link following. startUrls accepts page URLs and requestsFromUrl lists. linkSelector, globs, pseudoUrls, and excludes decide what gets enqueued. enqueueRequest adds a URL directly (globs, pseudo-URLs and excludes do not filter it, as in the original Web Scraper), and skipLinks stops selector links for that page.
• 📏 Limits that hold. maxPagesPerCrawl, maxResultsPerCrawl, and maxCrawlingDepth stop the crawl. maxRequestRetries retries transport failures, HTTP 401/403/408/429/5xx, and a page function that throws. The last HTTP response is what the page function sees, including a non-200 status.
• 🏠 Production proxy. proxyConfiguration uses Scrapely Proxy. If Scrapely Proxy is on and no group is set, the run uses the residential group. proxyRotation supports RECOMMENDED, PER_REQUEST, and UNTIL_FAILURE.
• 🤖 Optional robots.txt. respectRobotsTxtFile skips a disallowed link found by linkSelector. A URL you put in startUrls, requestsFromUrl, or enqueueRequest is still fetched.
• 📦 Dataset metadata on every record. Each object your function returns is stored with #error and #debug.

🚀 Why teams use this

• The document you asked for. HTTP when the server already sent the text. Chromium when scripts have to run first.
• Production runs. Residential proxy, page and function timeouts, retries, and concurrency up to 8 (one at a time in DEVELOPMENT) keep a crawl inside a normal 1024 MB run.
• The response that came back is the record. A non-200 HTTP response is still handed to the page function. A URL that never returns a response is left out of the dataset, and that run fails.

💡 Use cases

• Crawl a site and extract custom fields with your own page function.
• Prototype a scraper quickly with CSS selectors and link globs.
• Collect product, article, or listing data from sites without an API.

🧪 Example input

{
  "startUrls": [
    {
      "url": "https://docs.python.org/3/tutorial/"
    }
  ],
  "globs": [
    {
      "glob": "https://docs.python.org/3/tutorial/*"
    }
  ],
  "linkSelector": "a[href]",
  "pageFunction": "async function pageFunction(context) {\n    const $ = context.jQuery;\n    return { url: context.request.url, title: $('title').first().text() };\n}"
}

🧾 Sample output

One item from a real run of the example input, shortened. Long text is cut and lists show their first entries.

{
  "url": "https://docs.python.org/3/tutorial/introduction.html",
  "pageTitle": "3. An Informal Introduction to Python — Python 3.15.0 documentation",
  "h1": "3. An Informal Introduction to Python¶",
  "first_h2": "3.1. Using Python as a Calculator¶",
  "random_text_from_the_page": "2. Using the Python Interpreter"
}

💰 Pricing

You pay $0.0005 per result ($0.50 per 1,000 results) plus platform usage. A result is one record your page function returns with at least one field. Pages that return nothing, metadata-only records (null) and URLs that fail are not charged.

Worked example: 1,000 results ≈ $0.50 in event fees plus platform usage. On our test runs at the default 1024 MB, usage was about $0.0015 for the 19-page example input and about $0.010 for a 100-page crawl over HTTP (roughly $0.10 per 1,000 pages). With useChrome on, the remote browser adds browser credits: the same 19 pages cost about $0.045 in usage.

Set a maximum charge for the run if you want a hard cap. The run then stops cleanly once it is reached and keeps everything saved so far.

📥 Input

Required: startUrls, pageFunction, proxyConfiguration.

context inside the page function: jQuery (when injected), request, response (status, headers), customData, input, env, log, enqueueRequest, skipLinks, getValue, setValue, globalStore, saveSnapshot, waitFor.

📤 Output

Each object returned by pageFunction becomes one dataset record. An array becomes one record per element. null stores metadata only.

Always present:

• #error — false on a stored page-function record. A URL that never returns a response is not written as an error row; the run fails instead.
• #debug.requestId
• #debug.url
• #debug.method
• #debug.retryCount — 0 on the first attempt.
• #debug.errorMessages — null when there were none, otherwise short status or exception names such as HTTP 429 or TimeoutError.
• #debug.loadedUrl — the final URL after redirects, when a response arrived.
• #debug.statusCode — the HTTP status, when a response arrived.

Your own fields are whatever the page function read from the response. Empty strings, empty arrays, and empty objects are left off the record. A real false or 0 stays. A field named error is not copied. A plain-text body is exposed as text in the document; the words are the response body.

saveSnapshot() writes the HTML to the key-value store as SNAPSHOT-BODY (text/html), at most once every 2 seconds. It does not write a screenshot.