Node.js Web Scraping: Fetch, Cheerio, Playwright, Puppeteer and Crawlee

Node.js web scraping uses the same language the pages themselves are written in: fetch or axios download the HTML, Cheerio queries it with jQuery-style selectors, and Playwright or Puppeteer run a real browser when the content only appears after JavaScript. For anything bigger than a script, Crawlee adds the queue, retries and storage.

This guide walks through all of them with code we ran on 26 October 2026 (Node.js 23.9) against the practice sites books.toscrape.com and quotes.toscrape.com. At the end, the same Crawlee scraper becomes a cloud Actor from a Scrape.ly template, and you'll see how to call ready-made scrapers from Node with one client call.

Node.js web scraping libraries compared

The rule of thumb: start with fetch and Cheerio, switch to Crawlee when you crawl, and use a browser only for pages that need one.

Set up the project

Any current Node.js release works (fetch is built in since Node 18). The examples use ES modules and top-level await:

mkdir scraper && cd scraper
npm init -y
npm pkg set type=module
npm install cheerio axios

Your first scraper: fetch and Cheerio

Inspect a book on books.toscrape.com and you'll find each one in an <article class="product_pod">, with the full title in the link's title attribute and the price in p.price_color:

import * as cheerio from 'cheerio';

const response = await fetch('https://books.toscrape.com/', {
  headers: { 'User-Agent': 'books-tutorial/1.0 (you@example.com)' },
  signal: AbortSignal.timeout(15_000),
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const $ = cheerio.load(await response.text());

$('article.product_pod').slice(0, 3).each((i, el) => {
  const title = $(el).find('h3 a').attr('title');
  const price = $(el).find('p.price_color').text();
  console.log(title, price);
});
console.log($('article.product_pod').length, 'books on this page');

Output:

A Light in the Attic £51.77
Tipping the Velvet £53.74
Soumission £50.10
20 books on this page

Two habits to keep from day one. fetch doesn't throw on a 404 or 500, so check response.ok yourself, or you'll parse an error page and wonder why your selectors return nothing. And AbortSignal.timeout() gives the request a deadline; without one, a hung server hangs your script.

Crawl every page: axios, retries and delays

The catalogue has 50 pages, each with a next link (li.next a). This crawler follows it, retries network errors, 429 and 5xx responses with exponential backoff, pauses one second between pages and saves JSON:

import { writeFile } from 'node:fs/promises';
import { setTimeout as sleep } from 'node:timers/promises';
import axios from 'axios';
import * as cheerio from 'cheerio';

const RATINGS = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
const http = axios.create({
  timeout: 15_000,
  headers: { 'User-Agent': 'books-tutorial/1.0 (you@example.com)' },
});

async function get(url, attempt = 1) {
  try {
    return (await http.get(url)).data;
  } catch (err) {
    const status = err.response?.status;
    const retryable = !status || status === 429 || status >= 500;
    if (!retryable || attempt === 3) throw err;
    await sleep(1000 * 2 ** attempt); // 2 s, then 4 s
    return get(url, attempt + 1);
  }
}

function parse(html, pageUrl) {
  const $ = cheerio.load(html);
  const books = $('article.product_pod').map((i, el) => {
    const card = $(el);
    const link = card.find('h3 a');
    return {
      title: link.attr('title'),
      price: Number(card.find('p.price_color').text().replace('£', '')),
      rating: RATINGS[card.find('p.star-rating').attr('class').split(' ')[1]],
      inStock: card.find('p.availability').text().includes('In stock'),
      url: new URL(link.attr('href'), pageUrl).href,
    };
  }).get();
  const next = $('li.next a').attr('href');
  return { books, next: next ? new URL(next, pageUrl).href : null };
}

const maxPages = Number(process.argv[2] ?? 50);
let url = 'https://books.toscrape.com/catalogue/page-1.html';
const all = [];
for (let page = 1; url && page <= maxPages; page++) {
  const { books, next } = parse(await get(url), url);
  all.push(...books);
  url = next;
  await sleep(1000); // one request per second
}

await writeFile('books.json', JSON.stringify(all, null, 2));
console.log(all.length, 'books');
console.log(all[0]);

With node crawl.js 5 it fetched five pages in under 9 seconds and printed:

100 books
{
  title: 'A Light in the Attic',
  price: 51.77,
  rating: 3,
  inStock: true,
  url: 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'
}

Notes on the parts that matter:

• new URL(href, pageUrl) resolves relative links like page-2.html correctly. Never concatenate URL strings.
• The retry rule repeats only what can succeed on a second try: no response at all, 429, or a 5xx. A 404 fails immediately.
• .map(...).get() is the Cheerio idiom for turning a selection into a plain array. Without .get() you keep a Cheerio object.
• The rating hides in a class name (star-rating Three). Real sites hide data in odd places all the time.

Check for a JSON API first

Many JavaScript-heavy pages load their data from an API you can call directly. Open DevTools, choose Network, filter by Fetch/XHR and reload. The infinite-scroll version of the quotes site uses one:

const quotes = [];
for (let page = 1; ; page++) {
  const res = await fetch(`https://quotes.toscrape.com/api/quotes?page=${page}`);
  const data = await res.json();
  quotes.push(...data.quotes.map((q) => ({ text: q.text, author: q.author.name, tags: q.tags })));
  if (!data.has_next) break;
}
console.log(quotes.length, 'quotes');

100 quotes in about a second, no HTML parsing, no browser. Structured JSON also survives redesigns that would break CSS selectors.

Scrape JavaScript pages with Playwright or Puppeteer

quotes.toscrape.com/js renders its quotes with a script, so fetch sees an empty page. A headless browser runs the script first. Install Playwright with npm install playwright and npx playwright install chromium:

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/');
await page.waitForSelector('div.quote'); // rendered by JavaScript

const quotes = await page.$$eval('div.quote', (cards) =>
  cards.map((card) => ({
    text: card.querySelector('span.text').textContent,
    author: card.querySelector('small.author').textContent,
  })),
);
console.log(quotes.length, quotes[0]);
await browser.close();

It printed 10 and the first quote, by Albert Einstein. The Puppeteer version (npm install puppeteer, which downloads its own Chrome) is nearly identical:

import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('div.quote');

const quotes = await page.$$eval('div.quote', (cards) =>
  cards.map((card) => ({
    text: card.querySelector('span.text').textContent,
    author: card.querySelector('small.author').textContent,
  })),
);
console.log(quotes.length, quotes[0]);
await browser.close();

Same result. The function you pass to $$eval runs inside the page, so it uses browser APIs (querySelector, textContent), not Cheerio. Which one to pick? Playwright supports three engines and auto-waits on every action; Puppeteer is Chrome-first and slightly lighter. Our Playwright vs Puppeteer comparison goes deeper.

Crawl at scale with Crawlee

Once you crawl many pages, you end up writing a queue, deduplication, retries, concurrency limits and somewhere to put the results. Crawlee ships all of that. CheerioCrawler fetches over HTTP and hands each page to your handler with Cheerio already loaded:

import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
  maxRequestsPerCrawl: 5, // stop early while testing
  maxConcurrency: 2,
  async requestHandler({ $, request, enqueueLinks, log }) {
    log.info(`Scraping ${request.url}`);
    const books = $('article.product_pod').map((i, el) => ({
      title: $(el).find('h3 a').attr('title'),
      price: $(el).find('p.price_color').text(),
      url: new URL($(el).find('h3 a').attr('href'), request.loadedUrl).href,
    })).get();
    await Dataset.pushData(books);
    await enqueueLinks({ selector: 'li.next a' });
  },
});

await crawler.run(['https://books.toscrape.com/']);
await Dataset.exportToCSV('books');
console.log((await Dataset.getData()).total, 'books');

Install it with npm install crawlee. Our run finished with Total 5 requests: 5 succeeded, 0 failed and 100 books, saved under ./storage and exported to storage/key_value_stores/default/books.csv. enqueueLinks skips URLs it has already seen, failed requests are retried automatically (three times by default), and swapping CheerioCrawler for PlaywrightCrawler or PuppeteerCrawler gives you a browser with the same handler shape. Crawlee is also the base of the Scrape.ly JavaScript SDK, which is why the next step is short.

Deploy your Node.js scraper as a Scrape.ly Actor

A script on your laptop stops when the laptop sleeps. As an Actor on Scrape.ly, the same code runs in the cloud on a schedule, writes to a hosted dataset you can download as CSV or JSON, and uses Scrape.ly's proxy. Start from a template; there are three for scraping in JavaScript:

Install the CLI, log in and create the project:

npm install -g scrapely-cli
export SCRAPELY_API_URL=https://api.scrape.ly
scrapely login
scrapely create books-scraper --template crawlee-cheerio-js && cd books-scraper

The template's src/main.js calls Actor.init(), reads startUrls and maxRequestsPerCrawl from the input, sets up the platform proxy and runs the router in src/routes.js. Replace the default handler there with the books logic:

import { createCheerioRouter } from '@crawlee/cheerio';

export const router = createCheerioRouter();

router.addDefaultHandler(async ({ enqueueLinks, request, $, log, pushData }) => {
    log.info(`Scraping ${request.loadedUrl}`);
    const books = $('article.product_pod')
        .map((i, el) => ({
            title: $(el).find('h3 a').attr('title'),
            price: $(el).find('p.price_color').text(),
            inStock: $(el).find('p.availability').text().includes('In stock'),
            url: new URL($(el).find('h3 a').attr('href'), request.loadedUrl).href,
        }))
        .get();
    await pushData(books);
    await enqueueLinks({ selector: 'li.next a' });
});

Test it on your machine, then push it:

npm install scrapely   # the SDK, for local runs only
scrapely run -i '{"startUrls":[{"url":"https://books.toscrape.com/"}],"maxRequestsPerCrawl":5}'
scrapely push
scrapely call -i '{"startUrls":[{"url":"https://books.toscrape.com/"}],"maxRequestsPerCrawl":5}' -o

We created this project with CLI 0.1.16 and ran it locally: 5 requests, 0 failed, 100 books in storage/datasets/default. Locally the SDK logs that no proxy is configured and fetches directly; on the platform, Scrape.ly injects the proxy credentials into the run. The npm install scrapely line is there because the template doesn't list the SDK in package.json yet; pushed builds don't need it, since the scrapely/actor-node base image includes it. After push, add a schedule or a delivery to Google Sheets or Slack without touching the code; see how to schedule and automate web scraping.

Sites with bot protection need more than a proxy. Turn on the Scrape.ly browser for your Actor and rebuild: Playwright and Puppeteer launches then connect to a hosted Chromium with anti-bot fingerprints, Cloudflare Turnstile solving and its own residential proxy, with no code changes.

Call ready-made scrapers from Node.js

Often you don't need to write a scraper at all. The Scrape.ly Store has 570+ Actors for sites like Amazon, Google Maps and LinkedIn, and the Web Scraper Actor runs your own page function on any public site. Call either from Node with the client (npm install scrapely-client):

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });

const pageFunction = `async function pageFunction(context) {
  const $ = context.jQuery;
  return $('article.product_pod').map((i, el) => ({
    title: $(el).find('h3 a').attr('title'),
    price: $(el).find('p.price_color').text(),
    rating: $(el).find('p.star-rating').attr('class').split(' ')[1],
  })).get();
}`;

const run = await client.actor('querystone/web-scraper').call({
  startUrls: [{ url: 'https://books.toscrape.com/catalogue/category/books/science_22/index.html' }],
  pageFunction,
  proxyConfiguration: { useScrapelyProxy: true },
}, { maxTotalChargeUsd: 0.05 });

console.log(run.status); // SUCCEEDED, FAILED, ABORTED or TIMED-OUT
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, items[0]);

maxTotalChargeUsd caps what the run can spend. The client is a drop-in replacement for apify-client; always pass baseUrl, because it doesn't read it from the environment.

We ran this input on 26 October 2026: 1 page fetched through the residential proxy, 14 books saved, 5.2 seconds. The first five rows:

From the run log:

2026-10-26T14:42:10.155Z [Scrapely] Container started (setup 0.9 s)
2026-10-26T14:42:11.956Z [scrapely.proxy_configuration] INFO  Platform proxy access check succeeded.
2026-10-26T14:42:14.058Z [scrapely.actor] INFO  Done: 1 pages fetched, 14 records saved in 2 s
2026-10-26T14:42:14.058Z [scrapely.actor] INFO  [Status message]: Saved 14 results from 1 page.

Web Scraper charges $0.000715 per result with platform usage included, so the run cost 14 × $0.000715 = $0.0100 at list price (the charge on our account was lower because of a plan discount). The free plan's $5 of monthly credit covers about 7,000 rows like these.

What to watch out for

• Blocking. Fast requests from one IP get rate limited. Keep delays, honour 429 with backoff, and use rotating residential proxies for bigger jobs; our guide to web scraping without getting blocked covers the layers. On Scrape.ly, the proxy is part of the run.
• Memory with browsers. Every Playwright or Puppeteer page holds a renderer process. Close pages you're done with, cap concurrency, and block images when you don't need them.
• Selectors break. Validate each record (title present, price parses) and fail loudly. With Store Actors, the publisher fixes selectors when a site changes.
• Personal data and terms. Stick to public data, read the site's terms and robots.txt, and don't collect personal data you don't need. Scrape.ly Store Actors only collect publicly available data shown to any visitor.

FAQ

Is Node.js good for web scraping?

Yes. Node's non-blocking I/O handles many concurrent requests well, Cheerio is fast, and Playwright and Puppeteer were built for JavaScript first. Crawlee adds a full crawling framework, and you can deploy the same code to the cloud as a Scrape.ly Actor.

Which is better for Node.js web scraping, Cheerio or Puppeteer?

They do different jobs. Cheerio parses HTML you've already downloaded and is many times lighter, so use it whenever the data is in the page source. Puppeteer (or Playwright) runs a real browser, which you need only when JavaScript renders the content or you must click and scroll.

Is axios or fetch better for scraping?

Both work. fetch is built in and enough for most scripts. axios gives you instance-wide defaults, interceptors and a timeout option in one place, which is convenient for crawlers. For large crawls, Crawlee's HTTP crawler replaces both.

Can I do web scraping with JavaScript in the browser?

Not usefully for other sites: the browser's same-origin policy blocks reading most other domains' pages from your page's scripts. Run scraping code in Node.js instead, or in a headless browser you control.

JavaScript or Python for web scraping?

Both have mature tools. Python has BeautifulSoup and Scrapy; JavaScript has Cheerio, Playwright, Puppeteer and Crawlee. Pick the language your team knows. For the Python side, see web scraping with Python; for the JVM, Java web scraping; and for a language-neutral walkthrough, how to build a web scraper.