---
title: "Node.js Web Scraping: Cheerio to Crawlee | Scrape.ly"
description: "Node.js web scraping with tested code: fetch and axios with Cheerio, Playwright, Puppeteer, Crawlee, and deploying your scraper to the cloud as an Actor."
url: https://scrape.ly/blog/node-js-web-scraping
markdown: https://scrape.ly/blog/node-js-web-scraping.md
---

[Guides](https://scrape.ly/blog/category/guides.md)

# Node.js Web Scraping: Fetch, Cheerio, Playwright, Puppeteer and Crawlee

A hands-on guide to Node.js web scraping: fetch and axios with Cheerio, pagination with retries, Playwright and Puppeteer for JavaScript pages, Crawlee for real crawls, and deploying the scraper to the cloud from a Scrape.ly template. All code tested on 26 October 2026.

[Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md)

- Published

  Oct 26, 2026

- Reading time

  8 min read

![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a monitor showing the green Node.js hexagon logo beside a small web page, above lines of code, and drops it into a crate](https://scrape.ly/media/node-js-web-scraping.png?w=1080)

Node.js web scraping uses the same language the pages themselves are written in: `fetch` or axios download the HTML, Cheerio queries it with jQuery-style selectors, and Playwright or Puppeteer run a real browser when the content only appears after JavaScript. For anything bigger than a script, Crawlee adds the queue, retries and storage.

This guide walks through all of them with code we ran on 26 October 2026 (Node.js 23.9) against the practice sites [books.toscrape.com](https://books.toscrape.com/) and [quotes.toscrape.com](https://quotes.toscrape.com/). At the end, the same Crawlee scraper becomes a cloud Actor from a Scrape.ly template, and you'll see how to call ready-made scrapers from Node with one client call.

## Node.js web scraping libraries compared

| Library            | What it does                                             | Use it when                                     | Version tested |
| ------------------ | -------------------------------------------------------- | ----------------------------------------------- | -------------- |
| `fetch` (built in) | HTTP requests, no install                                | Simple pages and JSON APIs                      | Node.js 23.9.0 |
| axios              | HTTP client with interceptors and timeouts               | You want retries and shared config              | 1.20.0         |
| Cheerio            | Parses HTML, jQuery-style selectors                      | Always, next to fetch or axios                  | 1.2.0          |
| Playwright         | Drives Chromium, Firefox or WebKit                       | The data is rendered by JavaScript              | 1.64.1         |
| Puppeteer          | Drives Chrome over the DevTools protocol                 | Chrome-only browser automation                  | 25.16.0        |
| Crawlee            | Crawling framework: queue, retries, concurrency, storage | Crawls of many pages, or anything you'll deploy | 3.18.2         |

The rule of thumb: start with fetch and Cheerio, switch to Crawlee when you crawl, and use a browser only for pages that need one.

## Set up the project

Any current Node.js release works (`fetch` is built in since Node 18). The examples use ES modules and top-level `await`:

```shell
mkdir scraper && cd scraper
npm init -y
npm pkg set type=module
npm install cheerio axios
```

## Your first scraper: fetch and Cheerio

Inspect a book on books.toscrape.com and you'll find each one in an `<article class="product_pod">`, with the full title in the link's `title` attribute and the price in `p.price_color`:

```javascript
import * as cheerio from 'cheerio';

const response = await fetch('https://books.toscrape.com/', {
  headers: { 'User-Agent': 'books-tutorial/1.0 (you@example.com)' },
  signal: AbortSignal.timeout(15_000),
});
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const $ = cheerio.load(await response.text());

$('article.product_pod').slice(0, 3).each((i, el) => {
  const title = $(el).find('h3 a').attr('title');
  const price = $(el).find('p.price_color').text();
  console.log(title, price);
});
console.log($('article.product_pod').length, 'books on this page');
```

Output:

```plaintext
A Light in the Attic £51.77
Tipping the Velvet £53.74
Soumission £50.10
20 books on this page
```

Two habits to keep from day one. `fetch` doesn't throw on a 404 or 500, so check `response.ok` yourself, or you'll parse an error page and wonder why your selectors return nothing. And `AbortSignal.timeout()` gives the request a deadline; without one, a hung server hangs your script.

## Crawl every page: axios, retries and delays

The catalogue has 50 pages, each with a **next** link (`li.next a`). This crawler follows it, retries network errors, 429 and 5xx responses with exponential backoff, pauses one second between pages and saves JSON:

```javascript
import { writeFile } from 'node:fs/promises';
import { setTimeout as sleep } from 'node:timers/promises';
import axios from 'axios';
import * as cheerio from 'cheerio';

const RATINGS = { One: 1, Two: 2, Three: 3, Four: 4, Five: 5 };
const http = axios.create({
  timeout: 15_000,
  headers: { 'User-Agent': 'books-tutorial/1.0 (you@example.com)' },
});

async function get(url, attempt = 1) {
  try {
    return (await http.get(url)).data;
  } catch (err) {
    const status = err.response?.status;
    const retryable = !status  status === 429  status >= 500;
    if (!retryable  attempt === 3) throw err;
    await sleep(1000 * 2 ** attempt); // 2 s, then 4 s
    return get(url, attempt + 1);
  }
}

function parse(html, pageUrl) {
  const $ = cheerio.load(html);
  const books = $('article.product_pod').map((i, el) => {
    const card = $(el);
    const link = card.find('h3 a');
    return {
      title: link.attr('title'),
      price: Number(card.find('p.price_color').text().replace('£', '')),
      rating: RATINGS[card.find('p.star-rating').attr('class').split(' ')[1]],
      inStock: card.find('p.availability').text().includes('In stock'),
      url: new URL(link.attr('href'), pageUrl).href,
    };
  }).get();
  const next = $('li.next a').attr('href');
  return { books, next: next ? new URL(next, pageUrl).href : null };
}

const maxPages = Number(process.argv[2] ?? 50);
let url = 'https://books.toscrape.com/catalogue/page-1.html';
const all = [];
for (let page = 1; url && page <= maxPages; page++) {
  const { books, next } = parse(await get(url), url);
  all.push(...books);
  url = next;
  await sleep(1000); // one request per second
}

await writeFile('books.json', JSON.stringify(all, null, 2));
console.log(all.length, 'books');
console.log(all[0]);
```

With `node crawl.js 5` it fetched five pages in under 9 seconds and printed:

```plaintext
100 books
{
  title: 'A Light in the Attic',
  price: 51.77,
  rating: 3,
  inStock: true,
  url: 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'
}
```

Notes on the parts that matter:

- **`new URL(href, pageUrl)`** resolves relative links like `page-2.html` correctly. Never concatenate URL strings.
- **The retry rule** repeats only what can succeed on a second try: no response at all, 429, or a 5xx. A 404 fails immediately.
- **`.map(...).get()`** is the Cheerio idiom for turning a selection into a plain array. Without `.get()` you keep a Cheerio object.
- **The rating** hides in a class name (`star-rating Three`). Real sites hide data in odd places all the time.

### Check for a JSON API first

Many JavaScript-heavy pages load their data from an API you can call directly. Open DevTools, choose **Network**, filter by **Fetch/XHR** and reload. The infinite-scroll version of the quotes site uses one:

```javascript
const quotes = [];
for (let page = 1; ; page++) {
  const res = await fetch(`https://quotes.toscrape.com/api/quotes?page=${page}`);
  const data = await res.json();
  quotes.push(...data.quotes.map((q) => ({ text: q.text, author: q.author.name, tags: q.tags })));
  if (!data.has_next) break;
}
console.log(quotes.length, 'quotes');
```

`100 quotes` in about a second, no HTML parsing, no browser. Structured JSON also survives redesigns that would break CSS selectors.

## Scrape JavaScript pages with Playwright or Puppeteer

[quotes.toscrape.com/js](https://quotes.toscrape.com/js/) renders its quotes with a script, so `fetch` sees an empty page. A headless browser runs the script first. Install Playwright with `npm install playwright` and `npx playwright install chromium`:

```javascript
import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/');
await page.waitForSelector('div.quote'); // rendered by JavaScript

const quotes = await page.$$eval('div.quote', (cards) =>
  cards.map((card) => ({
    text: card.querySelector('span.text').textContent,
    author: card.querySelector('small.author').textContent,
  })),
);
console.log(quotes.length, quotes[0]);
await browser.close();
```

It printed `10` and the first quote, by Albert Einstein. The Puppeteer version (`npm install puppeteer`, which downloads its own Chrome) is nearly identical:

```javascript
import puppeteer from 'puppeteer';

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto('https://quotes.toscrape.com/js/', { waitUntil: 'domcontentloaded' });
await page.waitForSelector('div.quote');

const quotes = await page.$$eval('div.quote', (cards) =>
  cards.map((card) => ({
    text: card.querySelector('span.text').textContent,
    author: card.querySelector('small.author').textContent,
  })),
);
console.log(quotes.length, quotes[0]);
await browser.close();
```

Same result. The function you pass to `$$eval` runs inside the page, so it uses browser APIs (`querySelector`, `textContent`), not Cheerio. Which one to pick? Playwright supports three engines and auto-waits on every action; Puppeteer is Chrome-first and slightly lighter. Our [Playwright vs Puppeteer comparison](https://scrape.ly/blog/playwright-vs-puppeteer.md) goes deeper.

## Crawl at scale with Crawlee

Once you crawl many pages, you end up writing a queue, deduplication, retries, concurrency limits and somewhere to put the results. Crawlee ships all of that. `CheerioCrawler` fetches over HTTP and hands each page to your handler with Cheerio already loaded:

```javascript
import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
  maxRequestsPerCrawl: 5, // stop early while testing
  maxConcurrency: 2,
  async requestHandler({ $, request, enqueueLinks, log }) {
    log.info(`Scraping ${request.url}`);
    const books = $('article.product_pod').map((i, el) => ({
      title: $(el).find('h3 a').attr('title'),
      price: $(el).find('p.price_color').text(),
      url: new URL($(el).find('h3 a').attr('href'), request.loadedUrl).href,
    })).get();
    await Dataset.pushData(books);
    await enqueueLinks({ selector: 'li.next a' });
  },
});

await crawler.run(['https://books.toscrape.com/']);
await Dataset.exportToCSV('books');
console.log((await Dataset.getData()).total, 'books');
```

Install it with `npm install crawlee`. Our run finished with `Total 5 requests: 5 succeeded, 0 failed` and `100 books`, saved under `./storage` and exported to `storage/key_value_stores/default/books.csv`. `enqueueLinks` skips URLs it has already seen, failed requests are retried automatically (three times by default), and swapping `CheerioCrawler` for `PlaywrightCrawler` or `PuppeteerCrawler` gives you a browser with the same handler shape. Crawlee is also the base of the Scrape.ly JavaScript SDK, which is why the next step is short.

## Deploy your Node.js scraper as a Scrape.ly Actor

A script on your laptop stops when the laptop sleeps. As an Actor on Scrape.ly, the same code runs in the cloud on a schedule, writes to a hosted dataset you can download as CSV or JSON, and uses Scrape.ly's proxy. Start from a template; there are three for scraping in JavaScript:

| Template                                                                  | Starts with                     | For                                   |
| ------------------------------------------------------------------------- | ------------------------------- | ------------------------------------- |
| [`crawlee-cheerio-js`](https://scrape.ly/templates/crawlee-cheerio-js.md) | `CheerioCrawler`                | Static HTML, the fastest and cheapest |
| [`crawlee-playwright`](https://scrape.ly/templates/crawlee-playwright.md) | `PlaywrightCrawler` with Chrome | JavaScript-heavy pages                |
| `crawlee-puppeteer-chrome`                                                | `PuppeteerCrawler` with Chrome  | Puppeteer code you already have       |

Install the CLI, log in and create the project:

```shell
npm install -g scrapely-cli
export SCRAPELY_API_URL=https://api.scrape.ly
scrapely login
scrapely create books-scraper --template crawlee-cheerio-js && cd books-scraper
```

The template's `src/main.js` calls `Actor.init()`, reads `startUrls` and `maxRequestsPerCrawl` from the input, sets up the platform proxy and runs the router in `src/routes.js`. Replace the default handler there with the books logic:

```javascript
import { createCheerioRouter } from '@crawlee/cheerio';

export const router = createCheerioRouter();

router.addDefaultHandler(async ({ enqueueLinks, request, $, log, pushData }) => {
    log.info(`Scraping ${request.loadedUrl}`);
    const books = $('article.product_pod')
        .map((i, el) => ({
            title: $(el).find('h3 a').attr('title'),
            price: $(el).find('p.price_color').text(),
            inStock: $(el).find('p.availability').text().includes('In stock'),
            url: new URL($(el).find('h3 a').attr('href'), request.loadedUrl).href,
        }))
        .get();
    await pushData(books);
    await enqueueLinks({ selector: 'li.next a' });
});
```

Test it on your machine, then push it:

```shell
npm install scrapely   # the SDK, for local runs only
scrapely run -i '{"startUrls":[{"url":"https://books.toscrape.com/"}],"maxRequestsPerCrawl":5}'
scrapely push
scrapely call -i '{"startUrls":[{"url":"https://books.toscrape.com/"}],"maxRequestsPerCrawl":5}' -o
```

We created this project with CLI 0.1.16 and ran it locally: 5 requests, 0 failed, 100 books in `storage/datasets/default`. Locally the SDK logs that no proxy is configured and fetches directly; on the platform, Scrape.ly injects the proxy credentials into the run. The `npm install scrapely` line is there because the template doesn't list the SDK in `package.json` yet; pushed builds don't need it, since the `scrapely/actor-node` base image includes it. After `push`, add a schedule or a delivery to Google Sheets or Slack without touching the code; see [how to schedule and automate web scraping](https://scrape.ly/blog/schedule-web-scraping.md).

Sites with bot protection need more than a proxy. Turn on the Scrape.ly browser for your Actor and rebuild: Playwright and Puppeteer launches then connect to a hosted Chromium with anti-bot fingerprints, Cloudflare Turnstile solving and its own residential proxy, with no code changes.

## Call ready-made scrapers from Node.js

Often you don't need to write a scraper at all. The [Scrape.ly Store](https://scrape.ly/store.md) has 570+ Actors for sites like Amazon, Google Maps and LinkedIn, and the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) Actor runs your own page function on any public site. Call either from Node with the client (`npm install scrapely-client`):

```javascript
import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });

const pageFunction = `async function pageFunction(context) {
  const $ = context.jQuery;
  return $('article.product_pod').map((i, el) => ({
    title: $(el).find('h3 a').attr('title'),
    price: $(el).find('p.price_color').text(),
    rating: $(el).find('p.star-rating').attr('class').split(' ')[1],
  })).get();
}`;

const run = await client.actor('querystone/web-scraper').call({
  startUrls: [{ url: 'https://books.toscrape.com/catalogue/category/books/science_22/index.html' }],
  pageFunction,
  proxyConfiguration: { useScrapelyProxy: true },
}, { maxTotalChargeUsd: 0.05 });

console.log(run.status); // SUCCEEDED, FAILED, ABORTED or TIMED-OUT
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, items[0]);
```

`maxTotalChargeUsd` caps what the run can spend. The client is a drop-in replacement for `apify-client`; always pass `baseUrl`, because it doesn't read it from the environment.

We ran this input on 26 October 2026: 1 page fetched through the residential proxy, 14 books saved, 5.2 seconds. The first five rows:

| title                                                                | price  | rating |
| -------------------------------------------------------------------- | ------ | ------ |
| The Most Perfect Thing: Inside (and Outside) a Bird's Egg            | £42.96 | Four   |
| Immunity: How Elie Metchnikoff Changed the Course of Modern Medicine | £57.36 | Five   |
| Sorting the Beef from the Bull: The Science of Food Fraud Forensics  | £44.74 | Four   |
| Tipping Point for Planet Earth: How Close Are We to the Edge?        | £37.55 | One    |
| The Fabric of the Cosmos: Space, Time, and the Texture of Reality    | £55.91 | One    |

From the run log:

```plaintext
20261026T14:42:10.155Z [Scrapely] Container started (setup 0.9 s)
20261026T14:42:11.956Z [scrapely.proxy_configuration] INFO  Platform proxy access check succeeded.
20261026T14:42:14.058Z [scrapely.actor] INFO  Done: 1 pages fetched, 14 records saved in 2 s
20261026T14:42:14.058Z [scrapely.actor] INFO  [Status message]: Saved 14 results from 1 page.
```

Web Scraper charges $0.000715 per result with platform usage included, so the run cost 14 × $0.000715 = $0.0100 at list price (the charge on our account was lower because of a plan discount). The free plan's $5 of monthly credit covers about 7,000 rows like these.

Scrape.ly Store Web Scraper Run your own page function on any public site from Node.js. Link following, residential proxy and retries included. $0.71 per 1,000 results.

[Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md)

## What to watch out for

- **Blocking.** Fast requests from one IP get rate limited. Keep delays, honour 429 with backoff, and use rotating residential proxies for bigger jobs; our guide to [web scraping without getting blocked](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) covers the layers. On Scrape.ly, the proxy is part of the run.
- **Memory with browsers.** Every Playwright or Puppeteer page holds a renderer process. Close pages you're done with, cap concurrency, and block images when you don't need them.
- **Selectors break.** Validate each record (title present, price parses) and fail loudly. With Store Actors, the publisher fixes selectors when a site changes.
- **Personal data and terms.** Stick to public data, read the site's terms and `robots.txt`, and don't collect personal data you don't need. Scrape.ly Store Actors only collect publicly available data shown to any visitor.

## FAQ

### Is Node.js good for web scraping?

Yes. Node's non-blocking I/O handles many concurrent requests well, Cheerio is fast, and Playwright and Puppeteer were built for JavaScript first. Crawlee adds a full crawling framework, and you can deploy the same code to the cloud as a Scrape.ly Actor.

### Which is better for Node.js web scraping, Cheerio or Puppeteer?

They do different jobs. Cheerio parses HTML you've already downloaded and is many times lighter, so use it whenever the data is in the page source. Puppeteer (or Playwright) runs a real browser, which you need only when JavaScript renders the content or you must click and scroll.

### Is axios or fetch better for scraping?

Both work. `fetch` is built in and enough for most scripts. axios gives you instance-wide defaults, interceptors and a timeout option in one place, which is convenient for crawlers. For large crawls, Crawlee's HTTP crawler replaces both.

### Can I do web scraping with JavaScript in the browser?

Not usefully for other sites: the browser's same-origin policy blocks reading most other domains' pages from your page's scripts. Run scraping code in Node.js instead, or in a headless browser you control.

### JavaScript or Python for web scraping?

Both have mature tools. Python has BeautifulSoup and Scrapy; JavaScript has Cheerio, Playwright, Puppeteer and Crawlee. Pick the language your team knows. For the Python side, see [web scraping with Python](https://scrape.ly/blog/web-scraping-python.md); for the JVM, [Java web scraping](https://scrape.ly/blog/java-web-scraping.md); and for a language-neutral walkthrough, [how to build a web scraper](https://scrape.ly/blog/how-to-build-a-web-scraper.md).

About the author

[Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly

Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents.

- [Website](https://scrape.ly/index.md)

Keep reading

## More in Guides

- [![Pixel-art robot with an orange >\_ on its terminal head stands between two matching arcade cabinets, one with the teal Scrapy logo and one with a steaming soup bowl, thinking of the Scrape.ly mark next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/scrapy-vs-beautifulsoup.png?w=750)](https://scrape.ly/blog/scrapy-vs-beautifulsoup.md)
  ### [Scrapy vs BeautifulSoup: Which Python Scraping Tool to Use (Tested)](https://scrape.ly/blog/scrapy-vs-beautifulsoup.md)
  [Scrapy vs BeautifulSoup, tested on the same 1,000-book job: what each tool is, measured run times, retries, concurrency, exports and JavaScript pages, when to pick which, and how to run a Scrapy project in the cloud with one command.](https://scrape.ly/blog/scrapy-vs-beautifulsoup.md)
- [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from the one orange-lit globe in a ring of four blue-and-green globes turning round a small server, amber arrowheads marking the rotation, towards a web page with an image block](https://scrape.ly/media/rotating-proxies-for-web-scraping.png?w=750)](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md)
  ### [Rotating Proxies for Web Scraping: How Rotation Works, with Tested Code](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md)
  [Rotating proxies for web scraping spread your requests over many IP addresses. Per-request rotation vs sticky sessions, residential vs datacenter pools, tested Python and Crawlee code, and what rotation costs per GB on Scrape.ly and Evomi.](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md)
- [![Pixel-art robot with an orange >\_ on its terminal head, disguised in a dark fedora, a raised collar and a cloak, pulls a data card out of a browser window showing the green Puppeteer logo and drops it into a crate](https://scrape.ly/media/puppeteer-stealth.png?w=750)](https://scrape.ly/blog/puppeteer-stealth.md)
  ### [Puppeteer Stealth: Setup, What It Patches, and Where It Stops](https://scrape.ly/blog/puppeteer-stealth.md)
  [Puppeteer stealth is puppeteer-extra-plugin-stealth, a plugin that hides the signs of a headless Chrome. Its maintenance status in October 2026, tested setup with Puppeteer 25, before-and-after results, the header mismatches it creates, a Crawlee setup and the hosted alternative.](https://scrape.ly/blog/puppeteer-stealth.md)

## The internet is full of data. Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes.

[Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.)

No credit card. No setup calls.