Crawlee + Puppeteer template

Web scraper using headless Chrome driven by Puppeteer. Handles JavaScript-heavy sites.

Create it locally

$scrapely create my-actor --template crawlee-puppeteer-chrome

Main file

.JS

src/main.js

// Web scraping and browser automation library (Read more at https://crawlee.dev)
import { PuppeteerCrawler } from '@crawlee/puppeteer';
// Scrapely SDK - toolkit for building Scrapely Actors
import { Actor } from 'scrapely';

// this is ESM project, and as such, it requires you to specify extensions in your relative imports
// read more about this here: https://nodejs.org/docs/latest-v18.x/api/esm.html#mandatory-file-extensions
import { router } from './routes.js';

// The init() call configures the Actor to correctly work with the platform-provided environment - mainly the storage infrastructure. It is necessary that every Actor performs an init() call.
await Actor.init();

// Define the URLs to start the crawler with - get them from the input of the Actor or use a default list.
const input = await Actor.getInput();
const startUrls = input?.startUrls || [{ url: 'https://crawlee.dev' }];

// Create a proxy configuration that will rotate proxies from a proxy provider.
// `checkAccess` flag ensures the proxy credentials are valid, but the check can take a few hundred milliseconds.
// Disable it for short runs if you are sure your proxy configuration is correct
const proxyConfiguration = await Actor.createProxyConfiguration({ checkAccess: true });

// Create a PuppeteerCrawler that will use the proxy configuration and and handle requests with the router from routes.js file.
const crawler = new PuppeteerCrawler({
    proxyConfiguration,
    requestHandler: router,
    launchContext: {
        launchOptions: {
            args: [
                '--disable-gpu', // Mitigates the "crashing GPU process" issue in Docker containers
                '--no-sandbox', // Mitigates the "sandboxed" process issue in Docker containers
            ],
        },
    },
});

// Run the crawler with the start URLs and wait for it to finish.
await crawler.run(startUrls);

// Gracefully exit the Actor process. It's recommended to quit all Actors with an exit().
await Actor.exit();

Crawlee + Puppeteer

A production-ready Scrapely Actor template built on PuppeteerCrawler. Give it a list of starting URLs and it loads each page in a headless Chrome browser, enqueues the links it finds, and stores the page title and URL in the default dataset.

Included features

Crawlee and the Scrapely SDK — crawling, request queues and storage
Input schema — define and validate the Actor's input (.actor/input_schema.json)
Dataset schema — structured results with a preview view (.actor/dataset_schema.json)
Proxy configuration — rotate IP addresses to avoid blocking (Actor.createProxyConfiguration())
ESLint + Prettier + Vitest — linting, formatting and a runnable test

Getting started locally

npm install
npm run start      # run the Actor
npm run test       # run the test suite

Input is read from .actor/input_schema.json. By default the Actor starts from https://crawlee.dev; change startUrls to point it anywhere.

Deploying to Scrapely

Create an Actor from this template in the Scrapely Console (Develop new → template → Git). Scrapely creates a repository in your account, pushes this code, and builds the Actor. Every push to your repository triggers a new build.