Crawlee + Playwright template

Web scraper using headless Chromium driven by Playwright. Handles JavaScript-heavy sites.

Create it locally

$scrapely create my-actor --template crawlee-playwright

Main file

.JS

src/main.js

/**
 * This template is a production ready boilerplate for developing with `PlaywrightCrawler`.
 * Use this to bootstrap your projects using the most up-to-date code.
 * If you're looking for examples or want to learn more, see README.
 */

// For more information, see https://crawlee.dev
import { PlaywrightCrawler } from '@crawlee/playwright';
// Scrapely SDK - toolkit for building Scrapely Actors
import { Actor } from 'scrapely';

// this is ESM project, and as such, it requires you to specify extensions in your relative imports
// read more about this here: https://nodejs.org/docs/latest-v18.x/api/esm.html#mandatory-file-extensions
import { router } from './routes.js';

// Initialize the Scrapely SDK
await Actor.init();

const { startUrls = ['https://crawlee.dev'] } = (await Actor.getInput()) ?? {};

// `checkAccess` flag ensures the proxy credentials are valid, but the check can take a few hundred milliseconds.
// Disable it for short runs if you are sure your proxy configuration is correct
const proxyConfiguration = await Actor.createProxyConfiguration({ checkAccess: true });

const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    requestHandler: router,
    launchContext: {
        launchOptions: {
            args: [
                '--disable-gpu', // Mitigates the "crashing GPU process" issue in Docker containers
            ],
        },
    },
});

await crawler.run(startUrls);

// Exit successfully
await Actor.exit();

Crawlee + Playwright

A production-ready Scrapely Actor template built on PlaywrightCrawler. Give it a list of starting URLs and it loads each page in a headless Chromium browser, enqueues the links it finds, and stores the page title and URL in the default dataset.

Included features

Crawlee and the Scrapely SDK — crawling, request queues and storage
Input schema — define and validate the Actor's input (.actor/input_schema.json)
Dataset schema — structured results with a preview view (.actor/dataset_schema.json)
Proxy configuration — rotate IP addresses to avoid blocking (Actor.createProxyConfiguration())
ESLint + Prettier — linting and formatting

Getting started locally

npm install
npm run start      # run the Actor

Input is read from .actor/input_schema.json. By default the Actor starts from https://crawlee.dev; change startUrls to point it anywhere.

Deploying to Scrapely

Create an Actor from this template in the Scrapely Console (Develop new → template → Git). Scrapely creates a repository in your account, pushes this code, and builds the Actor. Every push to your repository triggers a new build.