Crawlee + Playwright (TypeScript) template

Web scraper using headless Chromium driven by Playwright, written in TypeScript.

Create it locally

$scrapely create my-actor --template crawlee-playwright-ts

Main file

.TS

src/main.ts

/**
 * This template is a production ready boilerplate for developing with `PlaywrightCrawler`.
 * Use this to bootstrap your projects using the most up-to-date code.
 * If you're looking for examples or want to learn more, see README.
 */

// For more information, see https://crawlee.dev
import { PlaywrightCrawler } from '@crawlee/playwright';
// Scrapely SDK - toolkit for building Scrapely Actors
import { Actor } from 'scrapely';

// this is ESM project, and as such, it requires you to specify extensions in your relative imports
// read more about this here: https://nodejs.org/docs/latest-v18.x/api/esm.html#mandatory-file-extensions
// note that we need to use `.js` even when inside TS files
import { router } from './routes.js';

interface Input {
    startUrls: {
        url: string;
        method?: 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'TRACE' | 'OPTIONS' | 'CONNECT' | 'PATCH';
        headers?: Record<string, string>;
        userData: Record<string, unknown>;
    }[];
    maxRequestsPerCrawl: number;
}

// Initialize the Scrapely SDK
await Actor.init();

// Structure of input is defined in input_schema.json
const { startUrls = ['https://crawlee.dev'], maxRequestsPerCrawl = 100 } =
    (await Actor.getInput<Input>()) ?? ({} as Input);

// `checkAccess` flag ensures the proxy credentials are valid, but the check can take a few hundred milliseconds.
// Disable it for short runs if you are sure your proxy configuration is correct
const proxyConfiguration = await Actor.createProxyConfiguration({ checkAccess: true });

const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    maxRequestsPerCrawl,
    requestHandler: router,
    launchContext: {
        launchOptions: {
            args: [
                '--disable-gpu', // Mitigates the "crashing GPU process" issue in Docker containers
            ],
        },
    },
});

await crawler.run(startUrls);

// Exit successfully
await Actor.exit();

Crawlee + Playwright (TypeScript)

A production-ready Scrapely Actor template built on PlaywrightCrawler in TypeScript. Give it a list of starting URLs and it loads each page in a headless Chromium browser, enqueues the links it finds, and stores the page title and URL in the default dataset.

Included features

Crawlee and the Scrapely SDK — crawling, request queues and storage
TypeScript with a strict tsconfig.json
Input schema — define and validate the Actor's input (.actor/input_schema.json)
Dataset schema — structured results with a preview view (.actor/dataset_schema.json)
Proxy configuration — rotate IP addresses to avoid blocking (Actor.createProxyConfiguration())
ESLint + Prettier + Vitest — linting, formatting and a runnable test

Getting started locally

npm install
npm run start      # run the Actor
npm run test       # run the test suite
npm run build      # compile to dist/

Input is read from .actor/input_schema.json. By default the Actor starts from https://crawlee.dev; change startUrls to point it anywhere.

Deploying to Scrapely

Create an Actor from this template in the Scrapely Console (Develop new → template → Git). Scrapely creates a repository in your account, pushes this code, and builds the Actor. Every push to your repository triggers a new build.