TypeScript site crawler (Crawlee + Cheerio) template

A fast crawler that follows links and gets data from static pages, with Crawlee handling retries, queues, and proxies. Good for blogs, news, or product listings, but it can't run client-side JavaScript.

Create it locally

$scrapely create my-actor --template ts-crawlee-cheerio

Main file

.TS

src/main.ts

// Crawlee - web scraping and browser automation library (Read more at https://crawlee.dev)
import { CheerioCrawler } from '@crawlee/cheerio';
import { Actor } from 'scrapely';

// this is ESM project, and as such, it requires you to specify extensions in your relative imports
// read more about this here: https://nodejs.org/docs/latest-v18.x/api/esm.html#mandatory-file-extensions
// note that we need to use `.js` even when inside TS files
import { router } from './routes.js';

interface Input {
    startUrls: {
        url: string;
        method?: 'GET' | 'HEAD' | 'POST' | 'PUT' | 'DELETE' | 'TRACE' | 'OPTIONS' | 'CONNECT' | 'PATCH';
        headers?: Record<string, string>;
        userData: Record<string, unknown>;
    }[];
    maxRequestsPerCrawl: number;
}

// The init() call configures the Actor to correctly work with the platform-provided environment - mainly the storage infrastructure. It is necessary that every Actor performs an init() call.
await Actor.init();

// Structure of input is defined in input_schema.json
const { startUrls = ['https://crawlee.dev'], maxRequestsPerCrawl = 100 } =
    (await Actor.getInput<Input>()) ?? ({} as Input);

// `checkAccess` flag ensures the proxy credentials are valid, but the check can take a few hundred milliseconds.
// Disable it for short runs if you are sure your proxy configuration is correct
const proxyConfiguration = await Actor.createProxyConfiguration({ checkAccess: true });

const crawler = new CheerioCrawler({
    proxyConfiguration,
    maxRequestsPerCrawl,
    requestHandler: router,
});

await crawler.run(startUrls);

// Gracefully exit the Actor process. It's recommended to quit all Actors with an exit()
await Actor.exit();

TypeScript site crawler (Crawlee + Cheerio)

Included features

Scrapely SDK — Actor lifecycle, input, storage and proxies
Crawlee — crawling, request queues, retries and storage
Cheerio — fast HTML parsing
Input schema — define and validate the Actor's input (.actor/input_schema.json)
Dataset schema — structured results with a preview view (.actor/dataset_schema.json)

Getting started locally

npm install
npm run start      # run the Actor
npm run build      # compile to dist/

Deploying to Scrapely

Create an Actor from this template in the Scrapely Console (Develop new → template → Git). Scrapely creates a repository in your account, pushes this code, and builds the Actor. Every push to your repository triggers a new build.