JavaScript minimal site crawler (Crawlee + Cheerio) template

A minimal site crawler for developers who already know the Scrapely SDK and Crawlee.

Create it locally

$scrapely create my-actor --template js-bootstrap-cheerio-crawler

Main file

.JS

src/main.js

import { CheerioCrawler } from '@crawlee/cheerio';
import { Actor } from 'scrapely';

// this is ESM project, and as such, it requires you to specify extensions in your relative imports
// read more about this here: https://nodejs.org/docs/latest-v18.x/api/esm.html#mandatory-file-extensions
import { router } from './routes.js';

await Actor.init();

// `checkAccess` flag ensures the proxy credentials are valid, but the check can take a few hundred milliseconds.
// Disable it for short runs if you are sure your proxy configuration is correct
const proxyConfiguration = await Actor.createProxyConfiguration({ checkAccess: true });

const crawler = new CheerioCrawler({
    proxyConfiguration,
    requestHandler: router,
});

await crawler.run(['https://example.com']);

await Actor.exit();

JavaScript minimal site crawler (Crawlee + Cheerio)

A minimal site crawler for developers who already know the Scrapely SDK and Crawlee.

Included features

Scrapely SDK — Actor lifecycle, input, storage and proxies
Crawlee — crawling, request queues, retries and storage
Cheerio — fast HTML parsing
Input schema — define and validate the Actor's input (.actor/input_schema.json)
Dataset schema — structured results with a preview view (.actor/dataset_schema.json)

Getting started locally

npm install
npm run start      # run the Actor

Deploying to Scrapely

Create an Actor from this template in the Scrapely Console (Develop new → template → Git). Scrapely creates a repository in your account, pushes this code, and builds the Actor. Every push to your repository triggers a new build.