Storages & Crawlee

Each run has a default dataset, key-value store, and request queue, and can open named ones. The SDK returns Crawlee's Dataset, KeyValueStore, and RequestQueue classes. On the platform they're backed by the Scrape.ly API; locally they use Crawlee storage in ./storage. For storage concepts and retention, see Storage.

Datasets, key-value stores, and request queues

Opening storages

Actor.openDataset<T>(nameOrIdOrOptions?, openOptions?): Promise<Dataset<T>>
Actor.openKeyValueStore(nameOrIdOrOptions?, openOptions?): Promise<KeyValueStore>
Actor.openRequestQueue(nameOrIdOrOptions?, openOptions?): Promise<RequestQueue>

All three require Actor.init(). With no argument (or null), they open the run's default storage. The first argument also accepts a string (ID or name) or an identifier object:

await Actor.openDataset();                       // default dataset
await Actor.openDataset('results');              // ID or name
await Actor.openDataset({ id: 'dataset-id' });
await Actor.openDataset({ name: 'named-results' });
await Actor.openDataset({ alias: 'results' });

An alias is looked up in ACTOR_STORAGES_JSON, which maps aliases to storage IDs per type (datasets, keyValueStores, requestQueues). If the alias isn't there, it's used as a storage name.

The second argument, OpenStorageOptions, is optional:

Dataset

const dataset = await Actor.openDataset<{ url: string; title: string }>();
await dataset.pushData({ url, title });

const { items } = await dataset.getData({ limit: 100 });

Use datasets for append-only results that callers list and export. Actor.pushData() is a shortcut for the default dataset; see Input & output. How callers download them is covered in Results.

Key-value store

const store = await Actor.openKeyValueStore();
await store.setValue('SCREENSHOT', pngBuffer, { contentType: 'image/png' });
const saved = await store.getValue<{ cursor: string }>('STATE');
const exists = await store.recordExists('OUTPUT');

await store.forEachKey(async (key) => {
    console.log(key);
});

Use key-value stores for JSON state, text, files, and screenshots. On a platform run the SDK adds these methods to every opened store:

Request queue

const queue = await Actor.openRequestQueue();
await queue.addRequest({ url: 'https://example.com/next' });

const request = await queue.fetchNextRequest();
if (request) {
    try {
        await processRequest(request);
        await queue.markRequestHandled(request);
    } catch (error) {
        await queue.reclaimRequest(request);
        throw error;
    }
}

A fetched request is locked to your run until you mark it handled or reclaim it. Requests are deduplicated by uniqueKey (the normalized URL unless you set one), so retries of the same URL don't create duplicates. Crawlers manage this for you; call these methods directly only when you process a queue by hand.

Crawlee integration

The SDK re-exports Crawlee's crawlers: BasicCrawler, BrowserCrawler, CheerioCrawler, HttpCrawler, JSDOMCrawler, LinkeDOMCrawler, PlaywrightCrawler, and PuppeteerCrawler. It also re-exports Dataset, KeyValueStore, RequestQueue, the logging classes (log, Log, LogLevel, Logger, LoggerJson, LoggerText), and the playwrightUtils and puppeteerUtils namespaces.

Request and RequestList aren't re-exported. Import them from crawlee if you need them.

Complete crawler example

import { Actor, PlaywrightCrawler } from 'scrapely';

type Input = { startUrls: string[] };

await Actor.main(async () => {
    const { startUrls } = await Actor.getInputOrThrow<Input>();
    if (!Array.isArray(startUrls) || startUrls.length === 0) {
        throw new Error('startUrls must contain at least one URL');
    }

    const crawler = new PlaywrightCrawler({
        maxConcurrency: 5,
        async requestHandler({ request, page, enqueueLinks }) {
            await Actor.pushData({
                url: request.url,
                title: await page.title(),
            });
            await enqueueLinks({ strategy: 'same-domain' });
        },
        async failedRequestHandler({ request, error }) {
            console.error(`Request failed: ${request.url}`, error);
        },
    });

    await crawler.run(startUrls);
});

Browser crawlers launch a browser in the container, or on your machine locally. If the Actor has Use Scrapely browser turned on, platform runs connect to the Scrape.ly browser instead; see Browsers and remote CDP.

Request list sources

ScrapelyRequestList turns the standard requestListSources input field into Crawlee requests:

import { Actor, CheerioCrawler, ScrapelyRequestList } from 'scrapely';

await Actor.main(async () => {
    const input = (await Actor.getInput<{ requestListSources?: any[] }>()) ?? {};
    const { requests } = await ScrapelyRequestList.open(input.requestListSources ?? []);

    const crawler = new CheerioCrawler({
        async requestHandler({ request, $ }) {
            await Actor.pushData({ url: request.url, title: $('title').text() });
        },
    });
    await crawler.run(requests);
});

Each source entry has either a url, or a requestsFromUrl (also requests_from_url) pointing at a text file. The SDK downloads that file and turns every URL found in it into a request. Entries may also set method, payload, headers, and userData (also user_data), which apply to every request created from them. If a requestsFromUrl download fails, open() throws.

utils.enqueueLinks()

The root enqueueLinks export (also Actor.utils.enqueueLinks) is the legacy SDK v2 helper for Cheerio pages. It is not the enqueueLinks from the crawler context.

import { utils } from 'scrapely';

await utils.enqueueLinks({
    $,
    requestQueue,
    selector: 'a.product',
    baseUrl: request.loadedUrl,
});

URLs are deduplicated before they're added. New code inside a crawler should use the context's enqueueLinks() instead.