---
title: "Storages & Crawlee · Scrape.ly Docs"
description: "Open datasets, key-value stores, and request queues, and run Crawlee crawlers."
url: https://scrape.ly/docs/sdk-node-storages
markdown: https://scrape.ly/docs/sdk-node-storages.md
---

# Storages & Crawlee

> Open datasets, key-value stores, and request queues, and run Crawlee crawlers.

Each run has a default dataset, key-value store, and request queue, and can open named ones. The SDK returns Crawlee's `Dataset`, `KeyValueStore`, and `RequestQueue` classes. On the platform they're backed by the Scrape.ly API; locally they use Crawlee storage in `./storage`. For storage concepts and retention, see [Storage](https://scrape.ly/docs/storage.md).

## Datasets, key-value stores, and request queues

### Opening storages

```ts
Actor.openDataset<T>(nameOrIdOrOptions?, openOptions?): Promise<Dataset<T>>
Actor.openKeyValueStore(nameOrIdOrOptions?, openOptions?): Promise<KeyValueStore>
Actor.openRequestQueue(nameOrIdOrOptions?, openOptions?): Promise<RequestQueue>
```

All three require `Actor.init()`. With no argument (or `null`), they open the run's default storage. The first argument also accepts a string (ID or name) or an identifier object:

```ts
await Actor.openDataset();                       // default dataset
await Actor.openDataset('results');              // ID or name
await Actor.openDataset({ id: 'dataset-id' });
await Actor.openDataset({ name: 'named-results' });
await Actor.openDataset({ alias: 'results' });
```

An `alias` is looked up in `ACTOR_STORAGES_JSON`, which maps aliases to storage IDs per type (`datasets`, `keyValueStores`, `requestQueues`). If the alias isn't there, it's used as a storage name.

The second argument, `OpenStorageOptions`, is optional:

| Option | Meaning |
| --- | --- |
| `config` | SDK `Configuration` instance to open the storage with |
| `forceCloud` | Use API-backed storage even if the SDK fell back to local storage. Needs `SCRAPELY_API_URL`, `SCRAPELY_TOKEN`, and `ACTOR_RUN_ID`; without them, the SDK logs a warning and uses local storage |

> [!NOTE]
> On a platform run, `KeyValueStore.open()` is patched and ignores its options, so `openOptions` has no effect on key-value stores there. Storage is already API-backed.

### Dataset

```ts
const dataset = await Actor.openDataset<{ url: string; title: string }>();
await dataset.pushData({ url, title });

const { items } = await dataset.getData({ limit: 100 });
```

Use datasets for append-only results that callers list and export. `Actor.pushData()` is a shortcut for the default dataset; see [Input & output](https://scrape.ly/docs/sdk-node-input-output.md#actorpushdata). How callers download them is covered in [Results](https://scrape.ly/docs/results.md).

### Key-value store

```ts
const store = await Actor.openKeyValueStore();
await store.setValue('SCREENSHOT', pngBuffer, { contentType: 'image/png' });
const saved = await store.getValue<{ cursor: string }>('STATE');
const exists = await store.recordExists('OUTPUT');

await store.forEachKey(async (key) => {
    console.log(key);
});
```

Use key-value stores for JSON state, text, files, and screenshots. On a platform run the SDK adds these methods to every opened store:

| Method | Returns |
| --- | --- |
| `listKeys({ prefix?, limit?, exclusiveStartKey?, collection? })` | One page of keys from the API |
| `getAutoSavedValue(key, defaultValue)` | An object that is saved back on every `persistState` event (see [`Actor.useState()`](https://scrape.ly/docs/sdk-node-proxy-events.md#actorusestate)) |
| `getPublicUrl(key)` | `{SCRAPELY_API_URL}/v2/key-value-stores/{storeId}/records/{key}` |

> [!WARNING]
> `getPublicUrl()` builds the URL from `SCRAPELY_API_URL`, the API address the container uses internally, so it isn't reachable from outside the platform. To give callers a link, use the API client, which uses the public API URL and signs the link when the store requires it:
>
> `const url = await Actor.newClient().keyValueStore(store.id).getRecordPublicUrl('OUTPUT');`
>
> See [Key-value stores API](https://scrape.ly/docs/api-key-value-stores.md) for access rules.

### Request queue

```ts
const queue = await Actor.openRequestQueue();
await queue.addRequest({ url: 'https://example.com/next' });

const request = await queue.fetchNextRequest();
if (request) {
    try {
        await processRequest(request);
        await queue.markRequestHandled(request);
    } catch (error) {
        await queue.reclaimRequest(request);
        throw error;
    }
}
```

A fetched request is locked to your run until you mark it handled or reclaim it. Requests are deduplicated by `uniqueKey` (the normalized URL unless you set one), so retries of the same URL don't create duplicates. Crawlers manage this for you; call these methods directly only when you process a queue by hand.

## Crawlee integration

The SDK re-exports Crawlee's crawlers: `BasicCrawler`, `BrowserCrawler`, `CheerioCrawler`, `HttpCrawler`, `JSDOMCrawler`, `LinkeDOMCrawler`, `PlaywrightCrawler`, and `PuppeteerCrawler`. It also re-exports `Dataset`, `KeyValueStore`, `RequestQueue`, the logging classes (`log`, `Log`, `LogLevel`, `Logger`, `LoggerJson`, `LoggerText`), and the `playwrightUtils` and `puppeteerUtils` namespaces.

`Request` and `RequestList` aren't re-exported. Import them from `crawlee` if you need them.

### Complete crawler example

```ts
import { Actor, PlaywrightCrawler } from 'scrapely';

type Input = { startUrls: string[] };

await Actor.main(async () => {
    const { startUrls } = await Actor.getInputOrThrow<Input>();
    if (!Array.isArray(startUrls) || startUrls.length === 0) {
        throw new Error('startUrls must contain at least one URL');
    }

    const crawler = new PlaywrightCrawler({
        maxConcurrency: 5,
        async requestHandler({ request, page, enqueueLinks }) {
            await Actor.pushData({
                url: request.url,
                title: await page.title(),
            });
            await enqueueLinks({ strategy: 'same-domain' });
        },
        async failedRequestHandler({ request, error }) {
            console.error(`Request failed: ${request.url}`, error);
        },
    });

    await crawler.run(startUrls);
});
```

Browser crawlers launch a browser in the container, or on your machine locally. If the Actor has **Use Scrapely browser** turned on, platform runs connect to the [Scrape.ly browser](https://scrape.ly/docs/browser.md) instead; see [Browsers and remote CDP](https://scrape.ly/docs/sdk-node-environment.md#browsers-and-remote-cdp).

### Request list sources

`ScrapelyRequestList` turns the standard `requestListSources` input field into Crawlee requests:

```ts
import { Actor, CheerioCrawler, ScrapelyRequestList } from 'scrapely';

await Actor.main(async () => {
    const input = (await Actor.getInput<{ requestListSources?: any[] }>()) ?? {};
    const { requests } = await ScrapelyRequestList.open(input.requestListSources ?? []);

    const crawler = new CheerioCrawler({
        async requestHandler({ request, $ }) {
            await Actor.pushData({ url: request.url, title: $('title').text() });
        },
    });
    await crawler.run(requests);
});
```

Each source entry has either a `url`, or a `requestsFromUrl` (also `requests_from_url`) pointing at a text file. The SDK downloads that file and turns every URL found in it into a request. Entries may also set `method`, `payload`, `headers`, and `userData` (also `user_data`), which apply to every request created from them. If a `requestsFromUrl` download fails, `open()` throws.

### `utils.enqueueLinks()`

The root `enqueueLinks` export (also `Actor.utils.enqueueLinks`) is the legacy SDK v2 helper for Cheerio pages. It is not the `enqueueLinks` from the crawler context.

```ts
import { utils } from 'scrapely';

await utils.enqueueLinks({
    $,
    requestQueue,
    selector: 'a.product',
    baseUrl: request.loadedUrl,
});
```

| Option | Default | Meaning |
| --- | --- | --- |
| `$` | required | Cheerio instance |
| `requestQueue` | required | Queue to add requests to |
| `selector` | `'a[href]'` | Links to collect |
| `baseUrl` | — | Resolves relative `href` values |
| `userData` | `{}` | Copied to every request |
| `transformRequestFunction` | — | Return a modified `{ url, userData }`, or `null` to skip |

URLs are deduplicated before they're added. New code inside a crawler should use the context's `enqueueLinks()` instead.

---

- Previous: [Input & output](https://scrape.ly/docs/sdk-node-input-output.md)
- Next: [Proxy & events](https://scrape.ly/docs/sdk-node-proxy-events.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)