---
title: "Storage · Scrape.ly Docs"
description: "Datasets, key-value stores, and request queues, with retention rules and limits."
url: https://scrape.ly/docs/storage
markdown: https://scrape.ly/docs/storage.md
---

# Storage

> Datasets, key-value stores, and request queues, with retention rules and limits.

Every run on Scrape.ly gets three storages: a **dataset** for results, a **key-value store** for files and single records, and a **request queue** for the URLs to crawl. This guide shows how to use each one from the SDKs, the CLI, and the API, and how long the platform keeps them.

## Overview

| Storage | Shape | Best for | Typical use |
| --- | --- | --- | --- |
| Dataset | Append-only list of JSON items | Uniform records | Products, articles, listings |
| Key-value store | Records addressed by key, any content type | Files, state, single outputs | Screenshots, HTML, `OUTPUT`, crawl state |
| Request queue | Deduplicated queue of requests | Crawl frontiers | URLs to visit, retries |

Each run has its own default (unnamed) dataset, key-value store, and request queue. Their IDs are on the run object as `defaultDatasetId`, `defaultKeyValueStoreId`, and `defaultRequestQueueId`. You can also create **named** storages, which outlive runs and can be shared between Actors.

On the platform, the SDKs reach storage through the API using the run's token and storage IDs. Locally, with no platform environment (for example under `scrapely run`), both SDKs use Crawlee's local storage in `./storage`. `scrapely run` empties the default local storages before each run unless you pass `--no-purge`.

## Retention

| Storage | How long it's kept |
| --- | --- |
| Unnamed (including each run's defaults) | 2 days (48 hours) on the free plan, except your latest 10 runs, which are kept with their results and logs until you delete them; 31, 60 or 90 days with an active Starter, Scale or Business plan |
| Named | Until you delete it, as long as your account has a balance |

If your balance reaches zero, nothing is deleted at once. Unnamed storage is deleted 3 days later. Named storage is kept for 3 days or your plan's unnamed retention, whichever is longer, then deleted. Topping up before then stops the clock.

Download anything you need to keep, or write it to a named storage.

## Datasets

A dataset is an append-only list of JSON objects.

### Pushing items

```js [JavaScript]
import { Actor } from 'scrapely';

await Actor.pushData({ url: 'https://example.com', title: 'Example' });

await Actor.pushData([
    { url: 'https://example.com/a', title: 'A' },
    { url: 'https://example.com/b', title: 'B' },
]);
```

```python [Python]
from scrapely import Actor

await Actor.push_data({"url": "https://example.com", "title": "Example"})

await Actor.push_data([
    {"url": "https://example.com/a", "title": "A"},
    {"url": "https://example.com/b", "title": "B"},
])
```

Items must be JSON objects. Fields whose names start with `#` are dropped when you read with `skipHidden=true` or `clean=true`.

For pay-per-event Actors, `pushData` can charge an event per item and stops writing once the user's budget runs out. See [Publish and monetize](https://scrape.ly/docs/publishing.md).

Open a named dataset. It's created if it doesn't exist yet:

```js [JavaScript]
const dataset = await Actor.openDataset('my-shared-results');
await dataset.pushData({ shared: true });
```

```python [Python]
dataset = await Actor.open_dataset(name="my-shared-results")
await dataset.push_data({"shared": True})
```

### Reading and exporting

From the CLI:

```bash
scrapely datasets ls
scrapely datasets get-items <datasetId> --limit 20
scrapely datasets get-items <datasetId> --format csv > results.csv
scrapely datasets create my-shared-results
```

`get-items` writes to standard output. It supports `--format` with `json` (default), `jsonl`, `csv`, `html`, `rss`, `xml`, or `xlsx`, plus `--limit` and `--offset`. `scrapely call -o` prints the default dataset when a remote run finishes. See [CLI storage commands](https://scrape.ly/docs/cli-storage.md).

From the API:

```bash
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/datasets/<datasetId>/items?format=csv&limit=100"
```

`GET /v2/datasets/:datasetId/items` also supports `offset`, `fields`, `omit`, `unwind`, `flatten`, `clean`, `skipHidden`, `skipEmpty`, `desc`, and CSV options such as `delimiter` and `skipHeaderRow`. Large reads are streamed. Statistics and `sortBy` refuse datasets with more than 10,000 items or 32 MB of data. See the [Datasets API](https://scrape.ly/docs/api-datasets.md).

## Key-value stores

A key-value store holds records addressed by string keys. Each record has a content type: JSON, text, images, or anything else.

### Setting and getting values

```js [JavaScript]
import { Actor } from 'scrapely';

await Actor.setValue('state', { lastCursor: 42 });
const state = await Actor.getValue('state');

await Actor.setValue('screenshot.png', buffer, { contentType: 'image/png' });
await Actor.setValue('state', null); // deletes the record
```

```python [Python]
from scrapely import Actor

await Actor.set_value("state", {"lastCursor": 42})
state = await Actor.get_value("state")
```

### Special records

- `INPUT` holds the run's input. The platform writes it when the run starts.
- `OUTPUT` is the conventional key for a single result. There's no separate output route, so read the record from the run's default store.

```bash
scrapely kvs get-value <storeId> OUTPUT
```

```bash
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/key-value-stores/<storeId>/records/OUTPUT"
```

### Keys and bulk download

```bash
scrapely kvs ls
scrapely kvs keys <storeId>
scrapely kvs get-value <storeId> screenshot.png > shot.png
```

`kvs` is an alias for `key-value-stores`.

Through the API, list keys with `prefix`, `limit`, and `exclusiveStartKey`. `GET /v2/key-value-stores/:storeId/records` downloads every record as a ZIP, filtered by `prefix` if you pass one.

```
GET /v2/key-value-stores/:storeId/keys
GET /v2/key-value-stores/:storeId/records
GET /v2/key-value-stores/:storeId/records/:key
PUT /v2/key-value-stores/:storeId/records/:key
DELETE /v2/key-value-stores/:storeId/records/:key
```

See the [Key-value stores API](https://scrape.ly/docs/api-key-value-stores.md).

## Request queues

A request queue is a deduplicated queue of requests, usually URLs. Crawlee crawlers use the run's default queue automatically. You can also drive a queue directly.

```js [JavaScript]
import { Actor } from 'scrapely';

const queue = await Actor.openRequestQueue();
await queue.addRequest({ url: 'https://example.com' });
const request = await queue.fetchNextRequest();
if (request) await queue.markRequestHandled(request);
```

```python [Python]
from scrapely import Actor

queue = await Actor.open_request_queue()
await queue.add_request("https://example.com")
```

In Python, `add_request()` takes a URL string or a Crawlee `Request`, not a plain dict. See the [Python SDK storages guide](https://scrape.ly/docs/sdk-python-storages.md).

The CLI has no request queue subcommands. Use the SDK or the API.

### Deduplication, batches, and locks

Queues deduplicate by `uniqueKey`, which defaults to the normalized URL. Adding the same request twice does nothing.

For parallel workers, use the batch and lock endpoints:

```
POST /v2/request-queues/:queueId/requests/batch
DELETE /v2/request-queues/:queueId/requests/batch
POST /v2/request-queues/:queueId/head/lock
PUT /v2/request-queues/:queueId/requests/:requestId/lock
DELETE /v2/request-queues/:queueId/requests/:requestId/lock
GET /v2/request-queues/:queueId/requests
```

- Batch add and batch delete take a JSON array of at most 25 requests.
- `head/lock` claims requests so two workers don't process the same URL.
- `GET .../requests` takes `filter=locked` or `filter=pending` and pages with a cursor.

See the [Request queues API](https://scrape.ly/docs/api-request-queues.md).

## Limits

- JSON request bodies are limited to 10 MB. Binary and text uploads to key-value stores are limited to 100 MB.
- Creating storages, writing key-value records, and adding requests all need a positive balance. Creating a storage with a name you already use returns the existing storage.
- Storage reads, writes, and stored data are billed as usage. See **Billing** in the dashboard or `GET /v2/users/me/limits`.

## What's next

- [Proxy](https://scrape.ly/docs/proxy.md): managed and custom proxies
- [Security](https://scrape.ly/docs/security.md): who can read your storages
- [CLI storage commands](https://scrape.ly/docs/cli-storage.md)
- [Datasets API](https://scrape.ly/docs/api-datasets.md), [Key-value stores API](https://scrape.ly/docs/api-key-value-stores.md), [Request queues API](https://scrape.ly/docs/api-request-queues.md)

---

- Previous: [How Actors work](https://scrape.ly/docs/actors.md)
- Next: [Proxy](https://scrape.ly/docs/proxy.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)