Storage

Every run on Scrape.ly gets three storages: a dataset for results, a key-value store for files and single records, and a request queue for the URLs to crawl. This guide shows how to use each one from the SDKs, the CLI, and the API, and how long the platform keeps them.

Overview

Each run has its own default (unnamed) dataset, key-value store, and request queue. Their IDs are on the run object as defaultDatasetId, defaultKeyValueStoreId, and defaultRequestQueueId. You can also create named storages, which outlive runs and can be shared between Actors.

On the platform, the SDKs reach storage through the API using the run's token and storage IDs. Locally, with no platform environment (for example under scrapely run), both SDKs use Crawlee's local storage in ./storage. scrapely run empties the default local storages before each run unless you pass --no-purge.

Retention

If your balance reaches zero, nothing is deleted at once. Unnamed storage is deleted 3 days later. Named storage is kept for 3 days or your plan's unnamed retention, whichever is longer, then deleted. Topping up before then stops the clock.

Download anything you need to keep, or write it to a named storage.

Datasets

A dataset is an append-only list of JSON objects.

Pushing items

import { Actor } from 'scrapely';

await Actor.pushData({ url: 'https://example.com', title: 'Example' });

await Actor.pushData([
    { url: 'https://example.com/a', title: 'A' },
    { url: 'https://example.com/b', title: 'B' },
]);

Items must be JSON objects. Fields whose names start with # are dropped when you read with skipHidden=true or clean=true.

For pay-per-event Actors, pushData can charge an event per item and stops writing once the user's budget runs out. See Publish and monetize.

Open a named dataset. It's created if it doesn't exist yet:

const dataset = await Actor.openDataset('my-shared-results');
await dataset.pushData({ shared: true });

Reading and exporting

From the CLI:

scrapely datasets ls
scrapely datasets get-items <datasetId> --limit 20
scrapely datasets get-items <datasetId> --format csv > results.csv
scrapely datasets create my-shared-results

get-items writes to standard output. It supports --format with json (default), jsonl, csv, html, rss, xml, or xlsx, plus --limit and --offset. scrapely call -o prints the default dataset when a remote run finishes. See CLI storage commands.

From the API:

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/datasets/<datasetId>/items?format=csv&limit=100"

GET /v2/datasets/:datasetId/items also supports offset, fields, omit, unwind, flatten, clean, skipHidden, skipEmpty, desc, and CSV options such as delimiter and skipHeaderRow. Large reads are streamed. Statistics and sortBy refuse datasets with more than 10,000 items or 32 MB of data. See the Datasets API.

Key-value stores

A key-value store holds records addressed by string keys. Each record has a content type: JSON, text, images, or anything else.

Setting and getting values

import { Actor } from 'scrapely';

await Actor.setValue('state', { lastCursor: 42 });
const state = await Actor.getValue('state');

await Actor.setValue('screenshot.png', buffer, { contentType: 'image/png' });
await Actor.setValue('state', null); // deletes the record

Special records

• INPUT holds the run's input. The platform writes it when the run starts.
• OUTPUT is the conventional key for a single result. There's no separate output route, so read the record from the run's default store.

scrapely kvs get-value <storeId> OUTPUT

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/key-value-stores/<storeId>/records/OUTPUT"

Keys and bulk download

scrapely kvs ls
scrapely kvs keys <storeId>
scrapely kvs get-value <storeId> screenshot.png > shot.png

kvs is an alias for key-value-stores.

Through the API, list keys with prefix, limit, and exclusiveStartKey. GET /v2/key-value-stores/:storeId/records downloads every record as a ZIP, filtered by prefix if you pass one.

See the Key-value stores API.

Request queues

A request queue is a deduplicated queue of requests, usually URLs. Crawlee crawlers use the run's default queue automatically. You can also drive a queue directly.

import { Actor } from 'scrapely';

const queue = await Actor.openRequestQueue();
await queue.addRequest({ url: 'https://example.com' });
const request = await queue.fetchNextRequest();
if (request) await queue.markRequestHandled(request);

In Python, add_request() takes a URL string or a Crawlee Request, not a plain dict. See the Python SDK storages guide.

The CLI has no request queue subcommands. Use the SDK or the API.

Deduplication, batches, and locks

Queues deduplicate by uniqueKey, which defaults to the normalized URL. Adding the same request twice does nothing.

For parallel workers, use the batch and lock endpoints:

• Batch add and batch delete take a JSON array of at most 25 requests.
• head/lock claims requests so two workers don't process the same URL.
• GET .../requests takes filter=locked or filter=pending and pages with a cursor.

See the Request queues API.

Limits

• JSON request bodies are limited to 10 MB. Binary and text uploads to key-value stores are limited to 100 MB.
• Creating storages, writing key-value records, and adding requests all need a positive balance. Creating a storage with a name you already use returns the existing storage.
• Storage reads, writes, and stored data are billed as usage. See Billing in the dashboard or GET /v2/users/me/limits.

What's next

• Proxy: managed and custom proxies
• Security: who can read your storages
• CLI storage commands
• Datasets API, Key-value stores API, Request queues API