---
title: "Storages · Scrape.ly Docs"
description: "Open datasets, key-value stores, and request queues, locally or in the cloud."
url: https://scrape.ly/docs/sdk-python-storages
markdown: https://scrape.ly/docs/sdk-python-storages.md
---

# Storages

> Open datasets, key-value stores, and request queues, locally or in the cloud.

Each run has a default dataset, a default key-value store and a default request queue. You can also open other named or unnamed storages. The `Actor.open_*` methods return Crawlee's `Dataset`, `KeyValueStore` and `RequestQueue` classes. On the platform, these are backed by the Scrape.ly storage API. On your machine, they're backed by Crawlee's file-system storage. For the concepts behind the three storage types, see [Storage](https://scrape.ly/docs/storage.md).

## Opening storages

```python
dataset = await Actor.open_dataset()
store = await Actor.open_key_value_store()
queue = await Actor.open_request_queue()
```

All three methods share one signature:

```python
await Actor.open_dataset(name_or_id=None, options=None, *, id=None, name=None, alias=None, force_cloud=False)
```

| Argument | Meaning |
| --- | --- |
| `name_or_id` | If the value looks like a storage ID (a UUID), it's an ID. Otherwise it's a name |
| `id` | Open by storage ID |
| `name` | Open or create a named storage |
| `alias` | Open the storage mapped to this alias in `ACTOR_STORAGES_JSON`. An unresolved alias is treated as a name |
| `force_cloud` | Use Scrape.ly cloud storage even off the platform. Requires `SCRAPELY_TOKEN` or `APIFY_TOKEN` |
| `options` | `OpenStorageOptions(force_cloud=False, config=None)` |

With no arguments, these methods open the run's default storage. Passing `name_or_id` together with `id`, `name` or `alias` raises `ValueError`.

```python
await Actor.open_dataset("results")                 # name
await Actor.open_dataset(id="0f9c2b1e-...")         # id
await Actor.open_key_value_store(alias="cache")     # alias
await Actor.open_request_queue(name="frontier", force_cloud=True)
```

## Dataset

```python
dataset = await Actor.open_dataset()
await dataset.push_data({"url": url, "title": title})

async for item in dataset.iterate_items():
    print(item)

page = await dataset.get_data(offset=0, limit=100)
```

Datasets are append-only. Their methods are `push_data()`, `get_data()`, `iterate_items()`, `list_items()` and `export_to()`. Writing through `dataset.push_data()` doesn't charge pay-per-event events. Use `Actor.push_data()` for charged writes.

## Key-value store

```python
store = await Actor.open_key_value_store()
await store.set_value("STATE", {"cursor": "next"})
saved = await store.get_value("STATE", default_value={})
await store.delete_value("STATE")
output_url = await store.get_public_url("OUTPUT")
```

Key-value stores hold named JSON values, text, binary data, screenshots and files. Their methods are `get_value()`, `set_value()`, `delete_value()`, `record_exists()`, `iterate_keys()`, `list_keys()`, `get_public_url()`, `get_auto_saved_value()` and `persist_autosaved_values()`.

## Request queue

```python
from crawlee import Request

queue = await Actor.open_request_queue()

await queue.add_request("https://example.com/next")
await queue.add_request(
    Request.from_url("https://example.com/page-2", user_data={"depth": 1}),
    forefront=True,
)

request = await queue.fetch_next_request()
if request:
    try:
        await process_request(request)
        await queue.mark_request_as_handled(request)
    except Exception:
        await queue.reclaim_request(request)
        raise
```

`add_request()` takes a URL string or a `Request`, not a plain dict. Use `Request.from_url()` to attach `user_data`, a `unique_key` or other fields. `add_requests()` takes a sequence and writes it in batches. Set `forefront=True` to put requests at the head of the queue.

Other methods include `get_request()`, `is_empty()`, `is_finished()`, `get_handled_count()`, `get_total_count()`, `drop()` and `purge()`. When you fetch a request, you must mark it as handled or reclaim it, as the Crawlee queue contract requires.

## Request lists from input

`ScrapelyRequestList` builds a Crawlee request list from the standard `requestListSources` input format. Entries take either `url` or `requestsFromUrl`. With `requestsFromUrl`, the SDK fetches that resource and extracts every URL in it. Each entry can also set `method`, `payload`, `headers` and `userData`.

```python
from scrapely import Actor, ScrapelyRequestList

async with Actor:
    actor_input = await Actor.get_input() or {}
    request_list = await ScrapelyRequestList.open(
        request_list_sources_input=actor_input.get("requestListSources", []),
    )
```

`ScrapelyRequestList.open(*, name=None, request_list_sources_input=None, http_client=None)` uses Crawlee's `ImpitHttpClient` to fetch remote lists unless you pass another HTTP client.

## Storage backends

`Actor.init()` registers a `SmartScrapelyStorageClient` with Crawlee. Every open picks a backend:

- On the platform (`SCRAPELY_IS_AT_HOME=1`), it uses the Scrape.ly API through `ScrapelyStorageClient`.
- Off the platform with `force_cloud=True`, it also uses the API, and it raises `RuntimeError` when no token is set.
- Otherwise, it uses Crawlee file-system storage in `./storage`, or in `CRAWLEE_STORAGE_DIR`.

Crawlers that you create inside `async with Actor:` use the same backend, because the SDK registers it on Crawlee's service locator.

You rarely configure storage yourself. To point the SDK at specific credentials, pass them to `init()`:

```python
from scrapely import Actor, InitOptions

await Actor.init(InitOptions(storage={
    "api_url": api_url,
    "token": token,
    "run_id": run_id,
}))
```

The lower-level `set_api_storage()` has this signature:

```python
set_api_storage(api_config=None, *, cloud_storage_client=None, local_storage_client=None)
```

`api_config` is the same `{"api_url", "token", "run_id"}` mapping. With no arguments, `set_api_storage()` sets up local file-system storage. The function returns the `SmartScrapelyStorageClient` it registered.

The root package also exports `ApiStorageClient` (an alias of `ScrapelyStorageClient`), `FileSystemStorageClient`, `MemoryStorageClient` and Crawlee's `StorageClient` base class. You can pass any `StorageClient` as `InitOptions.storage`. For example, use `MemoryStorageClient()` in tests.

## What's next

- [Events & state](https://scrape.ly/docs/sdk-python-events.md) covers keeping state across migrations.
- [Storage](https://scrape.ly/docs/storage.md) explains when to use each storage type.
- The [storage API](https://scrape.ly/docs/api-datasets.md) lets you read results from outside a run.

---

- Previous: [Input & output](https://scrape.ly/docs/sdk-python-input-output.md)
- Next: [Events & state](https://scrape.ly/docs/sdk-python-events.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)