Storages

Each run has a default dataset, a default key-value store and a default request queue. You can also open other named or unnamed storages. The Actor.open_* methods return Crawlee's Dataset, KeyValueStore and RequestQueue classes. On the platform, these are backed by the Scrape.ly storage API. On your machine, they're backed by Crawlee's file-system storage. For the concepts behind the three storage types, see Storage.

Opening storages

dataset = await Actor.open_dataset()
store = await Actor.open_key_value_store()
queue = await Actor.open_request_queue()

All three methods share one signature:

await Actor.open_dataset(name_or_id=None, options=None, *, id=None, name=None, alias=None, force_cloud=False)

With no arguments, these methods open the run's default storage. Passing name_or_id together with id, name or alias raises ValueError.

await Actor.open_dataset("results")                 # name
await Actor.open_dataset(id="0f9c2b1e-...")         # id
await Actor.open_key_value_store(alias="cache")     # alias
await Actor.open_request_queue(name="frontier", force_cloud=True)

Dataset

dataset = await Actor.open_dataset()
await dataset.push_data({"url": url, "title": title})

async for item in dataset.iterate_items():
    print(item)

page = await dataset.get_data(offset=0, limit=100)

Datasets are append-only. Their methods are push_data(), get_data(), iterate_items(), list_items() and export_to(). Writing through dataset.push_data() doesn't charge pay-per-event events. Use Actor.push_data() for charged writes.

Key-value store

store = await Actor.open_key_value_store()
await store.set_value("STATE", {"cursor": "next"})
saved = await store.get_value("STATE", default_value={})
await store.delete_value("STATE")
output_url = await store.get_public_url("OUTPUT")

Key-value stores hold named JSON values, text, binary data, screenshots and files. Their methods are get_value(), set_value(), delete_value(), record_exists(), iterate_keys(), list_keys(), get_public_url(), get_auto_saved_value() and persist_autosaved_values().

Request queue

from crawlee import Request

queue = await Actor.open_request_queue()

await queue.add_request("https://example.com/next")
await queue.add_request(
    Request.from_url("https://example.com/page-2", user_data={"depth": 1}),
    forefront=True,
)

request = await queue.fetch_next_request()
if request:
    try:
        await process_request(request)
        await queue.mark_request_as_handled(request)
    except Exception:
        await queue.reclaim_request(request)
        raise

add_request() takes a URL string or a Request, not a plain dict. Use Request.from_url() to attach user_data, a unique_key or other fields. add_requests() takes a sequence and writes it in batches. Set forefront=True to put requests at the head of the queue.

Other methods include get_request(), is_empty(), is_finished(), get_handled_count(), get_total_count(), drop() and purge(). When you fetch a request, you must mark it as handled or reclaim it, as the Crawlee queue contract requires.

Request lists from input

ScrapelyRequestList builds a Crawlee request list from the standard requestListSources input format. Entries take either url or requestsFromUrl. With requestsFromUrl, the SDK fetches that resource and extracts every URL in it. Each entry can also set method, payload, headers and userData.

from scrapely import Actor, ScrapelyRequestList

async with Actor:
    actor_input = await Actor.get_input() or {}
    request_list = await ScrapelyRequestList.open(
        request_list_sources_input=actor_input.get("requestListSources", []),
    )

ScrapelyRequestList.open(*, name=None, request_list_sources_input=None, http_client=None) uses Crawlee's ImpitHttpClient to fetch remote lists unless you pass another HTTP client.

Storage backends

Actor.init() registers a SmartScrapelyStorageClient with Crawlee. Every open picks a backend:

• On the platform (SCRAPELY_IS_AT_HOME=1), it uses the Scrape.ly API through ScrapelyStorageClient.
• Off the platform with force_cloud=True, it also uses the API, and it raises RuntimeError when no token is set.
• Otherwise, it uses Crawlee file-system storage in ./storage, or in CRAWLEE_STORAGE_DIR.

Crawlers that you create inside async with Actor: use the same backend, because the SDK registers it on Crawlee's service locator.

You rarely configure storage yourself. To point the SDK at specific credentials, pass them to init():

from scrapely import Actor, InitOptions

await Actor.init(InitOptions(storage={
    "api_url": api_url,
    "token": token,
    "run_id": run_id,
}))

The lower-level set_api_storage() has this signature:

set_api_storage(api_config=None, *, cloud_storage_client=None, local_storage_client=None)

api_config is the same {"api_url", "token", "run_id"} mapping. With no arguments, set_api_storage() sets up local file-system storage. The function returns the SmartScrapelyStorageClient it registered.

The root package also exports ApiStorageClient (an alias of ScrapelyStorageClient), FileSystemStorageClient, MemoryStorageClient and Crawlee's StorageClient base class. You can pass any StorageClient as InitOptions.storage. For example, use MemoryStorageClient() in tests.

What's next

• Events & state covers keeping state across migrations.
• Storage explains when to use each storage type.
• The storage API lets you read results from outside a run.