---
title: "Storage management · Scrape.ly Docs"
description: "Read and write datasets and key-value stores, and inspect request queues."
url: https://scrape.ly/docs/cli-storage
markdown: https://scrape.ly/docs/cli-storage.md
---

# Storage management

> Read and write datasets and key-value stores, and inspect request queues.

The `scrapely datasets` and `scrapely key-value-stores` commands read and write storages on the platform through the API. They work from any folder once you are logged in. For the storages of the run your code is in, use the [Actor runtime](https://scrape.ly/docs/cli-actor-runtime.md) commands instead.

Every storage argument takes a storage ID or the name of one of your named storages. The CLI looks names up in your storage list. Delete commands ask for confirmation unless you pass `-y, --yes`.

## Datasets

A dataset is an append-only list of JSON items. Each run has a default dataset for its results.

| Command | Purpose |
| --- | --- |
| `datasets ls` | List datasets |
| `datasets create [name]` | Create a dataset, or get the one with that name |
| `datasets info <datasetId>` | Show a dataset's ID, name, item count, and creation time |
| `datasets get-items <datasetId>` | Print items |
| `datasets push-items <datasetId> [item]` | Append items |
| `datasets rename <datasetIdOrName> [newName]` | Rename a dataset or remove its name |
| `datasets rm <datasetIdOrName>` | Delete a dataset |

### List datasets

```bash
scrapely datasets ls
scrapely datasets ls --unnamed --desc --limit 20
```

Each line shows the ID, the name, and the number of items.

| Option | Description |
| --- | --- |
| `--limit <number>` | Maximum number to return |
| `--offset <number>` | Number to skip |
| `--unnamed` | Only unnamed datasets, such as the default datasets of runs |
| `--desc` | Newest first |
| `--json` | Print the API response |

### Create a dataset

```bash
scrapely datasets create my-results
```

Without a name, `create` makes an unnamed dataset. With a name, it returns the existing dataset of that name if there is one. `--json` prints the dataset object.

### Get items

```bash
scrapely datasets get-items <datasetId>
scrapely datasets get-items <datasetId> --format csv > results.csv
scrapely datasets get-items <datasetId> --format jsonl | jq -r '.url'
```

`get-items` writes the items to stdout. Redirect it to save a file.

| Option | Description |
| --- | --- |
| `--format <format>` | `json` (default), `jsonl`, `csv`, `html`, `rss`, `xml`, or `xlsx` |
| `--limit <number>` | Maximum number of items |
| `--offset <number>` | Number of items to skip |

With `json`, the output is a JSON array of the items. The other formats are passed through as the API returns them.

> [!WARNING]
> `xlsx` is a binary format, and the CLI prints it as text, which corrupts the file. Download Excel files from the dashboard or with `GET /v2/datasets/<datasetId>/items?format=xlsx` instead.

To find a run's dataset, run `scrapely runs info <runId>` and use the `Dataset ID`.

### Push items

```bash
scrapely datasets push-items <datasetId> '{"url":"https://example.com","title":"Example"}'
scrapely datasets push-items <datasetId> '[{"url":"https://a.example"},{"url":"https://b.example"}]'
cat rows.json | scrapely datasets push-items <datasetId>
```

The item is a JSON object or an array of objects. Without the argument, the command reads it from stdin. Invalid JSON is rejected before anything is sent.

### Rename or delete a dataset

```bash
scrapely datasets rename <datasetIdOrName> new-name
scrapely datasets rename <datasetIdOrName> --unname
scrapely datasets rm <datasetIdOrName> -y
```

`--unname` removes the name instead of setting a new one. `rm` deletes the dataset and its items.

## Key-value stores

A key-value store holds records under string keys: JSON, text, HTML, images, or any other file. Each run has a default store, which holds its `INPUT` record and often an `OUTPUT` record.

`kvs` is an alias for `key-value-stores`:

```bash
scrapely key-value-stores ls
scrapely kvs ls
```

| Command | Purpose |
| --- | --- |
| `kvs ls` | List stores |
| `kvs create [name]` | Create a store, or get the one with that name |
| `kvs info <storeId>` | Show a store's ID, name, and creation time |
| `kvs keys <storeId>` | List the keys in a store |
| `kvs get-value <storeId> <key>` | Print a record |
| `kvs set-value <storeId> <key> [value]` | Write a record, or delete it when the value is empty |
| `kvs delete-value <storeId> <key>` | Delete a record |
| `kvs rename <storeIdOrName> [newName]` | Rename a store or remove its name |
| `kvs rm <storeIdOrName>` | Delete a store |

### List stores

```bash
scrapely kvs ls
```

`kvs ls` takes the same `--limit`, `--offset`, `--unnamed`, `--desc`, and `--json` options as `datasets ls`.

### Create a store

```bash
scrapely kvs create my-state
```

Without a name, `create` makes an unnamed store. With a name, it returns the existing store of that name if there is one. `--json` prints the store object.

### List keys

```bash
scrapely kvs keys <storeId>
scrapely kvs keys <storeId> --limit 100 --exclusive-start-key last-key-from-previous-page
```

Each line shows the key and the record size.

| Option | Description |
| --- | --- |
| `--limit <number>` | Maximum number of keys |
| `--exclusive-start-key <key>` | Start after this key |
| `--json` | Print the API response |

Keys are paged by key, not by offset. To get the next page, pass the last key you received as `--exclusive-start-key`.

### Read a record

```bash
scrapely kvs get-value <storeId> OUTPUT
scrapely kvs get-value <storeId> screenshot.png > screenshot.png
scrapely kvs get-value <storeId> screenshot.png --only-content-type
```

`get-value` writes the record to stdout as stored. Binary records are written byte for byte, so redirect them to a file. JSON is pretty-printed. `--only-content-type` prints only the record's content type. A missing key is an error.

### Write a record

```bash
scrapely kvs set-value <storeId> STATE '{"cursor":"next"}'
scrapely kvs set-value <storeId> page.html '<h1>Hello</h1>' -c text/html
```

| Option | Description |
| --- | --- |
| `-c, --content-type <type>` | Content type of the record. Default `application/json`. |

With the default content type, the value must be valid JSON. The value is always the command-line argument: `set-value` doesn't read stdin or files, so it is meant for text records. Upload binary files with the [key-value store API](https://scrape.ly/docs/api-key-value-stores.md).

An empty or missing value deletes the record without asking:

```bash
scrapely kvs set-value <storeId> TEMPORARY
```

### Delete a record

```bash
scrapely kvs delete-value <storeId> <key>
scrapely kvs delete-value <storeId> <key> -y
```

### Rename or delete a store

```bash
scrapely kvs rename <storeIdOrName> new-name
scrapely kvs rename <storeIdOrName> --unname
scrapely kvs rm <storeIdOrName> -y
```

## Request queues

`scrapely request-queues` exists but has no subcommands yet. To work with request queues from the terminal, call the [request queue API](https://scrape.ly/docs/api-request-queues.md) with `scrapely api`:

```bash
scrapely api GET /v2/request-queues
scrapely api GET /v2/request-queues/<queueId>/requests -p '{"limit":10}'
```

## What's next

- [Storage](https://scrape.ly/docs/storage.md): when to use each storage type.
- [Results](https://scrape.ly/docs/results.md): find, export, and share run results.
- [Actor runtime](https://scrape.ly/docs/cli-actor-runtime.md): read and write the current run's storages from inside a container.
- [Datasets API](https://scrape.ly/docs/api-datasets.md) and [key-value stores API](https://scrape.ly/docs/api-key-value-stores.md): the endpoints behind these commands.

---

- Previous: [Projects & running](https://scrape.ly/docs/cli-projects.md)
- Next: [Actor management](https://scrape.ly/docs/cli-actors.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)