---
title: "Key-value stores · Scrape.ly Docs"
description: "Store and read records by key, list keys, download a ZIP, and share signed URLs."
url: https://scrape.ly/docs/api-key-value-stores
markdown: https://scrape.ly/docs/api-key-value-stores.md
---

# Key-value stores

> Store and read records by key, list keys, download a ZIP, and share signed URLs.

Key-value stores hold records addressed by key, typically JSON or files such as an Actor's `INPUT` and `OUTPUT`. All endpoints are prefixed with `https://api.scrape.ly/v2`. Authenticate with `Authorization: Bearer $SCRAPELY_TOKEN`. Listing keys, downloading the ZIP, and reading a record also accept a [signed URL](#signed-urls) instead of a token.

Refer to a store by its UUID or, on every route except updating and deleting the store itself, by named ID: `username~store-name`, where `username` is your own public username. Named IDs only work with a Bearer token.

Unnamed stores expire after your account's unnamed-storage retention period: 2 days (48 hours) on the free plan, and 31, 60 or 90 days on the Starter, Scale or Business plan. On the free plan, the default storages of your latest 10 runs are kept until those runs are deleted. Named stores do not expire. Creating a named store that you already own returns the existing store.

A run's default store is also reachable under the run: `/v2/actor-runs/{runId}/key-value-store` and its sub-paths (for example `/v2/actor-runs/{runId}/key-value-store/records/OUTPUT`) are rewritten to `/v2/key-value-stores/{defaultKeyValueStoreId}/...`. You must own the run.

## Endpoints

| Method | Path | Description |
| --- | --- | --- |
| `GET` | `/v2/key-value-stores` | List stores you own |
| `POST` | `/v2/key-value-stores` | Create a store |
| `GET` | `/v2/key-value-stores/{storeId}` | Get store metadata |
| `PUT` | `/v2/key-value-stores/{storeId}` | Update name, schema, or access setting |
| `DELETE` | `/v2/key-value-stores/{storeId}` | Delete a store and its records |
| `GET` | `/v2/key-value-stores/{storeId}/keys` | List keys (key-cursor pagination) |
| `GET` | `/v2/key-value-stores/{storeId}/records` | Download matching records as a ZIP |
| `GET` | `/v2/key-value-stores/{storeId}/records/{recordKey}` | Get a record |
| `HEAD` | `/v2/key-value-stores/{storeId}/records/{recordKey}` | Check that a record exists |
| `PUT` | `/v2/key-value-stores/{storeId}/records/{recordKey}` | Create or overwrite a record |
| `DELETE` | `/v2/key-value-stores/{storeId}/records/{recordKey}` | Delete a record |

## Errors and limits

Errors use the standard envelope `{ "error": { "type": "...", "message": "..." } }`.

| Status | `error.type` | When |
| --- | --- | --- |
| `400` | `invalid-request` | Malformed store ID, or a `collection` filter the store schema cannot satisfy. |
| `400` | `schema-validation-error` | A record does not match its collection in the store schema. |
| `402` | `insufficient-balance` | Creating a store or writing a record with no available credit. |
| `403` | `forbidden` | Getting, updating, deleting, or writing to a store owned by another account. |
| `403` | `insufficient-permissions` | A limited-permission Actor run token tried to use a store outside its scope. |
| `404` | `record-not-found` | The store or record does not exist. Reading keys, the ZIP, or a record of a store you cannot access also returns `404`. |
| `409` | `name-taken` | Renaming to a name you already use for another store. |
| `413` | `request-too-large` | A JSON body over 10 MB, or a binary or text body over 100 MB. |
| `429` | `rate-limit-exceeded` | Storage rate limit. |

Key-value store routes share the storage rate limit with datasets and request queues: by default 5,000 requests per minute for your account, plus a per-IP ceiling of 5,000 requests per minute. See [Datasets API](https://scrape.ly/docs/api-datasets.md#errors-and-limits).

## Get list of key-value stores

```
GET /v2/key-value-stores
```

Lists stores you own. Unnamed stores are omitted unless you pass `unnamed=1`.

| Query param | Type | Description |
| --- | --- | --- |
| `unnamed` | boolean | `1` or `true` includes unnamed stores. |
| `name` | string | Case-insensitive substring match on the store name. |
| `sortBy` | string | `createdAt` (default), `modifiedAt`, or `name`. |
| `desc` | boolean | `1` or `true` reverses the sort order. |
| `offset` | number | Number of stores to skip (default `0`). |
| `limit` | number | Page size (default `1000`, max `1000`). |

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/key-value-stores?unnamed=1&limit=20"
```

```json [Response]
{
  "data": {
    "total": 1,
    "count": 1,
    "offset": 0,
    "limit": 20,
    "desc": false,
    "items": [
      {
        "id": "b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f",
        "name": "my-store",
        "userId": "uSrIdExAmPlE123",
        "username": "janedoe",
        "createdAt": "2026-08-14T15:00:00.000Z",
        "modifiedAt": "2026-08-14T15:02:00.000Z",
        "accessedAt": "2026-08-14T15:02:00.000Z",
        "actId": null,
        "actRunId": null,
        "stats": { "storageBytes": 4096 }
      }
    ]
  }
}
```

```api-test
{ "method": "GET", "path": "/v2/key-value-stores?limit=5&unnamed=1" }
```

## Create key-value store

```
POST /v2/key-value-stores
```

| Body / query field | Type | Description |
| --- | --- | --- |
| `name` | string | Optional store name. If you already own a store with this name, that store is returned with `200`. |

```bash
curl -X POST https://api.scrape.ly/v2/key-value-stores \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-store"}'
```

A new store returns `201` with a `Location` header. The body matches [Get store](#get-store) and includes `urlSigningSecretKey`.

## Get store

```
GET /v2/key-value-stores/{storeId}
```

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f
```

```json [Response]
{
  "data": {
    "id": "b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f",
    "name": "my-store",
    "userId": "uSrIdExAmPlE123",
    "username": "janedoe",
    "createdAt": "2026-08-14T15:00:00.000Z",
    "modifiedAt": "2026-08-14T15:02:00.000Z",
    "accessedAt": "2026-08-14T15:02:00.000Z",
    "actId": null,
    "actRunId": null,
    "actName": null,
    "runId": null,
    "runName": null,
    "taskId": null,
    "taskName": null,
    "consoleUrl": "https://my.scrape.ly/storage/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f",
    "keysPublicUrl": "https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/keys",
    "urlSigningSecretKey": "a1b2c3d4e5f6...",
    "generalAccess": "RESTRICTED",
    "schema": null,
    "stats": {
      "readCount": 4,
      "writeCount": 2,
      "deleteCount": 0,
      "listCount": 1,
      "storageBytes": 4096
    }
  }
}
```

`urlSigningSecretKey` is the key for [signed URLs](#signed-urls).

```api-test
{ "method": "GET", "path": "/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f" }
```

## Update store

```
PUT /v2/key-value-stores/{storeId}
```

`{storeId}` must be a UUID. Send only the fields you want to change.

| Body field | Type | Description |
| --- | --- | --- |
| `name` | string | New store name. An empty string makes the store unnamed. Returns `409 name-taken` if you already have another store with this name. |
| `schema` | object or string | Store schema (a JSON object or a JSON string). Its `collections` drive the `collection` filter and record validation. |
| `generalAccess` | string | `RESTRICTED`, `ANYONE_WITH_ID_CAN_READ`, `ANYONE_WITH_NAME_CAN_READ`, or `FOLLOW_USER_SETTING`. Other values are ignored. |

> [!WARNING]
> `generalAccess` is stored and returned, but it does not grant access: reads without your token still need a [signed URL](#signed-urls).

A schema with two collections:

```json
{
  "collections": {
    "screenshots": { "keyPrefix": "screenshot-", "contentTypes": ["application/octet-stream"] },
    "summary": { "key": "SUMMARY", "contentTypes": ["application/json"], "jsonSchema": { "type": "object", "required": ["total"] } }
  }
}
```

```bash
curl -X PUT https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "renamed-store"}'
```

Returns the updated store object, including `urlSigningSecretKey`.

## Delete store

```
DELETE /v2/key-value-stores/{storeId}
```

Deletes the store and all its records. `{storeId}` must be a UUID. Returns `204 No Content`.

```bash
curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f
```

## Get list of keys

```
GET /v2/key-value-stores/{storeId}/keys
```

Lists record keys in ascending string order. Pagination is by key, not offset: pass the last key of the previous page as `exclusiveStartKey` to continue after it.

| Query param | Type | Description |
| --- | --- | --- |
| `limit` | number | Maximum keys to return (default `1000`, max `1000`). |
| `exclusiveStartKey` | string | Return keys after this key. It must be an existing key; an unknown key starts from the first key. |
| `prefix` | string | Only keys that start with this prefix. |
| `collection` | string | Only keys in this collection of the store schema (matched by `keyPrefix` or `key`). Returns `400` if the store has no schema or the collection is not defined. |
| `signature` | string | Signed-URL token. See [Signed URLs](#signed-urls). |

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/keys?limit=2&exclusiveStartKey=INPUT"
```

```json [Response]
{
  "data": {
    "count": 2,
    "limit": 2,
    "exclusiveStartKey": "INPUT",
    "isTruncated": true,
    "nextExclusiveStartKey": "screenshot-1",
    "items": [
      { "key": "OUTPUT", "size": 2048, "recordPublicUrl": "https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT" },
      { "key": "screenshot-1", "size": 10924, "recordPublicUrl": "https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/screenshot-1" }
    ]
  }
}
```

- When `isTruncated` is `true`, request the next page with `exclusiveStartKey` set to `nextExclusiveStartKey`. On the last page `nextExclusiveStartKey` is omitted.
- `exclusiveStartKey` is `""` when you did not send one.
- `size` is the stored size in bytes. Binary records are stored base64-encoded, so their `size` is about a third larger than the file.
- `recordPublicUrl` is not signed. Use [Signed URLs](#signed-urls) to share a record.

```api-test
{ "method": "GET", "path": "/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/keys?limit=100" }
```

## Download records

```
GET /v2/key-value-stores/{storeId}/records
```

Downloads matching records as a ZIP archive (`Content-Type: application/zip`, `Content-Disposition: attachment; filename="{storeId}.zip"`). Each record is a file named by its key.

| Query param | Type | Description |
| --- | --- | --- |
| `prefix` | string | Only records whose keys start with this prefix. |
| `collection` | string | Only records in this collection of the store schema (matched by `keyPrefix`). Ignored if the store has no schema; an undefined collection returns an empty ZIP. |
| `signature` | string | Signed-URL token. See [Signed URLs](#signed-urls). |

```bash
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" -o store.zip \
  "https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records?prefix=screenshot-"
```

```api-test
{ "method": "GET", "path": "/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records" }
```

## Get record

```
GET /v2/key-value-stores/{storeId}/records/{recordKey}
```

Returns the record value **directly**, not wrapped in `data`:

- JSON records are returned as JSON.
- Records stored with a `text/*` content type are returned with that content type.
- `application/octet-stream` records are returned as binary.

A missing key returns `404 record-not-found` with the message `Record was not found`.

| Query param | Type | Description |
| --- | --- | --- |
| `attachment` | boolean | `1` or `true` sends `Content-Disposition: attachment`. |
| `signature` | string | Signed-URL token for this record. See [Signed URLs](#signed-urls). |

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT
```

```json [Response]
{ "results": 42, "status": "ok" }
```

```api-test
{ "method": "GET", "path": "/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT" }
```

## Check record exists

```
HEAD /v2/key-value-stores/{storeId}/records/{recordKey}
```

Returns `200` if the record exists and `404` if it does not, or if you cannot access the store. There is no response body. This route needs a Bearer token; signatures are not accepted.

```bash
curl -I -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT
```

## Store record

```
PUT /v2/key-value-stores/{storeId}/records/{recordKey}
```

Creates or overwrites a record. The request body is the value, and `Content-Type` decides how it is read:

| `Content-Type` | Stored as | Body limit |
| --- | --- | --- |
| `application/json` (default when the header is missing) | JSON. The top level must be an object or array. | 10 MB |
| `text/*` (for example `text/plain`, `text/html`) | Text, returned later with the same content type. | 100 MB |
| `application/octet-stream` | Binary, returned later as `application/octet-stream`. | 100 MB |

> [!WARNING]
> Upload images, PDFs, and other files with `Content-Type: application/octet-stream`. Other content types, such as `image/png`, are not read, and the record is saved as an empty object `{}`.

If the store schema has a collection matching this key (by `key` or `keyPrefix`), its `contentTypes` list and, for JSON, its `jsonSchema` are enforced. Violations return `400 schema-validation-error`.

Returns `201` with an empty JSON object `{}` and a `Location` header.

```bash
curl -X PUT \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"results": 42, "status": "ok"}'

curl -X PUT \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/screenshot-1 \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @screenshot.png
```

## Delete record

```
DELETE /v2/key-value-stores/{storeId}/records/{recordKey}
```

Deletes a single record. Returns `204 No Content`, including when the key does not exist.

```bash
curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT
```

## Signed URLs

A signed URL lets someone read from a store, or download dataset items, without your API token. Add `signature=<token>` to the URL. Signatures work on:

| Route | Bound to |
| --- | --- |
| `GET /v2/key-value-stores/{storeId}/keys` | That path |
| `GET /v2/key-value-stores/{storeId}/records` (ZIP) | That path |
| `GET /v2/key-value-stores/{storeId}/records/{recordKey}` | That path and record key |
| `GET` or `HEAD /v2/datasets/{datasetId}/items` | That method and path |

A signature for one route does not work on another, and the query string is not signed, so the recipient can change options such as `prefix` or `format`. Every signature expires; there are no permanent signatures. Signed URLs must use the store or dataset UUID, not a named ID. An invalid or expired signature without a token returns `404 record-not-found`.

The [JavaScript API client](https://scrape.ly/docs/client-js.md) mints signed URLs with `getRecordPublicUrl(key)` and `createKeysPublicUrl()` on a store, and `createItemsPublicUrl()` on a dataset. Each accepts `expiresInSecs` (default one hour):

```javascript
import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });
const store = client.keyValueStore('b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f');
const recordUrl = await store.getRecordPublicUrl('OUTPUT');
const keysUrl = await store.createKeysPublicUrl({ expiresInSecs: 24 * 3600 });
```

### Signature format

To mint a token yourself, for example for the ZIP download, read `urlSigningSecretKey` from [Get store](#get-store) or [Get dataset](https://scrape.ly/docs/api-datasets.md#get-dataset), then:

1. Pick `expiresAt`, a Unix time in milliseconds in the future.
2. Build the message `1.{expiresAt}.{storageId}.{METHOD}.{path}.{recordKey}`, where `path` is the URL path without the query string (for example `/v2/key-value-stores/{storeId}/records`) and `recordKey` is the record key for single-record reads and empty otherwise.
3. Compute HMAC-SHA256 of the message with `urlSigningSecretKey`, take the first 30 hex characters, and encode that number in base62 (`0-9`, `a-z`, `A-Z`).
4. The token is the base64url encoding of `1.{expiresAt}.{base62Hmac}`.

```javascript
import { createHmac } from 'node:crypto';

const BASE62 = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';

function hmacBase62(key, message) {
  let n = BigInt('0x' + createHmac('sha256', key).update(message).digest('hex').slice(0, 30));
  let out = '';
  while (n > 0n) { out = BASE62[Number(n % 62n)] + out; n /= 62n; }
  return out || '0';
}

function signStorageUrl({ secretKey, storageId, method, path, recordKey = '', expiresInSecs = 3600 }) {
  const expiresAt = Date.now() + expiresInSecs * 1000;
  const message = `1.${expiresAt}.${storageId}.${method}.${path}.${recordKey}`;
  return Buffer.from(`1.${expiresAt}.${hmacBase62(secretKey, message)}`).toString('base64url');
}

const storeId = 'b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f';
const path = `/v2/key-value-stores/${storeId}/records`;
const signature = signStorageUrl({ secretKey: process.env.KVS_SIGNING_KEY, storageId: storeId, method: 'GET', path });
console.log(`https://api.scrape.ly${path}?signature=${signature}`);
```

---

- Previous: [Datasets](https://scrape.ly/docs/api-datasets.md)
- Next: [Request queues](https://scrape.ly/docs/api-request-queues.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)