Key-value stores

Key-value stores hold records addressed by key, typically JSON or files such as an Actor's INPUT and OUTPUT. All endpoints are prefixed with https://api.scrape.ly/v2. Authenticate with Authorization: Bearer $SCRAPELY_TOKEN. Listing keys, downloading the ZIP, and reading a record also accept a signed URL instead of a token.

Refer to a store by its UUID or, on every route except updating and deleting the store itself, by named ID: username~store-name, where username is your own public username. Named IDs only work with a Bearer token.

Unnamed stores expire after your account's unnamed-storage retention period: 2 days (48 hours) on the free plan, and 31, 60 or 90 days on the Starter, Scale or Business plan. On the free plan, the default storages of your latest 10 runs are kept until those runs are deleted. Named stores do not expire. Creating a named store that you already own returns the existing store.

A run's default store is also reachable under the run: /v2/actor-runs/{runId}/key-value-store and its sub-paths (for example /v2/actor-runs/{runId}/key-value-store/records/OUTPUT) are rewritten to /v2/key-value-stores/{defaultKeyValueStoreId}/.... You must own the run.

Endpoints

Errors and limits

Errors use the standard envelope { "error": { "type": "...", "message": "..." } }.

Key-value store routes share the storage rate limit with datasets and request queues: by default 5,000 requests per minute for your account, plus a per-IP ceiling of 5,000 requests per minute. See Datasets API.

Get list of key-value stores

Lists stores you own. Unnamed stores are omitted unless you pass unnamed=1.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/key-value-stores?unnamed=1&limit=20"

Create key-value store

curl -X POST https://api.scrape.ly/v2/key-value-stores \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-store"}'

A new store returns 201 with a Location header. The body matches Get store and includes urlSigningSecretKey.

Get store

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f

urlSigningSecretKey is the key for signed URLs.

Update store

{storeId} must be a UUID. Send only the fields you want to change.

A schema with two collections:

{
  "collections": {
    "screenshots": { "keyPrefix": "screenshot-", "contentTypes": ["application/octet-stream"] },
    "summary": { "key": "SUMMARY", "contentTypes": ["application/json"], "jsonSchema": { "type": "object", "required": ["total"] } }
  }
}

curl -X PUT https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "renamed-store"}'

Returns the updated store object, including urlSigningSecretKey.

Delete store

Deletes the store and all its records. {storeId} must be a UUID. Returns 204 No Content.

curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f

Get list of keys

Lists record keys in ascending string order. Pagination is by key, not offset: pass the last key of the previous page as exclusiveStartKey to continue after it.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/keys?limit=2&exclusiveStartKey=INPUT"

• When isTruncated is true, request the next page with exclusiveStartKey set to nextExclusiveStartKey. On the last page nextExclusiveStartKey is omitted.
• exclusiveStartKey is "" when you did not send one.
• size is the stored size in bytes. Binary records are stored base64-encoded, so their size is about a third larger than the file.
• recordPublicUrl is not signed. Use Signed URLs to share a record.

Download records

Downloads matching records as a ZIP archive (Content-Type: application/zip, Content-Disposition: attachment; filename="{storeId}.zip"). Each record is a file named by its key.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" -o store.zip \
  "https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records?prefix=screenshot-"

Get record

Returns the record value directly, not wrapped in data:

• JSON records are returned as JSON.
• Records stored with a text/* content type are returned with that content type.
• application/octet-stream records are returned as binary.

A missing key returns 404 record-not-found with the message Record was not found.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT

Check record exists

HEAD /v2/key-value-stores/{storeId}/records/{recordKey}

Returns 200 if the record exists and 404 if it does not, or if you cannot access the store. There is no response body. This route needs a Bearer token; signatures are not accepted.

curl -I -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT

Store record

Creates or overwrites a record. The request body is the value, and Content-Type decides how it is read:

If the store schema has a collection matching this key (by key or keyPrefix), its contentTypes list and, for JSON, its jsonSchema are enforced. Violations return 400 schema-validation-error.

Returns 201 with an empty JSON object {} and a Location header.

curl -X PUT \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"results": 42, "status": "ok"}'

curl -X PUT \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/screenshot-1 \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @screenshot.png

Delete record

Deletes a single record. Returns 204 No Content, including when the key does not exist.

curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/key-value-stores/b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f/records/OUTPUT

Signed URLs

A signed URL lets someone read from a store, or download dataset items, without your API token. Add signature=<token> to the URL. Signatures work on:

A signature for one route does not work on another, and the query string is not signed, so the recipient can change options such as prefix or format. Every signature expires; there are no permanent signatures. Signed URLs must use the store or dataset UUID, not a named ID. An invalid or expired signature without a token returns 404 record-not-found.

The JavaScript API client mints signed URLs with getRecordPublicUrl(key) and createKeysPublicUrl() on a store, and createItemsPublicUrl() on a dataset. Each accepts expiresInSecs (default one hour):

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });
const store = client.keyValueStore('b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f');
const recordUrl = await store.getRecordPublicUrl('OUTPUT');
const keysUrl = await store.createKeysPublicUrl({ expiresInSecs: 24 * 3600 });

Signature format

To mint a token yourself, for example for the ZIP download, read urlSigningSecretKey from Get store or Get dataset, then:

1. Pick expiresAt, a Unix time in milliseconds in the future.
2. Build the message 1.{expiresAt}.{storageId}.{METHOD}.{path}.{recordKey}, where path is the URL path without the query string (for example /v2/key-value-stores/{storeId}/records) and recordKey is the record key for single-record reads and empty otherwise.
3. Compute HMAC-SHA256 of the message with urlSigningSecretKey, take the first 30 hex characters, and encode that number in base62 (0-9, a-z, A-Z).
4. The token is the base64url encoding of 1.{expiresAt}.{base62Hmac}.

import { createHmac } from 'node:crypto';

const BASE62 = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';

function hmacBase62(key, message) {
  let n = BigInt('0x' + createHmac('sha256', key).update(message).digest('hex').slice(0, 30));
  let out = '';
  while (n > 0n) { out = BASE62[Number(n % 62n)] + out; n /= 62n; }
  return out || '0';
}

function signStorageUrl({ secretKey, storageId, method, path, recordKey = '', expiresInSecs = 3600 }) {
  const expiresAt = Date.now() + expiresInSecs * 1000;
  const message = `1.${expiresAt}.${storageId}.${method}.${path}.${recordKey}`;
  return Buffer.from(`1.${expiresAt}.${hmacBase62(secretKey, message)}`).toString('base64url');
}

const storeId = 'b2d7e4a1-5c3f-4e8a-9f60-1a2b3c4d5e6f';
const path = `/v2/key-value-stores/${storeId}/records`;
const signature = signStorageUrl({ secretKey: process.env.KVS_SIGNING_KEY, storageId: storeId, method: 'GET', path });
console.log(`https://api.scrape.ly${path}?signature=${signature}`);