Datasets

Datasets store tabular results (dataset items) from Actor runs. All endpoints are prefixed with https://api.scrape.ly/v2. Authenticate with Authorization: Bearer $SCRAPELY_TOKEN. Get dataset items also accepts a signed URL instead of a token.

Refer to a dataset by its UUID. Get dataset and Get dataset statistics also accept a named ID, username~dataset-name, where username is your own public username. All other routes, including the items routes, require the UUID.

Unnamed datasets (created without a name) expire after your account's unnamed-storage retention period: 2 days (48 hours) on the free plan, and 31, 60 or 90 days on the Starter, Scale or Business plan. On the free plan, the default storages of your latest 10 runs are kept until those runs are deleted. Named datasets do not expire. Creating a named dataset that you already own returns the existing dataset.

A run's default dataset is also reachable under the run: /v2/actor-runs/{runId}/dataset and its sub-paths (for example /v2/actor-runs/{runId}/dataset/items) are rewritten to /v2/datasets/{defaultDatasetId}/.... You must own the run.

Endpoints

Errors and limits

Errors use the standard envelope { "error": { "type": "...", "message": "..." } }.

Dataset, key-value store, and request queue routes share a storage rate limit: by default 5,000 requests per minute for your account, plus a per-IP ceiling of 5,000 requests per minute. Responses carry X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset; a 429 also carries Retry-After.

Get list of datasets

Lists datasets you own. Unnamed datasets are omitted unless you pass unnamed=1.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/datasets?unnamed=1&limit=20&desc=1"

actId and actRunId are always null in list items. Use Get dataset for the run link.

Create dataset

Send name in the JSON body or as a query parameter. Omit it to create an unnamed dataset.

curl -X POST https://api.scrape.ly/v2/datasets \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-results"}'

A new dataset returns 201 with a Location header and the same body shape as Get dataset. urlSigningSecretKey is null in this response; read it with Get dataset.

Get dataset

Accepts a UUID or username~dataset-name.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10

• urlSigningSecretKey is the key for signed items URLs. Only this endpoint returns it.
• cleanItemCount counts items that have at least one field whose name does not start with #.
• fields and schema are always null, and generalAccess is always RESTRICTED.

Update dataset

Renames the dataset. {datasetId} must be a UUID. name is the only field this endpoint changes.

curl -X PUT https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10 \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "renamed-results"}'

Returns the updated dataset object (with urlSigningSecretKey set to null).

Delete dataset

Deletes the dataset and all its items. {datasetId} must be a UUID. Returns 204 No Content.

curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10

Get dataset items

GET /v2/datasets/{datasetId}/items
HEAD /v2/datasets/{datasetId}/items

Returns items directly, not wrapped in the data envelope. {datasetId} must be a UUID. HEAD returns the same headers without a body.

Pagination is reported in response headers:

Query parameters

Options are applied to each item in this order: fields, omit, unwind, flatten, skipHidden/clean, skipEmpty/clean, skipFailedPages, simplified. offset and limit select items before these options run, so filters can return fewer items than limit. There is no view parameter; it is ignored.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items?format=json&offset=0&limit=100&fields=url,title,price&clean=1"

Output formats

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" -o results.csv \
  "https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items?format=csv&bom=0&attachment=1"

PDF

format=pdf applies fields, omit, unwind, flatten, clean, offset, limit, and desc like the other formats, then lays the items out as a report.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" -o results.pdf \
  "https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items?format=pdf&fields=title,url,price&clean=1"

Large downloads

Requests for up to 1,000 items are built in memory. Larger requests, including requests with no limit, are streamed. You can run up to 4 streamed downloads at once; beyond that the API returns 429 rate-limit-exceeded with Retry-After: 5. If a streamed download fails partway, the connection is closed instead of ending normally, so a truncated file is detectable. A PDF is always finished in memory before it is sent, which is why it has its own caps.

Signed items URLs

A signed URL lets someone download items without your API token. Append signature=<token>, where the token is minted with the dataset's urlSigningSecretKey from Get dataset. A signature:

• covers the HTTP method and the path /v2/datasets/{datasetId}/items, not the query string, so the recipient can change format, limit, and the other options;
• expires: there are no permanent signatures;
• is only checked when the request has no Authorization header.

The JavaScript API client mints these URLs:

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });
const url = await client.dataset('3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10').createItemsPublicUrl({
  clean: true,
  expiresInSecs: 24 * 3600,
});
// Append &format=csv (or any other option) to the returned URL as needed.

Without expiresInSecs, the client signs the URL for one hour.

The token format is described in Signed URLs on the key-value stores page; datasets use the same scheme with an empty record key.

Store items

Appends items. {datasetId} must be a UUID. The body is a JSON array of objects, or a single JSON object (stored as one item). The request body is limited to 10 MB; split larger pushes into several requests.

Returns 201 with an empty JSON object {} and a Location header pointing at the items URL.

curl -X POST https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://example.com", "title": "Example"}]'

If the dataset was created by an Actor run whose dataset schema defines fields (JSON Schema draft-07), every item is validated first. If any item fails, nothing is stored and the response is 400:

{
  "error": {
    "type": "schema-validation-error",
    "message": "Schema validation failed",
    "data": {
      "invalidItems": [
        { "itemPosition": 0, "validationErrors": [{ "instancePath": "/price", "message": "must be number" }] }
      ]
    }
  }
}

A 429 storage-backpressure means nothing was written, so retry after Retry-After. A 503 service-unavailable may mean the items were stored, so a blind retry can store them twice. A 404 record-not-found means the dataset was deleted, or scheduled for deletion, during the push. See Errors and limits.

Get dataset statistics

Returns statistics for every top-level field that appears on any item. Each field has nullCount (items where the field is missing or null). Fields with numeric values also have min and max. Accepts a UUID or username~dataset-name. Datasets with more than 10,000 items or more than 32 MB return 400 invalid-request.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/statistics