---
title: "Datasets · Scrape.ly Docs"
description: "Create datasets, push items, and export them as JSON, CSV, Excel, XML, or RSS."
url: https://scrape.ly/docs/api-datasets
markdown: https://scrape.ly/docs/api-datasets.md
---

# Datasets

> Create datasets, push items, and export them as JSON, CSV, Excel, XML, or RSS.

Datasets store tabular results (dataset items) from Actor runs. All endpoints are prefixed with `https://api.scrape.ly/v2`. Authenticate with `Authorization: Bearer $SCRAPELY_TOKEN`. [Get dataset items](#get-dataset-items) also accepts a [signed URL](#signed-items-urls) instead of a token.

Refer to a dataset by its UUID. [Get dataset](#get-dataset) and [Get dataset statistics](#get-dataset-statistics) also accept a named ID, `username~dataset-name`, where `username` is your own public username. All other routes, including the items routes, require the UUID.

Unnamed datasets (created without a `name`) expire after your account's unnamed-storage retention period: 2 days (48 hours) on the free plan, and 31, 60 or 90 days on the Starter, Scale or Business plan. On the free plan, the default storages of your latest 10 runs are kept until those runs are deleted. Named datasets do not expire. Creating a named dataset that you already own returns the existing dataset.

A run's default dataset is also reachable under the run: `/v2/actor-runs/{runId}/dataset` and its sub-paths (for example `/v2/actor-runs/{runId}/dataset/items`) are rewritten to `/v2/datasets/{defaultDatasetId}/...`. You must own the run.

## Endpoints

| Method | Path | Description |
| --- | --- | --- |
| `GET` | `/v2/datasets` | List datasets you own |
| `POST` | `/v2/datasets` | Create a dataset |
| `GET` | `/v2/datasets/{datasetId}` | Get dataset metadata |
| `PUT` | `/v2/datasets/{datasetId}` | Rename a dataset |
| `DELETE` | `/v2/datasets/{datasetId}` | Delete a dataset and its items |
| `GET` | `/v2/datasets/{datasetId}/items` | Get items in JSON, JSONL, CSV, HTML, XLSX, XML, RSS, or PDF |
| `HEAD` | `/v2/datasets/{datasetId}/items` | Get the item headers without a body |
| `POST` | `/v2/datasets/{datasetId}/items` | Append items |
| `GET` | `/v2/datasets/{datasetId}/statistics` | Per-field statistics |

## Errors and limits

Errors use the standard envelope `{ "error": { "type": "...", "message": "..." } }`.

| Status | `error.type` | When |
| --- | --- | --- |
| `400` | `invalid-request` | Malformed dataset ID, or the dataset is too large to sort or to compute statistics. |
| `400` | `schema-validation-error` | Pushed items fail the dataset's schema. |
| `402` | `insufficient-balance` | Creating a dataset or pushing items with no available credit. |
| `403` | `forbidden` | The dataset belongs to another account. |
| `403` | `insufficient-permissions` | A limited-permission Actor run token tried to use a dataset outside its scope. |
| `404` | `record-not-found` | The dataset does not exist. [Get dataset items](#get-dataset-items) also returns this for datasets you cannot read. Also returned when the dataset is deleted, or scheduled for deletion, while items are being pushed. |
| `409` | `name-taken` | Renaming to a name you already use for another dataset. |
| `413` | `request-too-large` | A JSON request body over 10 MB. |
| `429` | `rate-limit-exceeded` | Storage rate limit, or too many large item downloads running at once. |
| `429` | `storage-backpressure` | Storage is catching up. Nothing was written; retry after `Retry-After` (5 seconds). |
| `503` | `service-unavailable` | Storage is temporarily unavailable on create, push, delete, or read. Retry after `Retry-After` (5 seconds). A `503` on push may mean the items were stored. |

Dataset, key-value store, and request queue routes share a storage rate limit: by default 5,000 requests per minute for your account, plus a per-IP ceiling of 5,000 requests per minute. Responses carry `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset`; a `429` also carries `Retry-After`.

## Get list of datasets

```
GET /v2/datasets
```

Lists datasets you own. Unnamed datasets are omitted unless you pass `unnamed=1`.

| Query param | Type | Description |
| --- | --- | --- |
| `unnamed` | boolean | `1` or `true` includes unnamed datasets. |
| `name` | string | Case-insensitive substring match on the dataset name. |
| `sortBy` | string | `createdAt` (default), `modifiedAt`, or `name`. |
| `desc` | boolean | `1` or `true` reverses the sort order. Default is ascending. |
| `offset` | number | Number of datasets to skip (default `0`). |
| `limit` | number | Page size (default `1000`, max `1000`). |

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/datasets?unnamed=1&limit=20&desc=1"
```

```json [Response]
{
  "data": {
    "total": 2,
    "count": 2,
    "offset": 0,
    "limit": 20,
    "desc": true,
    "items": [
      {
        "id": "3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10",
        "name": "my-results",
        "userId": "uSrIdExAmPlE123",
        "username": "janedoe",
        "createdAt": "2026-08-14T15:00:00.000Z",
        "modifiedAt": "2026-08-14T15:02:00.000Z",
        "accessedAt": "2026-08-14T15:02:00.000Z",
        "itemCount": 1523,
        "cleanItemCount": 1498,
        "actId": null,
        "actRunId": null,
        "stats": { "storageBytes": 2845012 }
      }
    ]
  }
}
```

`actId` and `actRunId` are always `null` in list items. Use [Get dataset](#get-dataset) for the run link.

```api-test
{ "method": "GET", "path": "/v2/datasets?limit=5&unnamed=1" }
```

## Create dataset

```
POST /v2/datasets
```

Send `name` in the JSON body or as a query parameter. Omit it to create an unnamed dataset.

| Body / query field | Type | Description |
| --- | --- | --- |
| `name` | string | Optional dataset name. If you already own a dataset with this name, that dataset is returned with `200` instead of creating a new one. |

```bash
curl -X POST https://api.scrape.ly/v2/datasets \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "my-results"}'
```

A new dataset returns `201` with a `Location` header and the same body shape as [Get dataset](#get-dataset). `urlSigningSecretKey` is `null` in this response; read it with [Get dataset](#get-dataset).

## Get dataset

```
GET /v2/datasets/{datasetId}
```

Accepts a UUID or `username~dataset-name`.

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10
```

```json [Response]
{
  "data": {
    "id": "3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10",
    "name": "my-results",
    "userId": "uSrIdExAmPlE123",
    "username": "janedoe",
    "createdAt": "2026-08-14T15:00:00.000Z",
    "modifiedAt": "2026-08-14T15:02:00.000Z",
    "accessedAt": "2026-08-14T15:02:00.000Z",
    "itemCount": 1523,
    "cleanItemCount": 1498,
    "actId": null,
    "actRunId": null,
    "runId": null,
    "taskId": null,
    "fields": null,
    "schema": null,
    "consoleUrl": "https://my.scrape.ly/storage/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10",
    "itemsPublicUrl": "https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items",
    "urlSigningSecretKey": "9b1f0c2d4e...",
    "generalAccess": "RESTRICTED",
    "stats": {
      "readCount": 12,
      "writeCount": 3,
      "storageBytes": 2845012
    }
  }
}
```

- `urlSigningSecretKey` is the key for [signed items URLs](#signed-items-urls). Only this endpoint returns it.
- `cleanItemCount` counts items that have at least one field whose name does not start with `#`.
- `fields` and `schema` are always `null`, and `generalAccess` is always `RESTRICTED`.

```api-test
{ "method": "GET", "path": "/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10" }
```

## Update dataset

```
PUT /v2/datasets/{datasetId}
```

Renames the dataset. `{datasetId}` must be a UUID. `name` is the only field this endpoint changes.

| Body field | Type | Description |
| --- | --- | --- |
| `name` | string | New dataset name. An empty string makes the dataset unnamed. Returns `409 name-taken` if you already have another dataset with this name. |

```bash
curl -X PUT https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10 \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "renamed-results"}'
```

Returns the updated dataset object (with `urlSigningSecretKey` set to `null`).

## Delete dataset

```
DELETE /v2/datasets/{datasetId}
```

Deletes the dataset and all its items. `{datasetId}` must be a UUID. Returns `204 No Content`.

```bash
curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10
```

## Get dataset items

```
GET /v2/datasets/{datasetId}/items
HEAD /v2/datasets/{datasetId}/items
```

Returns items **directly**, not wrapped in the `data` envelope. `{datasetId}` must be a UUID. `HEAD` returns the same headers without a body.

Pagination is reported in response headers:

| Header | Value |
| --- | --- |
| `X-Apify-Pagination-Offset` | The `offset` you sent. |
| `X-Apify-Pagination-Limit` | The `limit` you sent, or `999999999999` when no limit was set. |
| `X-Apify-Pagination-Count` | Items in this response (for streamed responses, items read before filters such as `skipEmpty`). |
| `X-Apify-Pagination-Total` | Items in the dataset. |
| `X-Apify-Pagination-Desc` | `true` or `false`. |

### Query parameters

| Query param | Type | Description |
| --- | --- | --- |
| `format` | string | `json` (default), `jsonl`, `csv`, `html`, `xlsx`, `xml`, `rss`, or `pdf`. Unknown values return JSON. |
| `offset` | number | Items to skip (default `0`). |
| `limit` | number | Maximum items to return. Missing, `0`, or negative returns all remaining items. |
| `desc` | boolean | `1` or `true` returns items newest-first. With `sortBy`, reverses the field order instead. |
| `sortBy` | string | Sort by this top-level item field across the whole dataset. Empty values (`null`, missing, `""`) sort last. Limited to datasets of at most 10,000 items and 32 MB; larger datasets return `400`. |
| `fields` | string | Comma-separated top-level fields to keep. |
| `omit` | string | Comma-separated top-level fields to drop. |
| `unwind` | string | Comma-separated fields to unwind. An array field produces one item per element; an object field has its keys merged into the item. |
| `flatten` | string | Comma-separated object fields to flatten one level into `field.child` keys. |
| `clean` | boolean | `1` or `true` is the same as `skipHidden=1&skipEmpty=1`. |
| `skipHidden` | boolean | Drop fields whose names start with `#`. |
| `skipEmpty` | boolean | Skip items with no fields left. |
| `skipFailedPages` | boolean | Skip items that have an `errorInfo` property. |
| `simplified` | boolean | Legacy crawler format: keep only `url`, `pageFunctionResult`, and `errorInfo`, then unwind `pageFunctionResult`. |
| `attachment` | boolean | `1` or `true` sets `Content-Disposition: attachment` (`attachment; filename="dataset.xlsx"` for XLSX). PDF always sends `Content-Disposition` with a filename from the dataset name: `inline` by default, `attachment` with this option. |
| `delimiter` | string | CSV field delimiter (default `,`). |
| `skipHeaderRow` | boolean | CSV: omit the header row. |
| `bom` | boolean | CSV: prepend a UTF-8 byte order mark. On by default; `bom=0` or `bom=false` turns it off. |
| `escapeFormulas` | boolean | CSV: a text value or column name that starts with `=`, `+`, `-`, `@`, a tab, or a carriage return gets a leading `'`, so spreadsheets show it as text instead of running it as a formula. Numbers such as `-12.5` stay as they are. On by default; `escapeFormulas=0` or `escapeFormulas=false` writes values unchanged. |
| `xmlRoot` | string | XML root element name (default `items`). |
| `xmlRow` | string | XML element name for each item (default `item`). |
| `signature` | string | Signed-URL token. See [Signed items URLs](#signed-items-urls). |

Options are applied to each item in this order: `fields`, `omit`, `unwind`, `flatten`, `skipHidden`/`clean`, `skipEmpty`/`clean`, `skipFailedPages`, `simplified`. `offset` and `limit` select items before these options run, so filters can return fewer items than `limit`. There is no `view` parameter; it is ignored.

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items?format=json&offset=0&limit=100&fields=url,title,price&clean=1"
```

```json [Response]
[
  { "url": "https://example.com", "title": "Example", "price": 19.99 },
  { "url": "https://example.org", "title": "Sample", "price": 7.5 }
]
```

```api-test
{ "method": "GET", "path": "/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items?format=json&limit=10&clean=1" }
```

### Output formats

| `format` | `Content-Type` | Notes |
| --- | --- | --- |
| `json` | `application/json` | One JSON array. |
| `jsonl` | `application/x-jsonlines` | One JSON object per line. |
| `csv` | `text/csv` | Nested objects and arrays become `parent/child` and `parent/0` columns. Every value is quoted and each line ends with CRLF. |
| `html` | `text/html` | A single `<table>` with the same columns as CSV. |
| `xlsx` | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | One sheet named `Data` with the same columns as CSV. |
| `xml` | `application/xml` | `<items><item>...</item></items>`, renamed by `xmlRoot` and `xmlRow`. A `@` key becomes attributes and a `#` key becomes text content. |
| `rss` | `application/rss+xml` | `<rss><channel><item>...</item></channel></rss>`. |
| `pdf` | `application/pdf` | A table with one column per top-level field. See [PDF](#pdf). |

```bash
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" -o results.csv \
  "https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items?format=csv&bom=0&attachment=1"
```

### PDF

`format=pdf` applies `fields`, `omit`, `unwind`, `flatten`, `clean`, `offset`, `limit`, and `desc` like the other formats, then lays the items out as a report.

| Aspect | Behavior |
| --- | --- |
| Heading | Dataset name, dataset ID, run ID, row count, and generation time (UTC). Unnamed datasets are titled with the Actor when you own the dataset, otherwise `Dataset {datasetId}`. |
| Columns | Top-level fields in `fields` order, or in first-seen order without `fields`, plus a `#` column with each item's position in the dataset. |
| Orientation | Portrait up to 6 columns, landscape from 7. |
| Wide datasets | More than 16 columns render as one key/value block per item instead of a table. |
| Pages | Column header repeated on every page; `Page X of Y` footer. |
| Values | Nested arrays and objects are written inline (`a, b`, `key: value`). Long text wraps and is cut with `…`. |
| Item cap | At most 2,000 items are read. The PDF notes how many of the selected items it contains; page on with `offset`. `X-Apify-Pagination-Count` is the number read. |
| Row cap | At most 5,000 rows after `unwind`. |
| Text cap | About 3 million characters. Rendering stops with a note in the PDF. |
| Fonts | DejaVu Sans: Latin, Greek, Cyrillic, and common symbols. Characters outside it, such as CJK, show as empty boxes; use another format for those datasets. |
| Filename | `Content-Disposition: inline` (or `attachment` with `attachment=1`) and a filename from the dataset name, or `dataset-{datasetId}.pdf`. |

```bash
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" -o results.pdf \
  "https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items?format=pdf&fields=title,url,price&clean=1"
```

### Large downloads

Requests for up to 1,000 items are built in memory. Larger requests, including requests with no `limit`, are streamed. You can run up to 4 streamed downloads at once; beyond that the API returns `429 rate-limit-exceeded` with `Retry-After: 5`. If a streamed download fails partway, the connection is closed instead of ending normally, so a truncated file is detectable. A PDF is always finished in memory before it is sent, which is why it has its own [caps](#pdf).

### Signed items URLs

A signed URL lets someone download items without your API token. Append `signature=<token>`, where the token is minted with the dataset's `urlSigningSecretKey` from [Get dataset](#get-dataset). A signature:

- covers the HTTP method and the path `/v2/datasets/{datasetId}/items`, not the query string, so the recipient can change `format`, `limit`, and the other options;
- expires: there are no permanent signatures;
- is only checked when the request has no `Authorization` header.

The [JavaScript API client](https://scrape.ly/docs/client-js.md) mints these URLs:

```javascript
import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });
const url = await client.dataset('3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10').createItemsPublicUrl({
  clean: true,
  expiresInSecs: 24 * 3600,
});
// Append &format=csv (or any other option) to the returned URL as needed.
```

Without `expiresInSecs`, the client signs the URL for one hour.

The token format is described in [Signed URLs](https://scrape.ly/docs/api-key-value-stores.md#signed-urls) on the key-value stores page; datasets use the same scheme with an empty record key.

## Store items

```
POST /v2/datasets/{datasetId}/items
```

Appends items. `{datasetId}` must be a UUID. The body is a JSON array of objects, or a single JSON object (stored as one item). The request body is limited to 10 MB; split larger pushes into several requests.

Returns `201` with an empty JSON object `{}` and a `Location` header pointing at the items URL.

```bash
curl -X POST https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/items \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{"url": "https://example.com", "title": "Example"}]'
```

If the dataset was created by an Actor run whose dataset schema defines `fields` (JSON Schema draft-07), every item is validated first. If any item fails, nothing is stored and the response is `400`:

```json
{
  "error": {
    "type": "schema-validation-error",
    "message": "Schema validation failed",
    "data": {
      "invalidItems": [
        { "itemPosition": 0, "validationErrors": [{ "instancePath": "/price", "message": "must be number" }] }
      ]
    }
  }
}
```

A `429 storage-backpressure` means nothing was written, so retry after `Retry-After`. A `503 service-unavailable` may mean the items were stored, so a blind retry can store them twice. A `404 record-not-found` means the dataset was deleted, or scheduled for deletion, during the push. See [Errors and limits](#errors-and-limits).

## Get dataset statistics

```
GET /v2/datasets/{datasetId}/statistics
```

Returns statistics for every top-level field that appears on any item. Each field has `nullCount` (items where the field is missing or `null`). Fields with numeric values also have `min` and `max`. Accepts a UUID or `username~dataset-name`. Datasets with more than 10,000 items or more than 32 MB return `400 invalid-request`.

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/statistics
```

```json [Response]
{
  "data": {
    "fieldStatistics": {
      "url": { "nullCount": 0 },
      "title": { "nullCount": 2 },
      "price": { "nullCount": 1, "min": 7.5, "max": 19.99 }
    }
  }
}
```

```api-test
{ "method": "GET", "path": "/v2/datasets/3f6c1a2e-8b4d-4c7a-9e21-5d0b7f3a9c10/statistics" }
```

---

- Previous: [Actor tasks](https://scrape.ly/docs/api-tasks.md)
- Next: [Key-value stores](https://scrape.ly/docs/api-key-value-stores.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)