---
title: "Request queues · Scrape.ly Docs"
description: "Add, lock, and update crawl requests so several workers can share one queue."
url: https://scrape.ly/docs/api-request-queues
markdown: https://scrape.ly/docs/api-request-queues.md
---

# Request queues

> Add, lock, and update crawl requests so several workers can share one queue.

Request queues hold a dynamic list of URLs to crawl, with locking so that several workers do not fetch the same request. All endpoints are prefixed with `https://api.scrape.ly/v2`. Authenticate with `Authorization: Bearer $SCRAPELY_TOKEN`.

Refer to a queue by its UUID or, on every route except updating and deleting the queue itself, by named ID: `username~queue-name`, where `username` is your own public username.

Unnamed queues expire after your account's unnamed-storage retention period: 2 days (48 hours) on the free plan, and 31, 60 or 90 days on the Starter, Scale or Business plan. On the free plan, the default storages of your latest 10 runs are kept until those runs are deleted. Named queues do not expire. Creating a named queue that you already own returns the existing queue.

A run's default queue is also reachable under the run: `/v2/actor-runs/{runId}/request-queue` and its sub-paths (for example `/v2/actor-runs/{runId}/request-queue/head`) are rewritten to `/v2/request-queues/{defaultRequestQueueId}/...`. You must own the run.

## Endpoints

| Method | Path | Description |
| --- | --- | --- |
| `GET` | `/v2/request-queues` | List queues you own |
| `POST` | `/v2/request-queues` | Create a queue |
| `GET` | `/v2/request-queues/{queueId}` | Get queue metadata |
| `PUT` | `/v2/request-queues/{queueId}` | Rename a queue |
| `DELETE` | `/v2/request-queues/{queueId}` | Delete a queue and its requests |
| `GET` | `/v2/request-queues/{queueId}/requests` | List requests |
| `POST` | `/v2/request-queues/{queueId}/requests` | Add a single request |
| `POST` | `/v2/request-queues/{queueId}/requests/batch` | Add up to 25 requests |
| `DELETE` | `/v2/request-queues/{queueId}/requests/batch` | Delete up to 25 requests |
| `GET` | `/v2/request-queues/{queueId}/requests/{requestId}` | Get a request |
| `PUT` | `/v2/request-queues/{queueId}/requests/{requestId}` | Update a request |
| `DELETE` | `/v2/request-queues/{queueId}/requests/{requestId}` | Delete a request |
| `GET` | `/v2/request-queues/{queueId}/head` | Peek at pending requests |
| `POST` | `/v2/request-queues/{queueId}/head/lock` | Fetch and lock pending requests |
| `PUT` | `/v2/request-queues/{queueId}/requests/{requestId}/lock` | Prolong a lock |
| `DELETE` | `/v2/request-queues/{queueId}/requests/{requestId}/lock` | Release a lock |
| `POST` | `/v2/request-queues/{queueId}/requests/unlock` | Release all locks held by a `clientKey` |

## Errors and limits

Errors use the standard envelope `{ "error": { "type": "...", "message": "..." } }`.

| Status | `error.type` | When |
| --- | --- | --- |
| `400` | `invalid-request` | Malformed queue ID, or `filter` combined with `exclusiveStartId`. |
| `400` | `invalid-request-id` | A request ID in the path or body that isn't 1 to 128 letters, digits, `-`, `_`, or `~`, or a body `id` on `PUT` that differs from the path. See [Request IDs](#request-ids). |
| `400` | `invalid-input` | Missing `uniqueKey` or `url`, a batch over 25 items, a missing `lockSecs`, or an invalid or missing `clientKey`. |
| `402` | `insufficient-balance` | Creating a queue or adding requests with no available credit. |
| `403` | `forbidden` | The queue belongs to another account (some routes return `404` instead), or the lock belongs to another client. |
| `403` | `insufficient-permissions` | A limited-permission Actor run token tried to use a queue outside its scope. |
| `404` | `record-not-found` | The queue or request does not exist. |
| `409` | `name-taken` | Renaming to a name you already use for another queue. |
| `409` | `invalid-state` | Prolonging a lock on a request that is handled, not locked, or whose lock expired. |
| `429` | `rate-limit-exceeded` | Storage rate limit. |

Request queue routes share the storage rate limit with datasets and key-value stores: by default 5,000 requests per minute for your account, plus a per-IP ceiling of 5,000 requests per minute. See [Datasets API](https://scrape.ly/docs/api-datasets.md#errors-and-limits).

### Request IDs

A request ID is 1 to 128 characters of letters, digits, `-`, `_`, and `~`. Dots, slashes, `%`, and other characters are refused, so an ID is always a single URL path segment. IDs the API generates from `uniqueKey` are 15 letters and digits, the same IDs Crawlee and the SDKs compute.

- Routes with a `{requestId}` in the path return `400 invalid-request-id` for any other ID, before looking anything up.
- **Add request** with an `id` outside these rules returns `400 invalid-request-id`. Without an `id` (or with `""` or `null`), the ID is generated.
- In the batch routes, an item with an invalid `id` is returned in `unprocessedRequests` with an `error` object; the rest of the batch is processed.

### Client keys and lock ownership

`clientKey` (1 to 32 characters) identifies a worker. When a list or head call comes from a different `clientKey` than earlier calls, or from none, the queue's `hadMultipleClients` becomes `true`.

A lock belongs to the `clientKey` that took it and, inside an Actor run, to that run as well. Prolonging or releasing a lock needs the same `clientKey` from the same run; a lock taken inside a run cannot be released with your personal token from outside it.

## Get list of request queues

```
GET /v2/request-queues
```

Lists queues you own. Unnamed queues are omitted unless you pass `unnamed=1`.

| Query param | Type | Description |
| --- | --- | --- |
| `unnamed` | boolean | `1` or `true` includes unnamed queues. |
| `name` | string | Case-insensitive substring match on the queue name. |
| `sortBy` | string | `createdAt` (default), `modifiedAt`, or `name`. |
| `desc` | boolean | `1` or `true` reverses the sort order. |
| `offset` | number | Number of queues to skip (default `0`). |
| `limit` | number | Page size (default `1000`, max `1000`). |

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/request-queues?unnamed=1&limit=20"
```

```json [Response]
{
  "data": {
    "total": 1,
    "count": 1,
    "offset": 0,
    "limit": 20,
    "desc": false,
    "items": [
      {
        "id": "7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e",
        "name": "crawl-queue",
        "userId": "uSrIdExAmPlE123",
        "username": null,
        "createdAt": "2026-08-14T15:00:00.000Z",
        "modifiedAt": "2026-08-14T15:02:00.000Z",
        "accessedAt": "2026-08-14T15:02:00.000Z",
        "totalRequestCount": 200,
        "handledRequestCount": 50,
        "pendingRequestCount": 150,
        "actId": null,
        "actRunId": null,
        "hadMultipleClients": false,
        "size": "12.4 KB"
      }
    ]
  }
}
```

`size` is a human-readable string. For the size in bytes, use `stats.storageBytes` from [Get request queue](#get-request-queue).

```api-test
{ "method": "GET", "path": "/v2/request-queues?limit=5&unnamed=1" }
```

## Create request queue

```
POST /v2/request-queues
```

| Body / query field | Type | Description |
| --- | --- | --- |
| `name` | string | Optional queue name. If you already own a queue with this name, that queue is returned with `200`. |

```bash
curl -X POST https://api.scrape.ly/v2/request-queues \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "crawl-queue"}'
```

A new queue returns `201` with a `Location` header. The body matches [Get request queue](#get-request-queue).

## Get request queue

```
GET /v2/request-queues/{queueId}
```

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e
```

```json [Response]
{
  "data": {
    "id": "7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e",
    "name": "crawl-queue",
    "username": null,
    "userId": "uSrIdExAmPlE123",
    "createdAt": "2026-08-14T15:00:00.000Z",
    "modifiedAt": "2026-08-14T15:02:00.000Z",
    "accessedAt": "2026-08-14T15:02:00.000Z",
    "totalRequestCount": 200,
    "handledRequestCount": 50,
    "pendingRequestCount": 150,
    "hadMultipleClients": false,
    "consoleUrl": "https://my.scrape.ly/storage/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e",
    "runId": null,
    "runName": null,
    "actId": null,
    "actRunId": null,
    "actName": null,
    "taskId": null,
    "taskName": null,
    "size": "12.4 KB",
    "stats": {
      "readCount": 8,
      "writeCount": 200,
      "deleteCount": 0,
      "headItemReadCount": 4,
      "storageBytes": 12697
    },
    "generalAccess": "RESTRICTED"
  }
}
```

```api-test
{ "method": "GET", "path": "/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e" }
```

## Update request queue

```
PUT /v2/request-queues/{queueId}
```

Renames the queue. `{queueId}` must be a UUID.

| Body field | Type | Description |
| --- | --- | --- |
| `name` | string | New queue name. Returns `409 name-taken` if you already have another queue with this name. |

```bash
curl -X PUT https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "renamed-queue"}'
```

## Delete request queue

```
DELETE /v2/request-queues/{queueId}
```

Deletes the queue and all its requests. `{queueId}` must be a UUID. Returns `204 No Content`.

```bash
curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e
```

## List requests

```
GET /v2/request-queues/{queueId}/requests
```

Lists requests in the queue, ordered by request ID (not by the order they were added). Filter with `filter`; there is no `status` parameter. Paginate with `cursor`.

| Query param | Type | Description |
| --- | --- | --- |
| `limit` | number | Page size (default `100`, max `10000`). |
| `cursor` | string | Opaque cursor from `nextCursor` on the previous page. |
| `exclusiveStartId` | string | Deprecated: start after this request ID. Cannot be combined with `filter`. |
| `clientKey` | string | Worker identity (1 to 32 characters). |
| `filter` | string or array | `locked` or `pending`. Pass both as `filter=locked&filter=pending` or as a JSON array, `filter=["locked","pending"]`. |

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests?filter=pending&limit=50&clientKey=worker-1"
```

```json [Response]
{
  "data": {
    "items": [
      {
        "id": "Xb3kQ9pLm2RtV7w",
        "retryCount": 0,
        "uniqueKey": "https://example.com",
        "url": "https://example.com",
        "method": "GET",
        "loadedUrl": null,
        "payload": null,
        "noRetry": false,
        "errorMessages": null,
        "headers": null,
        "userData": null,
        "handledAt": null
      }
    ],
    "count": 1,
    "limit": 50,
    "nextCursor": "eyJyZXF1ZXN0SWQiOiJYYjNrUTlwTG0yUnRWN3cifQ"
  }
}
```

`nextCursor` is present only when there are more requests. The response also echoes `cursor` and `exclusiveStartId` when you sent them.

```api-test
{ "method": "GET", "path": "/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests?filter=pending&limit=20" }
```

## Add request

```
POST /v2/request-queues/{queueId}/requests
```

Adds a single request. `uniqueKey` and `url` are required. The request ID is your `id` if you send one, otherwise it is derived from `uniqueKey`, so adding the same `uniqueKey` twice is deduplicated: the second call returns `200` with `wasAlreadyPresent: true` and leaves the stored request unchanged.

| Query param | Type | Description |
| --- | --- | --- |
| `forefront` | boolean | `1` or `true` puts the request at the front of the pending queue. All forefront requests share the front position and are ordered among themselves by request ID. |

| Body field | Type | Description |
| --- | --- | --- |
| `uniqueKey` | string | Deduplication key (required). |
| `url` | string | Request URL (required). |
| `method` | string | HTTP method (default `GET`). |
| `id` | string | Optional explicit request ID. See [Request IDs](#request-ids). |
| `payload` | string | Optional request body. |
| `headers` | object | Optional headers. |
| `userData` | object | Optional custom data. |
| `noRetry` | boolean | Disable retries. |

Any other fields you send are stored with the request and returned by [Get request](#get-request).

```bash
curl -X POST https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"uniqueKey": "https://example.com", "url": "https://example.com", "method": "GET"}'
```

A new request returns `201`:

```json
{
  "data": {
    "requestId": "Xb3kQ9pLm2RtV7w",
    "wasAlreadyPresent": false,
    "wasAlreadyHandled": false
  }
}
```

## Add requests (batch)

```
POST /v2/request-queues/{queueId}/requests/batch
```

The body is a **JSON array** of up to 25 request objects, not a `{ "requests": [...] }` wrapper. Each item follows [Add request](#add-request). `forefront` is a query parameter.

| Query param | Type | Description |
| --- | --- | --- |
| `forefront` | boolean | `1` or `true` puts new requests at the front of the pending queue. |

```bash
curl -X POST "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/batch" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[
    { "uniqueKey": "https://example.com", "url": "https://example.com" },
    { "uniqueKey": "https://example.org", "url": "https://example.org" }
  ]'
```

Returns `201`. Items that could not be stored are listed in `unprocessedRequests` with their `uniqueKey`, `url`, and `method`:

```json
{
  "data": {
    "processedRequests": [
      { "requestId": "Xb3kQ9pLm2RtV7w", "uniqueKey": "https://example.com", "wasAlreadyPresent": false, "wasAlreadyHandled": false },
      { "requestId": "Hn4sW8cZq1YuE6d", "uniqueKey": "https://example.org", "wasAlreadyPresent": true, "wasAlreadyHandled": true }
    ],
    "unprocessedRequests": []
  }
}
```

## Delete requests (batch)

```
DELETE /v2/request-queues/{queueId}/requests/batch
```

The body is a **JSON array** of up to 25 items, each with an `id` or a `uniqueKey`. Returns `200`. Items with neither are returned in `unprocessedRequests`. Requests that do not exist are skipped and appear in neither list.

```bash [Request]
curl -X DELETE https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/batch \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{ "uniqueKey": "https://example.com" }, { "id": "Hn4sW8cZq1YuE6d" }]'
```

```json [Response]
{
  "data": {
    "processedRequests": [
      { "uniqueKey": "https://example.com", "id": "Xb3kQ9pLm2RtV7w" },
      { "uniqueKey": "https://example.org", "id": "Hn4sW8cZq1YuE6d" }
    ],
    "unprocessedRequests": []
  }
}
```

## Get request

```
GET /v2/request-queues/{queueId}/requests/{requestId}
```

Returns the stored request object, including every field you sent when adding or updating it.

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w
```

```json [Response]
{
  "data": {
    "id": "Xb3kQ9pLm2RtV7w",
    "uniqueKey": "https://example.com",
    "url": "https://example.com",
    "method": "GET"
  }
}
```

```api-test
{ "method": "GET", "path": "/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w" }
```

## Update request

```
PUT /v2/request-queues/{queueId}/requests/{requestId}
```

Merges the body into the stored request. The handled state follows `handledAt` in the body:

- A `handledAt` timestamp marks the request handled and removes it from the pending and locked sets.
- A body **without** `handledAt` (or with `handledAt: null`) marks it unhandled and puts it back in the pending queue. Always send `handledAt` when you update a handled request, or it is crawled again.

If the request does not exist, it is added and the response is `200` with `wasAlreadyPresent: false`. `wasAlreadyHandled` reports the state before this update.

| Query param | Type | Description |
| --- | --- | --- |
| `forefront` | boolean | When the request goes back to pending, put it at the front. |

```bash [Request]
curl -X PUT https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"handledAt": "2026-08-14T15:05:00.000Z"}'
```

```json [Response]
{
  "data": {
    "requestId": "Xb3kQ9pLm2RtV7w",
    "wasAlreadyPresent": true,
    "wasAlreadyHandled": false
  }
}
```

## Delete request

```
DELETE /v2/request-queues/{queueId}/requests/{requestId}
```

Returns `204 No Content`, including when the request does not exist.

```bash
curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w
```

## Get head

```
GET /v2/request-queues/{queueId}/head
```

Returns pending requests from the front of the queue **without locking them**. The first `limit` pending requests are read and those with an active lock are dropped, so the response can hold fewer than `limit` items even when more unlocked requests exist.

| Query param | Type | Description |
| --- | --- | --- |
| `limit` | number | Maximum requests to return (default `100`, max `1000`). |
| `clientKey` | string | Worker identity (1 to 32 characters). |

```bash [Request]
curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/head?limit=10&clientKey=worker-1"
```

```json [Response]
{
  "data": {
    "limit": 10,
    "queueModifiedAt": "2026-08-14T15:02:00.000Z",
    "hadMultipleClients": false,
    "items": [
      {
        "id": "Xb3kQ9pLm2RtV7w",
        "retryCount": 0,
        "uniqueKey": "https://example.com",
        "url": "https://example.com",
        "method": "GET"
      }
    ]
  }
}
```

```api-test
{ "method": "GET", "path": "/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/head?limit=10" }
```

## Get head and lock

```
POST /v2/request-queues/{queueId}/head/lock
```

Fetches pending requests and locks them so other workers cannot take them. Lock options are **query parameters**, not a JSON body.

| Query param | Type | Description |
| --- | --- | --- |
| `lockSecs` | number | Lock duration in seconds (required). |
| `limit` | number | Maximum requests to lock (default `25`, max `25`). |
| `clientKey` | string | Lock owner (1 to 32 characters). A random key is generated if omitted, but then you cannot prolong or release the locks, so always send one. |

```bash [Request]
curl -X POST \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/head/lock?lockSecs=60&limit=10&clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"
```

```json [Response]
{
  "data": {
    "limit": 10,
    "queueModifiedAt": "2026-08-14T15:02:00.000Z",
    "hadMultipleClients": false,
    "lockSecs": 60,
    "queueHasLockedRequests": true,
    "items": [
      {
        "id": "Xb3kQ9pLm2RtV7w",
        "retryCount": 0,
        "uniqueKey": "https://example.com",
        "url": "https://example.com",
        "method": "GET",
        "lockExpiresAt": "2026-08-14T15:03:00.000Z"
      }
    ]
  }
}
```

`queueHasLockedRequests` is `true` when any client, including this call, holds an active lock.

## Prolong request lock

```
PUT /v2/request-queues/{queueId}/requests/{requestId}/lock
```

Extends a lock you hold. Both query parameters are required.

| Query param | Type | Description |
| --- | --- | --- |
| `lockSecs` | number | New lock duration in seconds, counted from now (required). |
| `clientKey` | string | The `clientKey` that took the lock (required). |

Returns `404` if the request does not exist, `409 invalid-state` if it is handled, not locked, or the lock has expired, and `403 forbidden` if another client holds the lock.

```bash [Request]
curl -X PUT \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w/lock?lockSecs=60&clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"
```

```json [Response]
{
  "data": {
    "lockExpiresAt": "2026-08-14T15:07:00.000Z"
  }
}
```

## Delete request lock

```
DELETE /v2/request-queues/{queueId}/requests/{requestId}/lock
```

Releases a lock you hold, so other workers can take the request again. Returns `204 No Content`, including when the lock was already released. Returns `403 forbidden` if another client holds the lock.

| Query param | Type | Description |
| --- | --- | --- |
| `clientKey` | string | The `clientKey` that took the lock (required). |
| `forefront` | boolean | `1` or `true` puts the request at the front of the pending queue. |

```bash
curl -X DELETE \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w/lock?clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"
```

## Unlock requests

```
POST /v2/request-queues/{queueId}/requests/unlock
```

Releases every lock held by a `clientKey`, for example when a worker shuts down.

| Query param | Type | Description |
| --- | --- | --- |
| `clientKey` | string | Lock owner to match (required, 1 to 32 characters). |
| `forefront` | boolean | Put the unlocked requests at the front of the pending queue. |
| `runId` | string | Release locks taken inside this run. Only honoured for an Actor run token with access to that run; otherwise the run of the calling token is used. |

```bash [Request]
curl -X POST \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/unlock?clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"
```

```json [Response]
{
  "data": {
    "unlockedCount": 3
  }
}
```

---

- Previous: [Key-value stores](https://scrape.ly/docs/api-key-value-stores.md)
- Next: [Automations](https://scrape.ly/docs/api-automations.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)