Request queues

Request queues hold a dynamic list of URLs to crawl, with locking so that several workers do not fetch the same request. All endpoints are prefixed with https://api.scrape.ly/v2. Authenticate with Authorization: Bearer $SCRAPELY_TOKEN.

Refer to a queue by its UUID or, on every route except updating and deleting the queue itself, by named ID: username~queue-name, where username is your own public username.

Unnamed queues expire after your account's unnamed-storage retention period: 2 days (48 hours) on the free plan, and 31, 60 or 90 days on the Starter, Scale or Business plan. On the free plan, the default storages of your latest 10 runs are kept until those runs are deleted. Named queues do not expire. Creating a named queue that you already own returns the existing queue.

A run's default queue is also reachable under the run: /v2/actor-runs/{runId}/request-queue and its sub-paths (for example /v2/actor-runs/{runId}/request-queue/head) are rewritten to /v2/request-queues/{defaultRequestQueueId}/.... You must own the run.

Endpoints

Errors and limits

Errors use the standard envelope { "error": { "type": "...", "message": "..." } }.

Request queue routes share the storage rate limit with datasets and key-value stores: by default 5,000 requests per minute for your account, plus a per-IP ceiling of 5,000 requests per minute. See Datasets API.

Request IDs

A request ID is 1 to 128 characters of letters, digits, -, _, and ~. Dots, slashes, %, and other characters are refused, so an ID is always a single URL path segment. IDs the API generates from uniqueKey are 15 letters and digits, the same IDs Crawlee and the SDKs compute.

• Routes with a {requestId} in the path return 400 invalid-request-id for any other ID, before looking anything up.
• Add request with an id outside these rules returns 400 invalid-request-id. Without an id (or with "" or null), the ID is generated.
• In the batch routes, an item with an invalid id is returned in unprocessedRequests with an error object; the rest of the batch is processed.

Client keys and lock ownership

clientKey (1 to 32 characters) identifies a worker. When a list or head call comes from a different clientKey than earlier calls, or from none, the queue's hadMultipleClients becomes true.

A lock belongs to the clientKey that took it and, inside an Actor run, to that run as well. Prolonging or releasing a lock needs the same clientKey from the same run; a lock taken inside a run cannot be released with your personal token from outside it.

Get list of request queues

Lists queues you own. Unnamed queues are omitted unless you pass unnamed=1.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/request-queues?unnamed=1&limit=20"

size is a human-readable string. For the size in bytes, use stats.storageBytes from Get request queue.

Create request queue

curl -X POST https://api.scrape.ly/v2/request-queues \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "crawl-queue"}'

A new queue returns 201 with a Location header. The body matches Get request queue.

Get request queue

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e

Update request queue

Renames the queue. {queueId} must be a UUID.

curl -X PUT https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "renamed-queue"}'

Delete request queue

Deletes the queue and all its requests. {queueId} must be a UUID. Returns 204 No Content.

curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e

List requests

Lists requests in the queue, ordered by request ID (not by the order they were added). Filter with filter; there is no status parameter. Paginate with cursor.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests?filter=pending&limit=50&clientKey=worker-1"

nextCursor is present only when there are more requests. The response also echoes cursor and exclusiveStartId when you sent them.

Add request

Adds a single request. uniqueKey and url are required. The request ID is your id if you send one, otherwise it is derived from uniqueKey, so adding the same uniqueKey twice is deduplicated: the second call returns 200 with wasAlreadyPresent: true and leaves the stored request unchanged.

Any other fields you send are stored with the request and returned by Get request.

curl -X POST https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"uniqueKey": "https://example.com", "url": "https://example.com", "method": "GET"}'

A new request returns 201:

{
  "data": {
    "requestId": "Xb3kQ9pLm2RtV7w",
    "wasAlreadyPresent": false,
    "wasAlreadyHandled": false
  }
}

Add requests (batch)

The body is a JSON array of up to 25 request objects, not a { "requests": [...] } wrapper. Each item follows Add request. forefront is a query parameter.

curl -X POST "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/batch" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[
    { "uniqueKey": "https://example.com", "url": "https://example.com" },
    { "uniqueKey": "https://example.org", "url": "https://example.org" }
  ]'

Returns 201. Items that could not be stored are listed in unprocessedRequests with their uniqueKey, url, and method:

{
  "data": {
    "processedRequests": [
      { "requestId": "Xb3kQ9pLm2RtV7w", "uniqueKey": "https://example.com", "wasAlreadyPresent": false, "wasAlreadyHandled": false },
      { "requestId": "Hn4sW8cZq1YuE6d", "uniqueKey": "https://example.org", "wasAlreadyPresent": true, "wasAlreadyHandled": true }
    ],
    "unprocessedRequests": []
  }
}

Delete requests (batch)

The body is a JSON array of up to 25 items, each with an id or a uniqueKey. Returns 200. Items with neither are returned in unprocessedRequests. Requests that do not exist are skipped and appear in neither list.

curl -X DELETE https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/batch \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{ "uniqueKey": "https://example.com" }, { "id": "Hn4sW8cZq1YuE6d" }]'

Get request

Returns the stored request object, including every field you sent when adding or updating it.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w

Update request

Merges the body into the stored request. The handled state follows handledAt in the body:

• A handledAt timestamp marks the request handled and removes it from the pending and locked sets.
• A body without handledAt (or with handledAt: null) marks it unhandled and puts it back in the pending queue. Always send handledAt when you update a handled request, or it is crawled again.

If the request does not exist, it is added and the response is 200 with wasAlreadyPresent: false. wasAlreadyHandled reports the state before this update.

curl -X PUT https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"handledAt": "2026-08-14T15:05:00.000Z"}'

Delete request

Returns 204 No Content, including when the request does not exist.

curl -X DELETE -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w

Get head

Returns pending requests from the front of the queue without locking them. The first limit pending requests are read and those with an active lock are dropped, so the response can hold fewer than limit items even when more unlocked requests exist.

curl -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/head?limit=10&clientKey=worker-1"

Get head and lock

Fetches pending requests and locks them so other workers cannot take them. Lock options are query parameters, not a JSON body.

curl -X POST \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/head/lock?lockSecs=60&limit=10&clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"

queueHasLockedRequests is true when any client, including this call, holds an active lock.

Prolong request lock

Extends a lock you hold. Both query parameters are required.

Returns 404 if the request does not exist, 409 invalid-state if it is handled, not locked, or the lock has expired, and 403 forbidden if another client holds the lock.

curl -X PUT \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w/lock?lockSecs=60&clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"

Delete request lock

Releases a lock you hold, so other workers can take the request again. Returns 204 No Content, including when the lock was already released. Returns 403 forbidden if another client holds the lock.

curl -X DELETE \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/Xb3kQ9pLm2RtV7w/lock?clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"

Unlock requests

Releases every lock held by a clientKey, for example when a worker shuts down.

curl -X POST \
  "https://api.scrape.ly/v2/request-queues/7a9e2c4b-1d3f-4b6a-8c5e-0f1a2b3c4d5e/requests/unlock?clientKey=worker-1" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN"