Python client

scrapely-client-python is the Python client for the Scrape.ly API, with a synchronous and an asynchronous interface. Use it from your own scripts, backends, notebooks, or data pipelines to start Actors, wait for runs, and read and write storage. It is not for writing Actors: code that runs inside an Actor should use the Python SDK.

The source code is on GitHub: scrapely-app/scrapely-client-python.

The client is a drop-in replacement for apify-client. Method names, arguments, and return models match, so most existing code works after you change the import and the API URL.

Install

The client requires Python 3.11 or newer.

pip install scrapely-client-python

The distribution is called scrapely-client-python, but the import name is scrapely_client:

from scrapely_client import ScrapelyClient, ScrapelyClientAsync

ApifyClient and ApifyClientAsync are exported as aliases. For Brotli response compression, install the extra: pip install "scrapely-client-python[brotli]".

Create a client

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(
    os.environ['SCRAPELY_TOKEN'],
    api_url='https://api.scrape.ly',
)

Create a token in the dashboard under Settings → API & Integrations. The client does not read the token or the API URL from environment variables; pass them explicitly. The token is sent as Authorization: Bearer <token>.

Every argument after token is keyword-only.

Sync and async clients

ScrapelyClient blocks; ScrapelyClientAsync has the same methods as coroutines. Resource accessors such as client.actor(...) are plain calls in both; only the API calls are awaited.

import asyncio
from scrapely_client import ScrapelyClientAsync

async def main() -> None:
    client = ScrapelyClientAsync('MY-TOKEN', api_url='https://api.scrape.ly')
    run = await client.actor('username/my-actor').call(run_input={'url': 'https://example.com'})
    async for item in client.dataset(run.default_dataset_id).iterate_items():
        print(item)

asyncio.run(main())

Run an Actor

actor(id).call() starts a run and waits until it finishes. Pass an Actor ID or a username/actor-name full name.

run = client.actor('username/my-actor').call(run_input={'url': 'https://example.com'})

print(run.status)  # SUCCEEDED, FAILED, ABORTED, or TIMED-OUT

for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

Return values are Pydantic models with snake_case attributes: run.id, run.act_id, run.status, run.default_dataset_id, run.default_key_value_store_id, and so on. call() returns the run for every final status; it does not raise when the run fails, so check run.status. It returns None only if the run can no longer be found.

On runs of someone else's Actor, run.build_id, run.build_number, and run.options.build are None: build info is owner only, and the run uses the User Build. Older client versions raise a validation error on these runs, so keep the client up to date.

To start a run without waiting, use start(), then wait later:

from datetime import timedelta

run = client.actor('username/my-actor').start(
    run_input={'url': 'https://example.com'},
    memory_mbytes=1024,
    run_timeout=timedelta(minutes=5),
)

finished = client.run(run.id).wait_for_finish(wait_duration=timedelta(minutes=2))

wait_for_finish() polls until the run reaches a final status or wait_duration elapses. Without wait_duration it waits indefinitely.

Run arguments for start() and call():

The API returns a run's log as a snapshot rather than a live stream, so the redirected log in call() may be incomplete. Fetch the full log after the run with client.run(run.id).log().get().

See Run Actors for how runs, memory, and timeouts work on the platform.

Other Actor methods

Runs

Datasets

dataset = client.dataset('DATASET_ID')

page = dataset.list_items(limit=100, fields=['title', 'url'])
print(page.total, len(page.items))

csv_bytes = dataset.get_items_as_bytes(item_format='csv', bom=True)

See Results and Storage for the data model.

Key-value stores

store = client.key_value_store('STORE_ID')

store.set_record('config', {'depth': 2})

record = store.get_record('OUTPUT')
if record:
    print(record['content_type'], record['value'])

Request queues

queue = client.request_queue('QUEUE_ID')

queue.add_request({'url': 'https://example.com', 'uniqueKey': 'https://example.com'})
head = queue.list_head(limit=10)

Available methods: add_request, batch_add_requests, get_request, update_request, delete_request, batch_delete_requests, list_head, list_and_lock_head, prolong_request_lock, delete_request_lock, unlock_requests, list_requests, iterate_requests, plus get, update, and delete. request_queues().get_or_create(name=...) returns a named queue.

Tasks

task = client.tasks().create(
    actor_id='username/my-actor',
    name='daily-crawl',
    task_input={'url': 'https://example.com'},
)

run = client.task(task.id).call()

Task names are limited to 30 characters.

Schedules

client.schedules().create(
    name='nightly',
    cron_expression='0 3 * * *',
    timezone='UTC',
    is_enabled=True,
    is_exclusive=False,
    actions=[{'type': 'RUN_ACTOR', 'actorId': 'ACTOR_ID'}],
)

cron_expression, is_enabled, and is_exclusive are required arguments, and the API requires at least one action. Actions are RUN_ACTOR (with actorId) or RUN_ACTOR_TASK (with actorTaskId). Other methods: schedules().list(), schedule(id).get(), .update(...), .delete(), and .get_log(). See Schedules API.

Automations

automation = client.automations().create({
    'title': 'Morning digest',
    'trigger': {'kind': 'schedule', 'cron': '0 7 * * 1-5', 'timezone': 'Europe/Berlin'},
    'steps': [{'kind': 'savedInput', 'savedInputId': 'SAVED_INPUT_ID', 'inputOverride': {'maxItems': 20}}],
    'filters': {'onlyNewItems': {'keyFields': ['url']}},
    'ai': {'mode': 'summarize', 'prompt': 'The five most interesting new items, with links.'},
    'deliveries': [{'channel': 'email'}],
})

run = client.automation(automation['id']).run()

The methods return the API's dictionaries with camelCase keys, and the async client has the same methods. automations() is Scrape.ly-only and comes with version 1.0.6 of scrapely-client-python.

Webhooks

client.webhooks().create(
    event_types=['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED'],
    actor_id='ACTOR_ID',
    request_url='https://example.com/hooks/scrapely',
)

Other methods: webhooks().list(), webhook(id).get(), .update(...), .delete(), .test(), .dispatches(), and webhook_dispatches().list(). See Webhooks API.

Builds, account, and Store

Builds are owner only, also for public Actors; other users get 403 on an Actor's builds and 404 on a build by ID.

Logs

text = client.run('RUN_ID').log().get()

with client.run('RUN_ID').log().stream() as response:
    if response:
        for chunk in response.iter_bytes():
            print(chunk.decode(), end='')

get() returns the log as a string, or None. With the async client, use async with and aiter_bytes(). The API sends the log as it is at request time and then closes the connection, so to follow a running Actor, poll get().

run(id).get_streamed_log(to_logger) returns a StreamedLog you can use as a context manager (or call start() and stop()) to redirect a run's log into a logging.Logger; it has the same snapshot limitation. The client logs under the scrapely_client logger name.

Pagination

Every collection and storage listing has two forms:

• list(...), list_items(...), list_keys(...) return a single page with its metadata.
• iterate(...), iterate_items(...), iterate_keys(...), iterate_requests(...) return an iterator that fetches pages as you go. With the async client, use async for.

for run in client.runs().iterate(desc=True):
    print(run.id, run.status)

limit and offset bound the overall range; chunk_size sets how many items each request fetches.

Timeouts, retries, and errors

Every API method takes a keyword-only timeout: a timedelta, one of 'short', 'medium', 'long', or 'no_timeout'. Each retry doubles the timeout, up to timeout_max. call() and wait_for_finish() default to 'no_timeout'.

The client retries 429, 5xx, and retryable network errors up to max_retries times with exponential backoff and jitter. Other 4xx responses fail immediately.

Failed calls raise ScrapelyApiError from scrapely_client.errors (also available as ApifyApiError). The client raises a subclass based on the HTTP status:

from scrapely_client.errors import NotFoundError, ScrapelyApiError

try:
    client.actor('username/missing').start()
except NotFoundError:
    print('No such Actor')
except ScrapelyApiError as err:
    print(err.status_code, err.type, err.message, err.attempt)

get() methods return None on 404 instead of raising. See API overview for error types.

Migrating from apify-client

• Install scrapely-client-python and import from scrapely_client. ApifyClient, ApifyClientAsync, and ApifyApiError still work as aliases.
• Pass api_url='https://api.scrape.ly' (and api_public_url if you build public URLs).
• Use a Scrape.ly token. Apify tokens do not work.
• task(id).publish() and .unpublish() are Scrape.ly additions.
• Endpoints that Scrape.ly does not implement return an error. The API reference lists what is available.