JavaScript client

scrapely-client is the JavaScript and TypeScript client for the Scrape.ly API. Use it from your own backend, script, or service to start Actors, wait for runs, read datasets and key-value stores, and manage tasks, schedules, and webhooks. It is not for writing Actors: code that runs inside an Actor should use the JavaScript SDK.

The source code is on GitHub: scrapely-app/scrapely-client.

The client is a drop-in replacement for apify-client. Class and method names match, so most existing apify-client code works after you change the import and the base URL.

Install

npm install scrapely-client

The package ships ES module and CommonJS builds with TypeScript types:

import { ScrapelyClient } from 'scrapely-client';
// or
const { ScrapelyClient } = require('scrapely-client');

ApifyClient is exported as an alias of ScrapelyClient, so import { ApifyClient } from 'scrapely-client' also works.

Browser and edge runtimes

Bundlers that honour the browser export condition pick up a browser bundle automatically. You can also import it explicitly:

import { ScrapelyClient } from 'scrapely-client/browser';

The browser bundle has no proxy support and no log streaming. Edge runtimes may need Node.js compatibility enabled. Do not ship your API token in code that other people load in their browser; call the API from your backend instead.

Create a client

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({
    token: process.env.SCRAPELY_TOKEN,
    baseUrl: 'https://api.scrape.ly',
});

Create a token in the dashboard under Settings → API & Integrations. The client does not read the token or the base URL from environment variables; pass them explicitly. Requests send it as Authorization: Bearer <token>.

In Node.js, outgoing requests honour the standard HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables.

Run an Actor

actor(id).call() starts a run and waits until it finishes. Pass an Actor ID or a username/actor-name full name.

const run = await client.actor('username/my-actor').call({ url: 'https://example.com' });

console.log(run.status); // SUCCEEDED, FAILED, ABORTED, or TIMED-OUT

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

call() resolves with the run object for every final status; it does not throw when the run fails. Check run.status yourself.

To start a run without waiting, use start(), then wait later:

const run = await client.actor('username/my-actor').start(
    { url: 'https://example.com' },
    { memory: 1024, timeout: 300 },
);

// ...later
const finished = await client.run(run.id).waitForFinish({ waitSecs: 120 });

waitForFinish() polls until the run reaches a final status or waitSecs elapses. Without waitSecs it waits indefinitely. It returns the latest run object either way.

On runs of someone else's Actor, run.buildId, run.buildNumber, and run.options.build are missing: build info is owner only, and the run uses the User Build.

Run options for start() and call():

The API returns a run's log as a snapshot rather than a live stream, so the redirected log in call() may be incomplete. Fetch the full log after the run with client.run(run.id).log().get().

See Run Actors for how runs, memory, and timeouts work on the platform.

Other Actor methods

Runs

const run = await client.run('RUN_ID').get();
await client.run('RUN_ID').abort({ gracefully: true });

Datasets

const dataset = client.dataset('DATASET_ID');

const page = await dataset.listItems({ limit: 100, fields: ['title', 'url'] });
console.log(page.total, page.items.length);

const csv = await dataset.downloadItems('csv', { bom: true });

See Results and Storage for the data model.

Key-value stores

const store = client.keyValueStore('STORE_ID');

await store.setRecord({ key: 'config', value: { depth: 2 } });

const record = await store.getRecord('OUTPUT');
if (record) console.log(record.contentType, record.value);

Request queues

const queue = client.requestQueue('QUEUE_ID');

await queue.addRequest({ url: 'https://example.com', uniqueKey: 'https://example.com' });
const { items } = await queue.listHead({ limit: 10 });

Available methods: addRequest(request, { forefront }), batchAddRequests(requests, { forefront }), getRequest(id), updateRequest(request), deleteRequest(id), batchDeleteRequests(requests), listHead({ limit }), listAndLockHead({ lockSecs, limit }), prolongRequestLock(id, { lockSecs }), deleteRequestLock(id), unlockRequests(), listRequests(), paginateRequests(), plus get(), update(), and delete(). requestQueues().getOrCreate(name) returns a named queue.

batchAddRequests splits large arrays into API-sized batches and retries requests the API reports as unprocessed.

Tasks

Tasks store an Actor's input and run options so you can run the same configuration repeatedly.

const task = await client.tasks().create({
    actId: 'username/my-actor',
    name: 'daily-crawl',
    input: { url: 'https://example.com' },
});

const run = await client.task(task.id).call();

The API requires actId and name; task names are limited to 30 characters.

Schedules

await client.schedules().create({
    name: 'nightly',
    cronExpression: '0 3 * * *',
    timezone: 'UTC',
    actions: [{ type: 'RUN_ACTOR', actorId: 'ACTOR_ID' }],
});

The API requires cronExpression and at least one action. Actions are RUN_ACTOR (with actorId) or RUN_ACTOR_TASK (with actorTaskId). isEnabled defaults to true. timezone defaults to UTC, and the cron expression is evaluated in it (see Cron expressions). Other methods: schedules().list(), schedule(id).get(), .update(fields), .delete(), and .getLog(). See Schedules API.

Automations

const automation = await client.automations().create({
    title: 'Morning digest',
    trigger: { kind: 'schedule', cron: '0 7 * * 1-5', timezone: 'Europe/Berlin' },
    steps: [{ kind: 'savedInput', savedInputId: 'SAVED_INPUT_ID', inputOverride: { maxItems: 20 } }],
    filters: { onlyNewItems: { keyFields: ['url'] } },
    ai: { mode: 'summarize', prompt: 'The five most interesting new items, with links.' },
    deliveries: [{ channel: 'email' }],
});

const run = await client.automation(automation.id).run();

automations() is Scrape.ly-only and comes with version 2.25.9 of scrapely-client.

Webhooks

await client.webhooks().create({
    eventTypes: ['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED'],
    condition: { actorId: 'ACTOR_ID' },
    requestUrl: 'https://example.com/hooks/scrapely',
});

The API requires requestUrl and at least one event type. Other methods: webhooks().list(), webhook(id).get(), .update(fields), .delete(), .test(), .dispatches(), and webhookDispatches().list(). See Webhooks API for event types and payloads.

Builds

Account

const me = await client.user().get();
const usage = await client.user().monthlyUsage();
const limits = await client.user().limits();

user() defaults to the account that owns the token.

Store

const { items } = await client.store().list({ search: 'google maps', limit: 10 });

Options: search, limit (the API defaults to 20 and caps it at 100), offset, sortBy, category, username, and pricingModel. See Store API.

Logs

const text = await client.run('RUN_ID').log().get();
const buildLog = await client.build('BUILD_ID').log().get();

log(id) on the root client accepts a run ID or a build ID. get() returns the log as a string, or undefined if it does not exist. In Node.js, stream() returns the log as a Readable. The API sends the log as it is at request time and then closes the connection, so to follow a running Actor, poll get().

Pagination

List methods return a value that is both a promise of one page and an async iterable over every item:

// One page
const page = await client.actors().list({ limit: 20 });

// Every item, fetched page by page
for await (const run of client.runs().list({ desc: true })) {
    console.log(run.id, run.status);
}

for await (const item of client.dataset('DATASET_ID').listItems({ chunkSize: 1000 })) {
    // ...
}

chunkSize sets how many items each underlying request fetches while iterating. limit and offset still bound the overall range. keyValueStore(id).listKeys() iterates the same way using exclusiveStartKey.

Errors and retries

The client retries network errors, invalid response bodies, 429, and 5xx responses up to maxRetries times with exponential backoff. Other 4xx responses fail immediately. Each attempt's timeout doubles, up to timeoutSecs. Streaming uploads are not retried.

Failed API calls throw ScrapelyApiError (also exported as ApifyApiError):

import { ScrapelyApiError } from 'scrapely-client';

try {
    await client.actor('username/missing').start();
} catch (err) {
    if (err instanceof ScrapelyApiError) {
        console.error(err.statusCode, err.type, err.message, err.attempt);
    }
    throw err;
}

The error carries statusCode, type, message, attempt, httpMethod, path, clientMethod, and data. See API overview for error types.

get() methods do not throw on 404; they return undefined. Date fields whose names end in At are converted to Date objects.

Migrating from apify-client

• Replace the import with scrapely-client. ApifyClient and ApifyApiError still work as aliases.
• Set baseUrl (and publicBaseUrl if you build public URLs) to https://api.scrape.ly.
• Use a Scrape.ly token. Apify tokens do not work.
• store().list() calls GET /v2/store/actors.
• task(id).publish() and .unpublish() are Scrape.ly additions.
• Endpoints that Scrape.ly does not implement return an error. The API reference lists what is available.