Proxy & events

This page covers how an Actor routes traffic through proxies, reacts to platform events, keeps state across restarts, and reports progress. For proxy pricing and the dashboard Proxy page, see Proxy.

Proxy configuration

Actor.createProxyConfiguration(options?) returns a ProxyConfiguration for crawlers and fetch clients, or undefined when there's no proxy to use. It works in three modes:

Platform proxy

Platform runs get proxy credentials in SCRAPELY_PROXY_HOSTNAME, SCRAPELY_PROXY_PORT, SCRAPELY_PROXY_USERNAME, SCRAPELY_PROXY_PASSWORD, and SCRAPELY_PROXY_STATUS_URL (mirrored as APIFY_PROXY_*). The password is the run ID, and the proxy only accepts connections from that run's container.

import { Actor, CheerioCrawler } from 'scrapely';

await Actor.main(async () => {
    const proxyConfiguration = await Actor.createProxyConfiguration({
        groups: ['RESIDENTIAL'],
        countryCode: 'US',
    });

    const crawler = new CheerioCrawler({
        proxyConfiguration,
        async requestHandler({ request, $, proxyInfo }) {
            await Actor.pushData({ url: request.url, title: $('title').text(), country: proxyInfo?.countryCode });
        },
    });
    await crawler.run(['https://example.com']);
});

The SDK encodes these options in the proxy username, e.g. groups-RESIDENTIAL,session-abc,country-US. Crawlers pass a session ID to keep the same exit IP per session. Session IDs are at most 50 characters from a-z, A-Z, 0-9, ., _, and ~; anything else throws. ProxyInfo from the platform proxy also carries groups, countryCode, and subdivisionCode.

Proxy traffic is billed per GB to the run and counts toward the run's maxTotalChargeUsd.

Your own proxies

const proxyConfiguration = await Actor.createProxyConfiguration({
    proxyUrls: ['http://user:pass@proxy-a.example.com:8000', 'http://user:pass@proxy-b.example.com:8000'],
});

You can also pass newUrlFunction(sessionId), or tiers:

const proxyConfiguration = await Actor.createProxyConfiguration({
    tieredProxyUrls: [
        [null],                                    // tier 0: no proxy
        ['auto'],                                  // tier 1: platform proxy
        ['http://user:pass@premium.example.com:8000'],
    ],
});

tieredProxyConfig takes a list of option objects instead. A tier with proxyUrls uses them; a tier without proxyUrls becomes a platform proxy tier with its own groups and countryCode. Tiers with newUrlFunction aren't supported and throw.

Errors and undefined

createProxyConfiguration() returns undefined when:

• useScrapelyProxy: false is set and no proxy URLs, function, or tiers are given. With proxyUrls, useScrapelyProxy: false still uses your proxies.
• No proxy URLs, function, or tiers are given and the run has no platform proxy credentials (for example, a local run). The SDK logs a warning explaining how to set SCRAPELY_PROXY_* locally from the credentials on the dashboard Proxy page. Using the platform proxy from outside Scrape.ly requires an active plan or a prepaid balance of at least $1.

It throws when:

• you combine proxyUrls or newUrlFunction with groups, countryCode, or subdivisionCode (Cannot combine custom proxies with Scrapely Proxy!);
• you pass both proxyUrls and newUrlFunction;
• you use 'auto' or a URL-less tier without platform proxy credentials;
• countryCode isn't two letters, or subdivisionCode is invalid or has no countryCode;
• on the platform, the proxy refuses the configuration, for example a group other than RESIDENTIAL or DATACENTER (Proxy group "GOOGLE_SERP" is not available on Scrapely. Available groups: RESIDENTIAL, DATACENTER.).

checkAccess (default true) sends a test request through the platform proxy (two attempts) and logs the result. When the proxy refuses the configuration, createProxyConfiguration() throws on the platform and logs a warning in local runs. When the check gets no answer or an HTTP error, it only logs a warning. Pass checkAccess: false to skip it; the proxy still refuses requests for unknown groups with 407.

Remote browser location

When the Actor's Use Scrapely browser setting is on, browser crawlers connect to the Scrape.ly browser over CDP. To pick its exit location and fingerprint, create a BrowserConfiguration and pass it next to the crawler's other options:

import { Actor, PlaywrightCrawler } from 'scrapely';

await Actor.main(async () => {
    const browserConfiguration = await Actor.createBrowserConfiguration({
        countryCode: 'US',
        state: 'California',
        os: 'windows',
    });

    const crawler = new PlaywrightCrawler({
        browserConfiguration,
        async requestHandler({ request, page }) {
            await Actor.pushData({ url: request.url, title: await page.title() });
        },
    });
    await crawler.run(['https://example.com']);
});

Invalid values throw BrowserConfiguration: "<field>" .... Locally, and in builds with the setting off, the option is accepted and ignored.

The browser's location comes from the first of these that is set:

1. browserConfiguration;
2. the country (and subdivision, as a state) of a platform proxyConfiguration on the same crawler;
3. the run's default country.

Proxy groups and session IDs don't apply to the Scrape.ly browser.

Events and graceful shutdown

Actor.on('persistState', saveCheckpoint);
Actor.off('persistState', saveCheckpoint);

Register listeners after Actor.init() (inside Actor.main()), so they attach to the platform event manager. off() without a listener removes all listeners for that event.

systemInfo also feeds Crawlee's autoscaling. The SDK reconnects to the events WebSocket every 5 seconds if the connection drops.

With gracefulShutdown on (the default), the SDK handles the last two for you: aborting calls Actor.exit() and migrating calls Actor.reboot(), after gracefulShutdownDelayMillis. Don't call Actor.reboot() from your own migrating listener as well. Save state in a persistState listener, or use useState().

After aborting the platform stops the container within about 30 seconds, so keep cleanup short.

Actor.events returns the underlying EventEmitter. Before init() it's a standalone emitter; after init() it's the platform event manager's.

State persistence and migration

Actor.useState()

useState() returns an object that is saved to a key-value store on every persistState event:

const state = await Actor.useState('crawl-state', { processed: 0, cursor: null as string | null });
state.processed += 1;

Mutate the returned object in place; don't reassign it. It must be JSON-serializable. If the record already exists (for example after a reboot), you get the saved value instead of the default.

The third argument selects another store:

const state = await Actor.useState('crawl-state', { cursor: 0 }, { keyValueStoreName: 'state-store' });

Actor.reboot()

reboot() restarts the container while keeping the run, its ID, and its storages. It:

1. emits persistState with isMigrating: true and waits 1 second;
2. sends POST /v2/actor-runs/{runId}/reboot;
3. waits briefly, flushes storage and charges, and calls process.exit(0).

Code after await Actor.reboot() doesn't run. The new container starts the Actor from the beginning, so restore progress from useState() or your own records. Only RUNNING, STANDBY, and READY runs can be rebooted.

Status messages

await Actor.setStatusMessage('Downloading page 3');
await Actor.setStatusMessage('Completed', { isStatusMessageTerminal: true });

setStatusMessage(message, options?) logs [Status message]: <message> at level (DEBUG, INFO, WARNING, or ERROR; default INFO) and updates the run with PUT /v2/actor-runs/{runId} (statusMessage, isStatusMessageTerminal). The level only affects the log line. It returns the updated run, or {} when there are no run credentials (local runs). It throws if the API rejects the update.

isStatusMessageTerminal: true marks the message as final for the run. Use it for the last message before exiting.