Environment & proxy

The platform describes each run to the container through environment variables, and the SDK parses them into a configuration object. This page lists what Actor.get_env() returns and which variables the platform sets. It also covers the remote browser, the Scrape.ly proxy and serving HTTP from a Python Actor.

Run environment

Actor.get_env()

Actor.get_env() is synchronous and works before init(). It returns an ApifyEnv object. You can read fields as attributes (env.actor_run_id) or by key (env["actor_run_id"]), and env.keys() lists the fields that are set. Fields without a value are None, except the storage IDs, which default to "default", and input_key, which defaults to "INPUT".

ApifyEnv also has compatibility fields, such as log_level, log_format, chrome_executable_path, dedicated_cpus and xvfb. Scrape.ly doesn't set them.

When a value is set under several names, the SDK applies them in this order, and later names win: APIFY_*, then SCRAPELY_*, then ACTOR_*, then CRAWLEE_STORAGE_DIR. So SCRAPELY_TOKEN overrides APIFY_TOKEN, and ACTOR_ID overrides APIFY_ACTOR_ID.

Variables the platform sets

You don't set these yourself. Most of them also have an APIFY_* copy.

Environment variables defined on the Actor version are added too. They can't override the run identity variables above (ids, build ids, ACTOR_META_ORIGIN, ACTOR_STANDBY_PORT).

Configuration

Actor.configuration (or Actor.config, as in Apify's SDK) returns the parsed Configuration. Use get(key, default=None) and set(key, value) with the snake_case keys from the table above, or read them as attributes, e.g. Actor.config.standby_port. Off the platform, set SCRAPELY_API_URL and SCRAPELY_TOKEN yourself if you use force_cloud or the API clients.

Run helpers

Remote browser

Where a browser runs depends on the Actor's Use Scrapely browser setting, which is off by default and applied per build. See Scrape.ly Browser.

Importing scrapely patches Crawlee's PlaywrightBrowserPlugin, raw Playwright (chromium.launch(), launch_persistent_context()), pyppeteer, and Camoufox. The scrapely/actor-python* images also load the patch through sitecustomize, so scripts that never import the SDK are covered. Your crawler code doesn't change:

from crawlee.crawlers import PlaywrightCrawler

crawler = PlaywrightCrawler(request_handler=handle_page)
await crawler.run(["https://example.com"])

set_cdp_url(url) overrides CDP_URL. The SDK reads the URL lazily at each browser launch.

Browser location and fingerprint

These options apply when the browser setting is on. The Scrape.ly browser has its own proxy and ignores custom proxies: a BYOP proxy_configuration passed to a browser crawler is ignored, with a warning. To choose the exit location and fingerprint, pass a BrowserConfiguration:

browser_configuration = await Actor.create_browser_configuration(country_code="US", state="California")

crawler = PlaywrightCrawler(
    browser_configuration=browser_configuration,
    request_handler=handle_page,
)

Actor.create_browser_configuration(actor_browser_input=None, *, country_code=None, state=None, city=None, os=None, timezone=None, language=None, adblock=None, solve_cloudflare=None) is async. It fills country_code, state and city from the input's browserCountry, browserState and browserCity fields when you don't pass them. You can also construct BrowserConfiguration(...) with the same keyword arguments.

Invalid values raise ValueError. The browser's location comes from the first source that sets one: browser_configuration, then the country and subdivision of a platform ProxyConfiguration, then the run's default. The first page pins a browser's location. If a later page on the same browser resolves to a different location, the SDK logs a warning. Off the platform, and in builds with the setting off, the option is accepted and ignored.

PlaywrightCrawler(browser_configuration=...) only works because the SDK patches it, and passing any other type raises TypeError. Raw chromium.launch(browser_configuration=...) works too.

Proxy configuration

Actor.create_proxy_configuration() builds a ProxyConfiguration for HTTP crawlers. It supports three modes:

• Platform proxy. The Scrape.ly proxy gateway, available in platform runs when the platform sets SCRAPELY_PROXY_*.
• Your own proxies. proxy_urls, new_url_function or tiered lists.
• Mixed. Your own lists with an "auto" (or "http://auto") entry that stands for the platform proxy.

For plans, groups and locations, see Proxy.

# Platform proxy, optionally with groups and location
proxy_configuration = await Actor.create_proxy_configuration(country_code="US")

# Your own proxies
proxy_configuration = await Actor.create_proxy_configuration(
    proxy_urls=["http://user:pass@proxy-a.example.com:8000", "http://proxy-b.example.com:8000"],
)

# From a proxy editor field in the input
actor_input = await Actor.get_input() or {}
proxy_configuration = await Actor.create_proxy_configuration(actor_input.get("proxyConfiguration"))

crawler = BeautifulSoupCrawler(proxy_configuration=proxy_configuration, request_handler=handler)

await Actor.create_proxy_configuration(
    actor_proxy_input=None, password=None, groups=None, country_code=None, subdivision_code=None,
    proxy_urls=None, new_url_function=None, tiered_proxy_urls=None, tiered_proxy_config=None,
    check_access=True,
)

The method needs an initialized SDK and returns None in two cases:

• The input has useScrapelyProxy: false and doesn't list any custom proxies.
• Nothing is configured and there are no platform proxy credentials. This happens, for example, off the platform, and the SDK logs a warning.

actor_proxy_input accepts the keys useScrapelyProxy, scrapelyProxyGroups, scrapelyProxyCountry, scrapelyProxySubdivision, proxyUrls, tieredProxyUrls and tieredProxyConfig. It also accepts the Apify aliases useApifyProxy, apifyProxyGroups, apifyProxyCountry and apifyProxySubdivision. When both names are set, the Scrapely name wins. Explicit arguments win over the input.

Actor.get_input() returns proxy objects under both names, so actor_input["proxyConfiguration"]["useScrapelyProxy"] and ["useApifyProxy"] always agree. See Apify names.

With check_access=True, the SDK makes a request through the gateway to the proxy status URL: 2 attempts, with a 4-second timeout each. When the proxy refuses the configuration, for example a group other than RESIDENTIAL or DATACENTER, it raises ConnectionError with the proxy's message, such as Proxy group "GOOGLE_SERP" is not available on Scrapely. Available groups: RESIDENTIAL, DATACENTER. See Unsupported groups. When the check gets no answer or an HTTP error, it only logs a warning.

Validation

These combinations raise ValueError when the configuration is created:

• A country_code that isn't two uppercase letters.
• A subdivision_code that isn't 1 to 3 uppercase letters or digits, or that has no country_code.
• Real custom URLs, or new_url_function, combined with groups, country_code, subdivision_code or password.
• A tiered_proxy_config tier that sets new_url_function.
• Needing the platform proxy (no custom proxies, or an "auto" entry) when there are no platform proxy credentials.

Using a ProxyConfiguration

url = await proxy_configuration.new_url(session_id="session_1")
info = await proxy_configuration.new_proxy_info(session_id="session_1")

new_url(session_id=None, request=None, proxy_tier=None) returns the next URL, rotating round-robin. A session ID, at most 50 characters from [a-zA-Z0-9._~], sticks to one URL within each tier. Platform URLs have the form http://<username>,groups-A+B,session-X,country-US_CA:<password>@<host>:<port>. new_proxy_info() returns a ProxyInfo with url, scheme, hostname, port, username, password, session_id, proxy_tier, groups, country_code and subdivision_code.

Serving HTTP

A Python Actor can run its own HTTP server on ACTOR_WEB_SERVER_PORT (4321). The platform exposes that port at ACTOR_WEB_SERVER_URL during a normal run, and it sends standby traffic to it for standby runs. The Python SDK has no standby helper server, so you start the server yourself. Actor.is_standby() tells you which mode you're in.

import asyncio
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

from scrapely import Actor


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(b"ok")


async def main():
    async with Actor:
        port = Actor.get_env().container_port or 4321
        server = ThreadingHTTPServer(("0.0.0.0", port), Handler)
        Actor.log.info("Listening on %s (standby=%s)", port, Actor.is_standby())
        await asyncio.to_thread(server.serve_forever)


if __name__ == "__main__":
    asyncio.run(main())

Bind to 0.0.0.0, not 127.0.0.1. The JavaScript SDK ships a standby helper. See Standby.