Errors & deployment

This page covers the exceptions the Python SDK actually raises, common failures and their fixes, and how to package, test and deploy a Python Actor on Scrape.ly.

Errors

What the Actor methods raise

from scrapely import Actor, CallOptions
from scrapely.storage import ApiClientError

try:
    run = await Actor.call("owner~child-actor", {"url": url}, CallOptions(timeout=600))
except ApiClientError as error:
    if error.status_code == 404:
        Actor.log.warning("Actor does not exist")
    else:
        raise
except TimeoutError:
    Actor.log.warning("Child run timed out and was aborted")
else:
    if run.status != "SUCCEEDED":
        Actor.log.warning("Child run ended with %s", run.status)

A failed child run isn't an exception, so check run.status. ApiClientError already retries HTTP 429 three times before it raises. Errors from Actor.apify_client come from scrapely_client. See the Python API client.

Exception hierarchy

The root package also exports a typed hierarchy:

ScrapelyClientError
├── InvalidResponseBodyError
└── ScrapelyApiError
    ├── InvalidRequestError    (400)
    ├── UnauthorizedError      (401)
    ├── ForbiddenError         (403)
    ├── NotFoundError          (404)
    ├── ConflictError          (409)
    ├── RateLimitError         (429)
    └── ServerError            (5xx)

The Actor methods don't raise these. Only raise_for_status(response, attempt=1, *, method="GET") raises them, for an httpx.Response you got yourself. ScrapelyApiError has response, attempt, method, status_code, type, message and data.

ChargeRejectedError is exported too, but the SDK handles it internally: a rejected charge returns charged_count=0.

Troubleshooting

Packaging

The SDK is preinstalled. Don't add scrapely or apify to requirements.txt. For details, see Getting the SDK.

Without a Dockerfile

If your project has requirements.txt or .py files and no Dockerfile, the build generates one. It uses the scrapely/actor-python:3.12 image, installs requirements.txt and runs python /actor/main.py. Put your entrypoint in main.py at the project root.

With a Dockerfile

A typical Python Actor uses this layout:

FROM scrapely/actor-python:3.14-latest

COPY requirements.txt ./
RUN pip install -r requirements.txt

COPY . ./
CMD ["python", "-m", "my_actor"]

Pick a browser image, such as scrapely/actor-python-playwright:3.14-latest, for Playwright crawlers. The platform rewrites the tag to the SDK release it currently ships. If you use an apify/actor-python* base, the build swaps it for the matching Scrape.ly image, but publishing requires a scrapely/actor-python* base. See Publishing.

A project usually contains:

.actor/
  scrapely.json
  input_schema.json
  dataset_schema.json
Dockerfile
requirements.txt
my_actor/
  __init__.py
  __main__.py
  main.py

Testing

On your machine:

• Unit-test pure helpers, such as parsing and URL normalization, with ordinary inputs.
• Run scrapely validate-schema to catch a broken input schema before you push.
• scrapely run runs the project locally with file-system storage in ./storage. It needs an environment where import scrapely works.

On the platform:

scrapely push
scrapely call -i '{"startUrls":[{"url":"https://example.com"}]}' -o
scrapely runs log <runId>
scrapely datasets get-items <datasetId> --format json --limit 20

Start with a small input. Test charging, webhooks, abort, reboot and metamorph with real runs, because they don't work off the platform.

Before you ship a larger crawl:

1. Keep scrapely and apify out of requirements.txt.
2. Validate the input before expensive work.
3. Await every dataset, key-value store and queue write.
4. Persist crawl state with use_state() and persist_autosaved_values() if a migration would lose progress.
5. Handle Event.ABORTING if you need to clean up.
6. Keep tokens and proxy credentials out of source and logs.

Production template

import asyncio

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from scrapely import Actor, Event


async def main() -> None:
    async with Actor:
        async def on_aborting() -> None:
            await asyncio.sleep(1)
            await Actor.exit()

        Actor.on(Event.ABORTING, on_aborting)

        actor_input = await Actor.get_input() or {}
        start_urls = [item["url"] for item in actor_input.get("startUrls", [])]
        if not start_urls:
            await Actor.fail("startUrls is required")

        proxy_configuration = await Actor.create_proxy_configuration(actor_input.get("proxyConfiguration"))
        crawler = BeautifulSoupCrawler(
            proxy_configuration=proxy_configuration,
            max_requests_per_crawl=actor_input.get("maxRequestsPerCrawl", 100),
        )

        @crawler.router.default_handler
        async def handler(context: BeautifulSoupCrawlingContext) -> None:
            title = context.soup.title.string.strip() if context.soup.title else ""
            await context.push_data({"url": context.request.url, "title": title})
            await context.enqueue_links()

        await crawler.run(start_urls)


if __name__ == "__main__":
    asyncio.run(main())

context.push_data() writes to the default dataset through Crawlee. Use Actor.push_data(..., charged_event_name=...) when each item should be charged.