Crawlee + BeautifulSoup template

Fast web scraper using BeautifulSoup for HTML parsing in Python. No browser needed — ideal for static pages.

Create it locally

$scrapely create my-actor --template crawlee-bs4-py

Main file

.PY

my_actor/main.py

"""Module defines the main entry point for the Scrapely Actor.

Feel free to modify this file to suit your specific needs.

To build Scrapely Actors, utilize the Scrapely SDK toolkit.
"""

from __future__ import annotations

import asyncio

from scrapely import Actor, Event
from crawlee.crawlers import BeautifulSoupCrawler

from .routes import router


async def main() -> None:
    """Define a main entry point for the Scrapely Actor.

    This coroutine is executed using `asyncio.run()`, so it must remain an asynchronous function for proper execution.
    Asynchronous execution is required for communication with the Scrapely platform, and it also enhances performance in
    the field of web scraping significantly.
    """
    # Enter the context of the Actor.
    async with Actor:
        # Handle graceful abort - Actor is being stopped by user or platform
        async def on_aborting() -> None:
            # Persist any state, do any cleanup you need, and terminate the Actor using
            # `await Actor.exit()` explicitly as soon as possible. This will help ensure that
            # the Actor is doing best effort to honor any potential limits on costs of a
            # single run set by the user.
            # Wait 1 second to allow Crawlee/SDK state persistence operations to complete
            # This is a temporary workaround until SDK implements proper state persistence in the aborting event
            await asyncio.sleep(1)
            await Actor.exit()

        Actor.on(Event.ABORTING, on_aborting)

        # Retrieve the Actor input, and use default values if not provided.
        actor_input = await Actor.get_input() or {}
        start_urls = [
            url.get('url')
            for url in actor_input.get(
                'start_urls',
                [{'url': 'https://crawlee.dev'}],
            )
        ]

        # Exit if no start URLs are provided.
        if not start_urls:
            Actor.log.info('No start URLs specified in Actor input, exiting...')
            await Actor.exit()

        # Create a crawler.
        crawler = BeautifulSoupCrawler(
            # Limit the crawl to max requests. Remove or increase it for crawling all links.
            max_requests_per_crawl=10,
            # Set the request handler to the request router defined in routes.py.
            request_handler=router,
        )

        # Run the crawler with the starting requests.
        await crawler.run(start_urls)

Crawlee + BeautifulSoup (Python)

A fast, production-ready Scrapely Actor template built on BeautifulSoupCrawler. Give it a list of starting URLs and it fetches each page over plain HTTP, parses the HTML with BeautifulSoup (no browser needed), and stores the page title and URL in the default dataset.

Included features

Crawlee for Python and the Scrapely SDK — crawling, queues and storage
Input schema — define and validate the Actor's input (.actor/input_schema.json)
Dataset schema — structured results with a preview view (.actor/dataset_schema.json)
Proxy configuration — rotate IP addresses to avoid blocking

Getting started locally

python -m pip install -r requirements.txt
scrapely run --purge      # run the Actor locally

Input is read from .actor/input_schema.json. By default the Actor starts from https://crawlee.dev; change start_urls to point it anywhere.

Deploying to Scrapely

Create an Actor from this template in the Scrapely Console (Develop new → template → Git). Scrapely creates a repository in your account, pushes this code, and builds the Actor. Every push to your repository triggers a new build.