A crawler that follows links and gets data from static pages, with Crawlee handling retries and request queues. Uses Parsel, with native XPath and CSS selectors. Can't run client-side JavaScript.
Create it locally
$scrapely create my-actor --template python-crawlee-parsel
Main file
.PY
my_actor/main.py"""Module defines the main entry point for the Scrapely Actor.
Feel free to modify this file to suit your specific needs.
To build Scrapely Actors, utilize the Scrapely SDK toolkit, read more at the official documentation:
"""
from __future__ import annotations
from scrapely import Actor
from crawlee.crawlers import ParselCrawler
from .routes import router
async def main() -> None:
"""Define a main entry point for the Scrapely Actor.
This coroutine is executed using `asyncio.run()`, so it must remain an asynchronous function for proper execution.
Asynchronous execution is required for communication with Scrapely platform, and it also enhances performance in
the field of web scraping significantly.
"""
# Enter the context of the Actor.
async with Actor:
# Retrieve the Actor input, and use default values if not provided.
actor_input = await Actor.get_input() or {}
start_urls = [
url.get('url')
for url in actor_input.get(
'start_urls',
[{'url': 'https://crawlee.dev'}],
)
]
# Exit if no start URLs are provided.
if not start_urls:
Actor.log.info('No start URLs specified in Actor input, exiting...')
await Actor.exit()
# Create a crawler.
crawler = ParselCrawler(
# Limit the crawl to max requests. Remove or increase it for crawling all links.
max_requests_per_crawl=10,
# Set the request handler to the request router defined in routes.py.
request_handler=router,
)
# Run the crawler with the starting requests.
await crawler.run(start_urls)
Python site crawler (Crawlee + Parsel)
A crawler that follows links and gets data from static pages, with Crawlee handling retries and request queues. Uses Parsel, with native XPath and CSS selectors. Can't run client-side JavaScript.
Included features
- Scrapely SDK — Actor lifecycle, input, storage and proxies
- Crawlee — crawling, request queues, retries and storage
- Parsel — XPath and CSS selectors
- Input schema — define and validate the Actor's input (
.actor/input_schema.json) - Dataset schema — structured results with a preview view (
.actor/dataset_schema.json)
Getting started locally
python -m pip install -r requirements.txt
scrapely run # run the Actor locally
Deploying to Scrapely
Create an Actor from this template in the Scrapely Console (Develop new → template → Git). Scrapely creates a repository in your account, pushes this code, and builds the Actor. Every push to your repository triggers a new build.
