How to Automate Web Scraping: Schedules, Cron, Filters and Deliveries

To automate web scraping, you need three things: a trigger that starts the scraper without you (a schedule or another event), a filter that keeps only what's new or important, and a delivery that puts the results where you work, such as a spreadsheet, a Slack channel or your own API. Get those right and a scraper stops being a script you remember to run and becomes a feed you rely on.

This guide covers the ways to automate web scraping, from a crontab line to GitHub Actions to Scrape.ly automations, with cron syntax, a worked price-watch example built on a real run, the API and webhook code, and the mistakes that make scheduled scrapers fail quietly.

Ways to automate web scraping compared

The GitHub facts come from its workflow trigger docs: the schedule event "can be delayed during periods of high loads", the shortest interval is every 5 minutes, and in a public repository scheduled workflows are disabled after 60 days without repository activity.

Cron expressions in 2 minutes

Almost every scheduler, including Scrape.ly, describes the timetable with a five-field cron expression: minute, hour, day of month, month, day of week.

Two rules trip people up. When you restrict both day of month and day of week, a day matches if either matches: 0 9 1 * MON runs on the 1st and on every Monday. And a cron expression means nothing without a time zone. On Scrape.ly each schedule has an IANA time zone (default UTC) and follows its daylight saving changes, so 0 9 * * * with Europe/Berlin runs at 09:00 Berlin time all year. The cron reference lists every form, including how runs behave on the night the clocks change.

Option 1: cron on your own machine

If your scraper is a script on a server that never sleeps, one line in crontab -e schedules it:

0 6 * * * cd /home/you/book-tracker && /usr/bin/python3 scraper.py >> scraper.log 2>&1

Use absolute paths (cron runs with a minimal environment), redirect output to a log, and remember that nothing tells you when it starts failing. Everything after "run it" is yours to build: retries, proxies, deduplication, delivery and alerts. Our guide to building a web scraper shows a scraper with SQLite storage that's ready for a cron job.

Option 2: GitHub Actions

For a scraper that lives in a repository, a scheduled workflow runs it on GitHub's machines:

name: scrape
on:
  schedule:
    - cron: "17 6 * * *"   # off the full hour, when load is lower
  workflow_dispatch:
jobs:
  scrape:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - run: pip install requests beautifulsoup4
      - run: python scraper.py

It's handy for small jobs that already live in a repository. The catches are the ones above (delays, dropped jobs at peak times, workflows disabled in quiet public repos), plus requests that come from data-center IP ranges, which many sites rate-limit or block, and results that end up as files you still have to move somewhere.

Option 3: Scrape.ly automations

On Scrape.ly the scraper is an Actor (a ready-made one from the Store or your own code), and an automation runs it. Open Automations in the side menu, click New automation, and fill in up to six sections:

1. When. On a schedule (hourly, daily, weekly, monthly, or a custom cron expression) with a time zone; Next runs previews the next three times. Or After another run finishes, to chain scrapers.
2. Run. Up to 10 Actors, each with a saved input or a preset, all started together. Run options sets Max results, Max cost per run, memory and timeout.
3. Keep. Only when there are results, Only new items since the last run (compared by a key field such as url), and Only rows where… conditions such as price is less than 30. Number comparisons also read prices written as text, such as €1,299.00.
4. AI step (optional). Summarize, Filter (decide whether to send at all), Compare with last run ("Which prices went down, and by how much?") or Extract. It reads up to 200 items per Actor run and is billed per use, like the AI chat.
5. Send. Up to 10 deliveries: dashboard, email, Slack, Discord, Google Sheets, Amazon S3 or R2, GitHub, a webhook, or another automation.
6. Options. Skip if the previous run is still going and Alert me if it fails are both on by default.

Run once now on the automation's page checks the whole chain immediately. Each firing appears in the run history with the runs, the filter result, the AI output and its cost, and each delivery's status. The automations docs cover every option.

A worked example: a daily price watch

Say you want to know each morning which travel books on books.toscrape.com (a sandbox store for practice) cost under £30. The scraper is the Web Scraper Actor with a one-line page function that returns every book on the category page:

{
  "startUrls": [{ "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html" }],
  "pageFunction": "async function pageFunction(context) { const $ = context.jQuery; return $('article.product_pod').map((i, el) => ({ title: $(el).find('h3 a').attr('title'), price: $(el).find('p.price_color').text(), inStock: $(el).find('p.availability').text().includes('In stock') })).get(); }",
  "proxyConfiguration": { "useScrapelyProxy": true }
}

We ran exactly this input on 26 October 2026, which is what each scheduled firing would do. It took 7.6 seconds and returned 11 books through Scrape.ly's residential proxy:

Save the input (Save as new… in the Input menu), then create the automation: When daily at 09:00 in your time zone; Run that saved input with Max cost per run set to $0.05; Keep only rows where price is less than 30, plus Only new items since the last run with key field title; Send to Google Sheets and Slack. Of the 11 rows above, the price condition keeps two, and from the second morning on, "only new items" means you hear about a book only when it first drops under £30 (or newly appears below it).

The cost: Web Scraper charges $0.000715 per result with platform usage included, so one firing is 11 × $0.000715 = $0.0079 at list price, and a month of daily runs is about $0.24. That's about 5% of the free plan's $5 of monthly credit. The same pattern scales to real retailers with the Store's shop scrapers; our price scraping guide shows it with Google Shopping and Amazon, and scraping a website into Google Sheets covers the sheet side, including how to build a price history by appending rows.

Automate scraping with the API

Everything in the editor is also an API call, so you can create automations from code, for example one per client or per product list. With the JavaScript client (npm install scrapely-client):

import { ScrapelyClient } from 'scrapely-client';

const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' });

const automation = await client.automations().create({
  title: 'Travel books under £30',
  trigger: { kind: 'schedule', cron: '0 9 * * *', timezone: 'Europe/London' },
  steps: [{
    kind: 'savedInput',
    savedInputId: 'SAVED_INPUT_ID',
    runOptions: { maxTotalChargeUsd: 0.05 },
  }],
  filters: {
    onlyNewItems: { keyFields: ['title'] },
    conditions: [{ field: 'price', op: 'LESS_THAN', value: 30 }],
  },
  ai: { mode: 'summarize', prompt: 'List the new books under £30 with their prices.' },
  deliveries: [{ channel: 'slack' }, { channel: 'email' }],
});

const run = await client.automation(automation.id).run(); // test it now

The key needs the automations:write scope. If you only need the timetable (no filters, AI or deliveries), the Schedules API is simpler, and existing schedule clients keep working with it. In Python (pip install scrapely-client-python):

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

client.schedules().create(
    name="travel-books-daily",
    cron_expression="0 9 * * *",
    timezone="Europe/London",
    is_enabled=True,
    is_exclusive=True,
    actions=[{"type": "RUN_ACTOR_TASK", "actorTaskId": "SAVED_INPUT_ID"}],
)

Every schedule is an automation, so it shows up in the same list, and the CLI reads them too: scrapely api GET /v2/schedules and scrapely api GET /v2/schedules/<scheduleId>/log, which lists each firing as "Successfully started 1 run(s)" or the error.

React to runs with webhooks

Schedules start runs; webhooks tell your systems when a run finished. A webhook sends a POST to your URL on events such as ACTOR.RUN.SUCCEEDED or ACTOR.RUN.FAILED, with the run's ID, status, timing and defaultDatasetId, so your endpoint can fetch the results the moment they exist:

await client.webhooks().create({
  eventTypes: ['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED'],
  condition: { actorId: 'ACTOR_ID' },
  requestUrl: 'https://example.com/hooks/scrapely?token=LONG_RANDOM_SECRET',
});

Failed deliveries (network errors, 429, 5xx) are retried, and every request carries an X-Scrapely-Webhook-Dispatch-Id header that stays the same across retries, so your endpoint can ignore duplicates. Put a secret in the URL or a custom header and check it, since you can't verify the platform's signature yourself. A webhook delivery on an automation goes further: its POST contains the kept items themselves (up to 100) and the AI text, so a small endpoint can write them straight into your database. Without code, the Start another actor after this one recipe chains scrapers: crawl a list, then enrich each result with a second Actor.

What to watch out for

• No cap means no cap. A scheduled run has no spending limit unless you set Max cost per run (or maxTotalChargeUsd). Set one on every automation, especially broad searches.
• Overlapping runs. A slow run plus a frequent schedule means two copies scraping at once. Keep Skip if the previous run is still going on.
• Missed times. Scheduled runs start up to about a minute late, and a time missed during an outage is skipped, not caught up. Don't build anything that assumes exact-minute timing.
• Scraping too often. Hourly runs of a page that changes daily cost 24 times as much and load the site for nothing; our Travel example would go from about $0.24 to $5.66 a month. Match the schedule to how often the data changes, and see our web scraping best practices for polite rates.
• Silent breakage. A redesign can turn a scraper's output into empty rows. Only when there are results stops empty deliveries, and Alert me if it fails emails you on failures.
• Plan limits. The free plan includes 5 schedules; automations and schedules share that limit.

FAQ

How do I schedule a web scraper to run daily?

On your own machine, add a crontab line such as 0 6 * * * that runs the script. In the cloud, create a Scrape.ly automation with On a schedule → Daily, pick the time and time zone, and add your Actor and saved input under Run.

Can I automate web scraping without coding?

Yes. Pick a ready-made scraper from the Scrape.ly Store, fill in its input form, and create an automation with a schedule, filters and deliveries to Google Sheets, Slack or email, all in the dashboard. Our list of no-code web scrapers compares other options.

How often should I run a scraper?

As often as the data changes and you act on it. Prices and stock: daily or a few times a day. Reviews and job listings: daily. Rankings and directories: weekly. More frequent runs cost more and load the site without giving you new information.

How do I get scraped data into Google Sheets automatically?

Add a Google Sheets delivery to the automation and share the sheet with Scrape.ly first. Each run appends the kept items as rows; with Only new items since the last run the sheet grows only by new entries.

How do I get notified only when something changes?

Use Only new items since the last run with a key field, or the AI step's Compare with last run mode, which leads with what changed. Combine either with Only when there are results so quiet days send nothing.

Is automated web scraping legal?

Automation doesn't change the legal picture; what you collect and how does. Stick to public data, respect site terms and rate limits, and handle personal data under privacy law. This isn't legal advice. Scrape.ly Store Actors only collect publicly available data shown to any visitor.