--- title: "Automate Web Scraping: Schedules and Webhooks | Scrape.ly" description: "Automate web scraping: schedule scrapers with cron, keep only new items, add an AI step, and send results to Google Sheets, Slack, email or a webhook." url: https://scrape.ly/blog/schedule-web-scraping markdown: https://scrape.ly/blog/schedule-web-scraping.md --- [Guides](https://scrape.ly/blog/category/guides.md) # How to Automate Web Scraping: Schedules, Cron, Filters and Deliveries How to automate web scraping end to end: run scrapers on a schedule with cron expressions, keep only new or changed items, add an AI summary, and deliver results to Google Sheets, Slack, email or a webhook. Compared with cron and GitHub Actions, with real run costs. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 26, 2026 - Reading time 9 min read ![Pixel-art robot with an orange >\_ on its terminal head carries a data card to a crate at the end of a conveyor belt that runs out of a wall calendar with orange scheduled days, the past ones ticked green, and a clock on top, another card riding the belt](https://scrape.ly/media/schedule-web-scraping.png?w=1080) To automate web scraping, you need three things: a trigger that starts the scraper without you (a schedule or another event), a filter that keeps only what's new or important, and a delivery that puts the results where you work, such as a spreadsheet, a Slack channel or your own API. Get those right and a scraper stops being a script you remember to run and becomes a feed you rely on. This guide covers the ways to automate web scraping, from a crontab line to GitHub Actions to Scrape.ly automations, with cron syntax, a worked price-watch example built on a real run, the API and webhook code, and the mistakes that make scheduled scrapers fail quietly. ## Ways to automate web scraping compared | | cron on your server | GitHub Actions schedule | Scrape.ly automation | | ---------------------------------------- | ---------------------- | ----------------------------------------------------- | ----------------------------------------------- | | Always on | Only if the machine is | Yes | Yes | | Timing | Exact | Can be delayed under load; queued jobs may be dropped | Up to about a minute after the scheduled time | | Proxies and anti-bot | You set them up | You set them up | Residential proxy and anti-bot browser built in | | Keep only new items | You write it | You write it | Built-in filter | | Deliver to Sheets, Slack, email, webhook | You write it | You write it | Built-in deliveries | | Failure alerts | You build them | You configure them | Email and dashboard alert, on by default | | Cost cap per run | No | No | Maximum charge per run | The GitHub facts come from its [workflow trigger docs](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows): the schedule event "can be delayed during periods of high loads", the shortest interval is every 5 minutes, and in a public repository scheduled workflows are disabled after 60 days without repository activity. ## Cron expressions in 2 minutes Almost every scheduler, including Scrape.ly, describes the timetable with a five-field cron expression: minute, hour, day of month, month, day of week. | Expression | Runs | | ------------------ | ---------------------------------------------------------- | | `0 9 * * *` | Every day at 09:00 | | `30 8 * * MON-FRI` | Weekdays at 08:30 | | `0 */6 * * *` | Every 6 hours, on the hour | | `*/15 * * * *` | Every 15 minutes | | `0 8,12,18 * * *` | At 08:00, 12:00 and 18:00 | | `0 0 1 * *` | Midnight on the 1st of each month | | `@daily` | Midnight every day (also `@hourly`, `@weekly`, `@monthly`) | Two rules trip people up. When you restrict both day of month and day of week, a day matches if **either** matches: `0 9 1 * MON` runs on the 1st and on every Monday. And a cron expression means nothing without a time zone. On Scrape.ly each schedule has an IANA time zone (default UTC) and follows its daylight saving changes, so `0 9 * * *` with `Europe/Berlin` runs at 09:00 Berlin time all year. The [cron reference](https://scrape.ly/docs/api-schedules.md) lists every form, including how runs behave on the night the clocks change. ## Option 1: cron on your own machine If your scraper is a script on a server that never sleeps, one line in `crontab -e` schedules it: ```shell 0 6 * * * cd /home/you/book-tracker && /usr/bin/python3 scraper.py >> scraper.log 2>&1 ``` Use absolute paths (cron runs with a minimal environment), redirect output to a log, and remember that nothing tells you when it starts failing. Everything after "run it" is yours to build: retries, proxies, deduplication, delivery and alerts. Our guide to [building a web scraper](https://scrape.ly/blog/how-to-build-a-web-scraper.md) shows a scraper with SQLite storage that's ready for a cron job. ## Option 2: GitHub Actions For a scraper that lives in a repository, a scheduled workflow runs it on GitHub's machines: ```yaml name: scrape on: schedule: cron: "17 6 * * *" # off the full hour, when load is lower workflow_dispatch: jobs: scrape: runson: ubuntulatest steps: uses: actionscheckout@v4 uses: actionssetuppython@v5 with: pythonversion: "3.13" run: pip install requests beautifulsoup4 run: python scraper.py ``` It's handy for small jobs that already live in a repository. The catches are the ones above (delays, dropped jobs at peak times, workflows disabled in quiet public repos), plus requests that come from data-center IP ranges, which many sites rate-limit or block, and results that end up as files you still have to move somewhere. ## Option 3: Scrape.ly automations On Scrape.ly the scraper is an Actor (a ready-made one from the [Store](https://scrape.ly/store.md) or your own code), and an automation runs it. Open **Automations** in the side menu, click **New automation**, and fill in up to six sections: 1. **When.** **On a schedule** (hourly, daily, weekly, monthly, or a custom cron expression) with a time zone; **Next runs** previews the next three times. Or **After another run finishes**, to chain scrapers. 2. **Run.** Up to 10 Actors, each with a saved input or a preset, all started together. **Run options** sets **Max results**, **Max cost per run**, memory and timeout. 3. **Keep.** **Only when there are results**, **Only new items since the last run** (compared by a key field such as `url`), and **Only rows where…** conditions such as `price` **is less than** `30`. Number comparisons also read prices written as text, such as `€1,299.00`. 4. **AI step (optional).** **Summarize**, **Filter** (decide whether to send at all), **Compare with last run** ("Which prices went down, and by how much?") or **Extract**. It reads up to 200 items per Actor run and is billed per use, like the AI chat. 5. **Send.** Up to 10 deliveries: dashboard, email, Slack, Discord, Google Sheets, Amazon S3 or R2, GitHub, a webhook, or another automation. 6. **Options.** **Skip if the previous run is still going** and **Alert me if it fails** are both on by default. **Run once now** on the automation's page checks the whole chain immediately. Each firing appears in the run history with the runs, the filter result, the AI output and its cost, and each delivery's status. The [automations docs](https://scrape.ly/docs/automations.md) cover every option. ### A worked example: a daily price watch Say you want to know each morning which travel books on [books.toscrape.com](https://books.toscrape.com/) (a sandbox store for practice) cost under £30. The scraper is the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) Actor with a one-line page function that returns every book on the category page: ```json { "startUrls": [{ "url": "https://books.toscrape.com/catalogue/category/books/travel_2/index.html" }], "pageFunction": "async function pageFunction(context) { const $ = context.jQuery; return $('article.product_pod').map((i, el) => ({ title: $(el).find('h3 a').attr('title'), price: $(el).find('p.price_color').text(), inStock: $(el).find('p.availability').text().includes('In stock') })).get(); }", "proxyConfiguration": { "useScrapelyProxy": true } } ``` We ran exactly this input on 26 October 2026, which is what each scheduled firing would do. It took 7.6 seconds and returned 11 books through Scrape.ly's residential proxy: | title | price | inStock | | ------------------------------------------------------------------------------------------------- | ------ | ------- | | It's Only the Himalayas | £45.17 | true | | Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel | £36.94 | true | | The Great Railway Bazaar | £30.54 | true | | The Road to Little Dribbling: Adventures of an American in Britain (Notes From a Small Island #2) | £23.21 | true | | 1,000 Places to See Before You Die | £26.08 | true | Save the input (**Save as new…** in the **Input** menu), then create the automation: **When** daily at 09:00 in your time zone; **Run** that saved input with **Max cost per run** set to $0.05; **Keep** only rows where `price` is less than `30`, plus **Only new items since the last run** with key field `title`; **Send** to Google Sheets and Slack. Of the 11 rows above, the price condition keeps two, and from the second morning on, "only new items" means you hear about a book only when it first drops under £30 (or newly appears below it). The cost: Web Scraper charges $0.000715 per result with platform usage included, so one firing is 11 × $0.000715 = $0.0079 at list price, and a month of daily runs is about $0.24. That's about 5% of the free plan's $5 of monthly credit. The same pattern scales to real retailers with the Store's shop scrapers; our [price scraping guide](https://scrape.ly/blog/price-scraping.md) shows it with Google Shopping and Amazon, and [scraping a website into Google Sheets](https://scrape.ly/blog/scrape-website-to-google-sheets.md) covers the sheet side, including how to build a price history by appending rows. Scrape.ly Store Schedule your own scraper Web Scraper runs a page function on any public site; put it in an automation and get new rows in Sheets or Slack every morning. $0.71 per 1,000 results. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) ## Automate scraping with the API Everything in the editor is also an API call, so you can create automations from code, for example one per client or per product list. With the JavaScript client (`npm install scrapely-client`): ```javascript import { ScrapelyClient } from 'scrapely-client'; const client = new ScrapelyClient({ token: process.env.SCRAPELY_TOKEN, baseUrl: 'https://api.scrape.ly' }); const automation = await client.automations().create({ title: 'Travel books under £30', trigger: { kind: 'schedule', cron: '0 9 * * *', timezone: 'Europe/London' }, steps: [{ kind: 'savedInput', savedInputId: 'SAVED_INPUT_ID', runOptions: { maxTotalChargeUsd: 0.05 }, }], filters: { onlyNewItems: { keyFields: ['title'] }, conditions: [{ field: 'price', op: 'LESS_THAN', value: 30 }], }, ai: { mode: 'summarize', prompt: 'List the new books under £30 with their prices.' }, deliveries: [{ channel: 'slack' }, { channel: 'email' }], }); const run = await client.automation(automation.id).run(); // test it now ``` The key needs the `automations:write` scope. If you only need the timetable (no filters, AI or deliveries), the Schedules API is simpler, and existing schedule clients keep working with it. In Python (`pip install scrapely-client-python`): ```python import os from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") client.schedules().create( name="travel-books-daily", cron_expression="0 9 * * *", timezone="Europe/London", is_enabled=True, is_exclusive=True, actions=[{"type": "RUN_ACTOR_TASK", "actorTaskId": "SAVED_INPUT_ID"}], ) ``` Every schedule is an automation, so it shows up in the same list, and the CLI reads them too: `scrapely api GET /v2/schedules` and `scrapely api GET /v2/schedules//log`, which lists each firing as "Successfully started 1 run(s)" or the error. ## React to runs with webhooks Schedules start runs; webhooks tell your systems when a run finished. A webhook sends a `POST` to your URL on events such as `ACTOR.RUN.SUCCEEDED` or `ACTOR.RUN.FAILED`, with the run's ID, status, timing and `defaultDatasetId`, so your endpoint can fetch the results the moment they exist: ```javascript await client.webhooks().create({ eventTypes: ['ACTOR.RUN.SUCCEEDED', 'ACTOR.RUN.FAILED'], condition: { actorId: 'ACTOR_ID' }, requestUrl: 'https://example.com/hooks/scrapely?token=LONG_RANDOM_SECRET', }); ``` Failed deliveries (network errors, 429, 5xx) are retried, and every request carries an `X-Scrapely-Webhook-Dispatch-Id` header that stays the same across retries, so your endpoint can ignore duplicates. Put a secret in the URL or a custom header and check it, since you can't verify the platform's signature yourself. A webhook delivery on an automation goes further: its `POST` contains the kept items themselves (up to 100) and the AI text, so a small endpoint can write them straight into your database. Without code, the **Start another actor after this one** recipe chains scrapers: crawl a list, then enrich each result with a second Actor. ## What to watch out for - **No cap means no cap.** A scheduled run has no spending limit unless you set **Max cost per run** (or `maxTotalChargeUsd`). Set one on every automation, especially broad searches. - **Overlapping runs.** A slow run plus a frequent schedule means two copies scraping at once. Keep **Skip if the previous run is still going** on. - **Missed times.** Scheduled runs start up to about a minute late, and a time missed during an outage is skipped, not caught up. Don't build anything that assumes exact-minute timing. - **Scraping too often.** Hourly runs of a page that changes daily cost 24 times as much and load the site for nothing; our Travel example would go from about $0.24 to $5.66 a month. Match the schedule to how often the data changes, and see our [web scraping best practices](https://scrape.ly/blog/web-scraping-best-practices.md) for polite rates. - **Silent breakage.** A redesign can turn a scraper's output into empty rows. **Only when there are results** stops empty deliveries, and **Alert me if it fails** emails you on failures. - **Plan limits.** The free plan includes 5 schedules; automations and schedules share that limit. ## FAQ ### How do I schedule a web scraper to run daily? On your own machine, add a crontab line such as `0 6 * * *` that runs the script. In the cloud, create a Scrape.ly automation with **On a schedule** → **Daily**, pick the time and time zone, and add your Actor and saved input under **Run**. ### Can I automate web scraping without coding? Yes. Pick a ready-made scraper from the [Scrape.ly Store](https://scrape.ly/store.md), fill in its input form, and create an automation with a schedule, filters and deliveries to Google Sheets, Slack or email, all in the dashboard. Our list of [no-code web scrapers](https://scrape.ly/blog/no-code-web-scrapers.md) compares other options. ### How often should I run a scraper? As often as the data changes and you act on it. Prices and stock: daily or a few times a day. Reviews and job listings: daily. Rankings and directories: weekly. More frequent runs cost more and load the site without giving you new information. ### How do I get scraped data into Google Sheets automatically? Add a **Google Sheets** delivery to the automation and share the sheet with Scrape.ly first. Each run appends the kept items as rows; with **Only new items since the last run** the sheet grows only by new entries. ### How do I get notified only when something changes? Use **Only new items since the last run** with a key field, or the AI step's **Compare with last run** mode, which leads with what changed. Combine either with **Only when there are results** so quiet days send nothing. ### Is automated web scraping legal? Automation doesn't change the legal picture; what you collect and how does. Stick to public data, respect site terms and rate limits, and handle personal data under privacy law. This isn't legal advice. Scrape.ly Store Actors only collect publicly available data shown to any visitor. About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a monitor showing the green Node.js hexagon logo beside a small web page, above lines of code, and drops it into a crate](https://scrape.ly/media/node-js-web-scraping.png?w=750)](https://scrape.ly/blog/node-js-web-scraping.md) ### [Node.js Web Scraping: Fetch, Cheerio, Playwright, Puppeteer and Crawlee](https://scrape.ly/blog/node-js-web-scraping.md) [A hands-on guide to Node.js web scraping: fetch and axios with Cheerio, pagination with retries, Playwright and Puppeteer for JavaScript pages, Crawlee for real crawls, and deploying the scraper to the cloud from a Scrape.ly template. All code tested on 26 October 2026.](https://scrape.ly/blog/node-js-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands between two matching arcade cabinets, one with the teal Scrapy logo and one with a steaming soup bowl, thinking of the Scrape.ly mark next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/scrapy-vs-beautifulsoup.png?w=750)](https://scrape.ly/blog/scrapy-vs-beautifulsoup.md) ### [Scrapy vs BeautifulSoup: Which Python Scraping Tool to Use (Tested)](https://scrape.ly/blog/scrapy-vs-beautifulsoup.md) [Scrapy vs BeautifulSoup, tested on the same 1,000-book job: what each tool is, measured run times, retries, concurrency, exports and JavaScript pages, when to pick which, and how to run a Scrapy project in the cloud with one command.](https://scrape.ly/blog/scrapy-vs-beautifulsoup.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from the one orange-lit globe in a ring of four blue-and-green globes turning round a small server, amber arrowheads marking the rotation, towards a web page with an image block](https://scrape.ly/media/rotating-proxies-for-web-scraping.png?w=750)](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) ### [Rotating Proxies for Web Scraping: How Rotation Works, with Tested Code](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) [Rotating proxies for web scraping spread your requests over many IP addresses. Per-request rotation vs sticky sessions, residential vs datacenter pools, tested Python and Crawlee code, and what rotation costs per GB on Scrape.ly and Evomi.](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.