How to Build a Web Scraper: 8 Steps From First Request to the Cloud

Here's how to build a web scraper in one sentence: decide which fields you need, inspect the page to find where they live, download the HTML, parse the fields out with selectors, follow pagination, and store the results somewhere you can query. Then schedule it and harden it, because a scraper that ran once is a script; a scraper that runs every day is a system.

This guide builds one end to end in Python: a book tracker that crawls a category of books.toscrape.com (a sandbox site made for practice), visits every product page, validates each record, and keeps a price history in SQLite. Every snippet was run on 26 October 2026. At the end we compare building your own with running a ready-made scraper on Scrape.ly, and show how to deploy your code to the cloud if you do build it.

How to build a web scraper: the 8 steps at a glance

Step 1: Define the data you need

Write down one finished record before you write any code. For a price tracker, one record per book:

{"upc": "e00eb4fd7b871a48", "title": "Sharp Objects", "price": 47.82, "stock": 20, "url": "https://books.toscrape.com/catalogue/sharp-objects_997/index.html"}

Three decisions hide in that line. A stable key (upc here, a product ID or URL elsewhere) lets you update records instead of duplicating them. Clean types (price as a number, stock as an integer) save you from cleaning the data later in a spreadsheet. And only the fields you need: every extra field is one more selector that can break.

Step 2: Inspect the site

Open the page in your browser, right-click the data and choose Inspect. On a category page of books.toscrape.com, each book is an <article class="product_pod"> with a link to its product page, and a li.next a link leads to the next page. On the product page, the UPC, price and stock sit in a table (table.table-striped), one <th>/<td> pair per row.

Before you settle on HTML, run three checks:

• View the page source (Ctrl+U or Cmd+Option+U). If the data is there, plain HTTP requests work. If the source is mostly scripts and the data appears only in DevTools, the page is rendered by JavaScript, and you'll need a headless browser or the API behind it.
• Look for a JSON API. In DevTools, open Network, filter by Fetch/XHR and reload. Structured JSON is easier to parse and survives redesigns.
• Read robots.txt and the terms. books.toscrape.com has no robots.txt (it returns 404), so nothing is disallowed. On real sites, respect what it says and check the terms of use.

Step 3: Fetch pages politely

Use one Session for the whole run so connections are reused, send an honest User-Agent, set a timeout, retry temporary failures with backoff, and pause between requests:

import sqlite3
import sys
import time
from datetime import datetime, timezone
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

START_URL = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
DELAY = 1.0  # seconds between requests


def make_session() -> requests.Session:
    session = requests.Session()
    session.headers["User-Agent"] = "book-tracker/1.0 (you@example.com)"
    retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
    session.mount("https://", HTTPAdapter(max_retries=retry))
    return session


def fetch(session: requests.Session, url: str) -> BeautifulSoup:
    response = session.get(url, timeout=15)
    response.raise_for_status()
    response.encoding = "utf-8"  # the site sends no charset
    time.sleep(DELAY)
    return BeautifulSoup(response.text, "html.parser")

raise_for_status() turns a 404 or 500 into an exception instead of letting you parse an error page. The encoding line matters on this site: it sends Content-Type: text/html without a charset, so requests would decode the page as ISO-8859-1 and every £ would come out as £.

Step 4: Parse and validate each record

Selectors pull the fields out; validation catches the day they stop matching. The product table becomes a dictionary first, which is sturdier than counting cells:

def parse_book(soup: BeautifulSoup, url: str) -> dict:
    table = {row.th.text: row.td.text for row in soup.select("table.table-striped tr")}
    book = {
        "upc": table["UPC"],
        "title": soup.select_one("div.product_main h1").text.strip(),
        "price": float(table["Price (incl. tax)"].lstrip("£")),
        "stock": int("".join(c for c in table["Availability"] if c.isdigit()) or 0),
        "url": url,
    }
    if not book["title"] or book["price"] <= 0:
        raise ValueError(f"suspicious record at {url}: {book}")
    return book

Reading "UPC" by its header label means a reordered table doesn't silently shift every column. The availability text is "In stock (20 available)", so the code keeps the digits. And if a redesign empties the title or zeroes the price, the record is rejected with a message instead of quietly landing in your database.

Step 5: Follow pagination and detail links

A category spans several pages. Collect the product links from each listing page and follow next until there isn't one:

def parse_listing(soup: BeautifulSoup, page_url: str) -> tuple[list[str], str | None]:
    links = [urljoin(page_url, a["href"]) for a in soup.select("article.product_pod h3 a")]
    next_link = soup.select_one("li.next a")
    return links, urljoin(page_url, next_link["href"]) if next_link else None

Links on this site are relative (../../../sharp-objects_997/index.html), and urljoin resolves them against the current page. Gluing strings together is the most common pagination bug there is.

Step 6: Store the results with history

CSV is fine for a one-off. For a scraper you'll run again, use a database: you can update records by key, and keep every price you've ever seen. SQLite ships with Python and needs no server:

def save(db: sqlite3.Connection, books: list[dict]) -> int:
    now = datetime.now(timezone.utc).isoformat(timespec="seconds")
    changed = 0
    for b in books:
        row = db.execute("SELECT price FROM books WHERE upc = ?", (b["upc"],)).fetchone()
        if row is None or row[0] != b["price"]:
            changed += 1
            db.execute("INSERT INTO prices (upc, price, seen_at) VALUES (?, ?, ?)", (b["upc"], b["price"], now))
        db.execute(
            """INSERT INTO books (upc, title, price, stock, url, updated_at)
               VALUES (:upc, :title, :price, :stock, :url, :updated_at)
               ON CONFLICT(upc) DO UPDATE SET price = excluded.price, stock = excluded.stock,
                                              updated_at = excluded.updated_at""",
            {**b, "updated_at": now},
        )
    db.commit()
    return changed


def main(max_books: int) -> None:
    db = sqlite3.connect("books.db")
    db.execute("CREATE TABLE IF NOT EXISTS books (upc TEXT PRIMARY KEY, title TEXT, price REAL, "
               "stock INTEGER, url TEXT, updated_at TEXT)")
    db.execute("CREATE TABLE IF NOT EXISTS prices (upc TEXT, price REAL, seen_at TEXT)")

    session = make_session()
    url, book_urls = START_URL, []
    while url:
        links, url = parse_listing(fetch(session, url), url)
        book_urls += links

    books = []
    for book_url in book_urls[:max_books]:
        try:
            books.append(parse_book(fetch(session, book_url), book_url))
        except (requests.RequestException, KeyError, ValueError) as err:
            print("skipped", book_url, err, file=sys.stderr)

    changed = save(db, books)
    print(f"{len(book_urls)} books listed, {len(books)} scraped, {changed} new or changed prices")
    print(books[0])


if __name__ == "__main__":
    main(int(sys.argv[1]) if len(sys.argv) > 1 else 1000)

Save all of the steps as scraper.py and run it with pip install requests beautifulsoup4 and python scraper.py. Our first run made 34 requests (2 listing pages, 32 product pages) in 70 seconds, almost all of it the polite one-second pauses:

32 books listed, 32 scraped, 32 new or changed prices
{'upc': 'e00eb4fd7b871a48', 'title': 'Sharp Objects', 'price': 47.82, 'stock': 20, 'url': 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html'}

The second run printed 0 new or changed prices: the books table was updated in place and the prices table only grows when a price moves. That's the core of every price tracker. One bad page is logged and skipped instead of killing the run, which matters more the longer a crawl gets.

Step 7: Schedule the scraper

A tracker is only useful if it runs without you. On a Linux or macOS machine that's always on, one crontab line (crontab -e) runs it every morning at 6:00:

0 6 * * * cd /home/you/book-tracker && /usr/bin/python3 scraper.py >> scraper.log 2>&1

The catch is the machine: a laptop sleeps, a server needs patching, and nobody notices when the job starts failing. A cloud schedule fixes that. On Scrape.ly, a schedule is an automation with a cron expression and a time zone, and it can filter the results, add an AI summary and send them to Google Sheets, Slack or a webhook, with an email alert when a run fails. Our guide to scheduling and automating web scraping walks through the setup and cron syntax.

Step 8: Scale and maintain it

What breaks first when a scraper grows:

• Speed. One request at a time is polite but slow. Fetch several pages at once with a concurrency limit, or move to a framework that does it for you; see Scrapy vs BeautifulSoup for measured timings.
• Blocking. Real sites rate-limit by IP and fingerprint. You'll need rotating residential proxies and, for protected sites, a browser that doesn't look automated. Our guide to web scraping without getting blocked covers each layer.
• JavaScript. Pages that render with scripts need Playwright or Puppeteer, which cost far more memory per page than an HTTP request.
• Maintenance. Sites redesign. Validation (step 4) tells you when; somebody still has to fix the selectors.

The same steps apply in any language. For JavaScript, see Node.js web scraping; for the JVM, Java web scraping.

Build vs use a ready-made scraper

Steps 1 to 6 took an afternoon. Steps 7 and 8 are where the hours go, every month, for as long as the scraper lives. For most targets, a ready-made scraper on Scrape.ly is the better deal:

The Scrape.ly Store has 570+ Actors for sites like Amazon, Google Maps, LinkedIn and Zillow. For a site without one, the Web Scraper Actor does steps 3 to 6 for you: you give it start URLs, a link selector and a small JavaScript page function, and it handles fetching, retries, the proxy and storage. Here is the same Mystery-category job from the Python client (pip install scrapely-client-python):

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

page_function = """async function pageFunction(context) {
    const $ = context.jQuery;
    if (!$('div.product_main').length) return null; // listing page: only follow links
    return {
        url: context.request.url,
        title: $('div.product_main h1').text().trim(),
        price: $('div.product_main p.price_color').text().trim(),
        inStock: $('div.product_main p.availability').text().includes('In stock'),
        upc: $('table.table-striped td').first().text().trim(),
    };
}"""

run = client.actor("querystone/web-scraper").call(
    run_input={
        "startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"}],
        "linkSelector": "article.product_pod h3 a",
        "globs": [{"glob": "https://books.toscrape.com/catalogue/*/index.html"}],
        "pageFunction": page_function,
        "maxPagesPerCrawl": 6,
        "maxResultsPerCrawl": 5,
        "maxConcurrency": 2,
        "respectRobotsTxtFile": True,
        "proxyConfiguration": {"useScrapelyProxy": True},
    }
)
print(run.status)  # SUCCEEDED, FAILED, ABORTED or TIMED-OUT
for item in client.dataset(run.default_dataset_id).iterate_items():
    if item.get("title"):
        print(item["upc"], item["title"], item["price"])

We ran this input on 23 October 2026 (a reused run): 5 pages fetched through Scrape.ly's residential proxy in 12.4 seconds, 4 book records saved, with the same UPCs our own scraper found:

At today's list price of $0.000715 per result, platform usage included, those 4 records cost 4 × $0.000715 = $0.0029, and the whole 32-book category about $0.023. The free plan's $5 of monthly credit covers roughly 7,000 results. Raise maxResultsPerCrawl, set a maximum charge, and put it on a schedule.

Or deploy your own scraper as an Actor

If you'd rather keep your code, run it on Scrape.ly as your own Actor. Wrap main() with the Python SDK so results go to the run's dataset:

import asyncio

from scrapely import Actor

from scraper import START_URL, fetch, make_session, parse_book, parse_listing


async def main() -> None:
    async with Actor:
        actor_input = await Actor.get_input() or {}
        max_books = actor_input.get("maxBooks", 20)
        session = make_session()
        url, book_urls = actor_input.get("startUrl", START_URL), []
        while url:
            links, url = parse_listing(fetch(session, url), url)
            book_urls += links
        for book_url in book_urls[:max_books]:
            await Actor.push_data(parse_book(fetch(session, book_url), book_url))


if __name__ == "__main__":
    asyncio.run(main())

Save that as main.py next to scraper.py, list requests and beautifulsoup4 in requirements.txt (not the SDK: it's preinstalled in the base image), and add .actor/scrapely.json with {"actorSpecification": 1, "name": "book-tracker", "version": "0.1"}. Then, with the CLI (npm install -g scrapely-cli, scrapely login):

scrapely run -i '{"maxBooks": 3}'
scrapely push
scrapely call -i '{"maxBooks": 20}' -o

scrapely run used our project's .venv and saved 3 books to ./storage/datasets/default locally. Without a Dockerfile, the platform build generates one on the scrapely/actor-python image, installs requirements.txt and runs main.py. Already on Scrapy? Run scrapely init in the project folder and it writes the wrapper for you. The Python deploy guide has the details, and for JavaScript, scrapely create gives you a working Crawlee project from a template in one command.

What to watch out for

• Getting blocked. Delays and retries keep you polite; they don't make you invisible. On sites with bot protection, plan for residential proxies and a real browser, or use an Actor that already handles both.
• Silent breakage. A scraper that writes empty fields is worse than one that crashes. Validate records and alert on runs that return far fewer items than usual.
• Duplicates. Without a stable key, every run appends the same rows again. Upsert by ID, as in step 6.
• Personal data and terms. Scrape public pages, collect only what you need, respect robots.txt and the site's terms, and treat personal data under privacy law. Scrape.ly Store Actors only collect publicly available data shown to any visitor.

FAQ

How long does it take to build a web scraper?

A working script for one simple site takes an hour or two, like steps 1 to 6 here. Making it reliable (scheduling, monitoring, proxies, fixing selectors after redesigns) is ongoing work. A ready-made Actor skips most of that.

What programming language is best for building a web scraper?

Python is the most common choice thanks to requests, BeautifulSoup and Scrapy. JavaScript with Cheerio, Playwright and Crawlee is equally capable, and Java works well inside JVM services. The steps are the same in every language.

Can I build a web scraper without coding?

Yes. Run a ready-made scraper from the Scrape.ly Store with a form instead of code, then download the results as CSV, JSON or Excel. Our guide to no-code web scrapers compares the options.

How do I make my web scraper run automatically?

Use cron or Task Scheduler on a machine that's always on, or a cloud schedule. On Scrape.ly, an automation runs your Actor on a cron expression in your time zone and can send the results to Google Sheets, Slack, email or a webhook.

Is it legal to build a web scraper?

Building one is legal; what you scrape and how matters. Public data is generally fair game in many places, but respect site terms, don't overload servers and handle personal data under privacy law. This isn't legal advice; see is web scraping legal.