12 Web Scraping Best Practices for Reliable, Responsible Scrapers

Web scraping best practices are the habits that separate a scraper that runs for a year from one that gets blocked in a week: respect the site's rules, keep your load low, handle errors, and check your data. Most of them take a few lines of code, and they also make your scraper cheaper and your data more trustworthy.

Below are 12 web scraping best practices, each with what to do, a tested Python snippet where code helps, and the setting that does it for you on Scrape.ly. All code was run on 23 October 2026 against books.toscrape.com, a sandbox site built for scraping practice.

12 web scraping best practices at a glance

1. Respect robots.txt and the site's terms

Robots.txt tells automated clients which paths a site would rather they didn't visit, and sometimes how fast to go (Crawl-delay). It isn't law in itself, but honouring it is the norm for good-faith scrapers, and it keeps you out of login pages, carts and search endpoints you didn't need anyway. Terms of service matter most when you've accepted them, for example by creating an account and logging in. Our guide is web scraping legal? explains why.

Python ships a robots.txt parser; the snippet under practice 2 uses it. On Scrape.ly, the Web Scraper and the Website Content Crawler both have a respectRobotsTxtFile switch that skips disallowed links they discover.

2. Limit your request rate

The fastest way to get blocked, and to cause real harm, is to hammer a site with parallel requests. Keep one host at a modest pace, and slow down further if responses get slower. This function combines practices 1, 2, 3 and 7: it checks robots.txt, honours Crawl-delay, waits at least a second between requests to the same host, identifies itself and retries with backoff:

import time
from urllib.parse import urljoin, urlparse
from urllib.robotparser import RobotFileParser

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

USER_AGENT = "acme-price-monitor/1.0 (+https://example.com/bot; bot@example.com)"

session = requests.Session()
session.headers["User-Agent"] = USER_AGENT
retry = Retry(
    total=4,
    backoff_factor=2,  # waits 2 s, 4 s, 8 s between retries
    status_forcelist=[429, 500, 502, 503, 504],
    respect_retry_after_header=True,
)
session.mount("https://", HTTPAdapter(max_retries=retry))

_robots: dict[str, RobotFileParser] = {}
_last_request: dict[str, float] = {}

def polite_get(url: str, min_delay: float = 1.0) -> requests.Response | None:
    host = urlparse(url).netloc
    if host not in _robots:
        parser = RobotFileParser(urljoin(url, "/robots.txt"))
        parser.read()
        _robots[host] = parser
    if not _robots[host].can_fetch(USER_AGENT, url):
        return None  # disallowed: skip it
    delay = _robots[host].crawl_delay(USER_AGENT) or min_delay
    wait = _last_request.get(host, 0) + delay - time.monotonic()
    if wait > 0:
        time.sleep(wait)
    _last_request[host] = time.monotonic()
    return session.get(url, timeout=15)

Two calls to books.toscrape.com returned 200 a second apart. (That site has no robots.txt; the parser treats a missing file as "everything allowed".)

On Scrape.ly, cap parallel requests with maxConcurrency on the Web Scraper (default 50) and the Website Content Crawler. For a small site, 2 to 5 is plenty.

3. Identify your scraper

A User-Agent such as acme-price-monitor/1.0 (+https://example.com/bot) tells a site owner who you are and how to reach you, so a problem becomes an email instead of a ban. Pretending to be a person while sending thousands of requests is how scrapers end up blocked wholesale. Set it once on the session, as above. In Scrape.ly Actors that accept headers, such as the Website Content Crawler's customHttpHeaders, pass the same header there; in your own Actors, set it on the crawler.

4. Cache, and skip pages that haven't changed

Don't download the same page twice when once will do. Save raw responses during development so you can rework your parser without new requests. In production, ask the server whether the page changed. Many servers send an ETag; send it back and an unchanged page costs a tiny 304 Not Modified instead of a full download:

import requests

session = requests.Session()
etags: dict[str, str] = {}

def get_if_changed(url: str) -> bytes | None:
    headers = {"If-None-Match": etags[url]} if url in etags else {}
    response = session.get(url, headers=headers, timeout=15)
    if response.status_code == 304:
        return None  # unchanged since last time: nothing to parse
    if etag := response.headers.get("ETag"):
        etags[url] = etag
    return response.content

On books.toscrape.com the first call returned the 51 KB page and the second returned None, because the server answered 304. On Scrape.ly, keep the etags map between scheduled runs in a named key-value store: named storages outlive runs, while each run's default storage expires with your plan's retention.

5. Use the official API when it's enough

If a site offers an API with the fields you need, at a volume and price you accept, use it: it's stable, documented and agreed. Scraping is the right tool when the API is missing, lacks fields, caps volume or costs more per record than you can justify. Our web scraping vs API guide compares real prices. Scrape.ly's API pages give you the third option: a maintained scraper for platforms such as Google Maps, Amazon or Reddit that you call like any REST API.

6. Rotate proxies responsibly

Proxies spread requests across IP addresses so one address doesn't hit a site's rate limit. They're a tool for reliability, not a licence to send more traffic than the site can handle: keep your total rate low (practice 2) whatever your IP count.

• Start without proxies or with datacenter IPs on sites that don't block them; they're cheaper.
• Move to residential IPs for sites that block datacenter ranges, and pick a country when content is localised.
• Rotate per session rather than per request when a site ties carts, cookies or pagination to an IP.

On Scrape.ly, every run can use the managed proxies without a separate account: residential from $4 per GB on the free plan down to $2 on Business, datacenter from $1 down to $0.75. The Web Scraper's proxyRotation setting offers RECOMMENDED, PER_REQUEST and UNTIL_FAILURE. Our pillar on web scraping without getting blocked covers fingerprints, headers and browsers, and Scrape.ly anti-blocking shows what the platform handles for you.

7. Handle errors and retry with backoff

Networks fail, servers return 503 under load, and rate limits answer 429. Retry those with growing pauses (the Retry object above waits 2, 4 and 8 seconds and honours Retry-After), but don't retry 404s, and stop after a few attempts. Log every failed URL so you can rerun just those.

The Web Scraper retries transport failures and HTTP 401, 403, 408, 429 and 5xx responses up to maxRequestRetries times (default 3). A polite configuration for a small site looks like this:

{
  "startUrls": [{ "url": "https://books.toscrape.com/" }],
  "pageFunction": "async function pageFunction(context) { return { url: context.request.url, title: context.jQuery('title').text() }; }",
  "respectRobotsTxtFile": true,
  "maxConcurrency": 2,
  "maxRequestRetries": 3,
  "maxPagesPerCrawl": 500,
  "proxyRotation": "UNTIL_FAILURE",
  "proxyConfiguration": { "useScrapelyProxy": true }
}

Errors can also cost money: a pagination bug that loops forever keeps fetching. Give every run a hard ceiling. With the Python client (pip install scrapely-client-python):

import os
from decimal import Decimal

from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

run = client.actor("querystone/web-scraper").call(
    run_input={
        "startUrls": [{"url": "https://books.toscrape.com/"}],
        "pageFunction": "async function pageFunction(context) { return { url: context.request.url }; }",
        "maxPagesPerCrawl": 500,
        "proxyConfiguration": {"useScrapelyProxy": True},
    },
    max_items=500,
    max_total_charge_usd=Decimal("0.50"),
)
print(run.status)  # SUCCEEDED, FAILED, ABORTED or TIMED-OUT

When a run reaches its maximum charge, Scrape.ly stops it. With a prepaid balance on top, there's no way to get a surprise bill.

8. Validate every record

A scraper that "succeeds" with empty prices is worse than one that fails, because nobody notices. Check each record for required fields, types and plausible ranges before you store it:

REQUIRED = {"title": str, "price_gbp": float, "rating": int, "url": str}

def validate(book: dict) -> list[str]:
    problems = [f"missing {k}" for k in REQUIRED if book.get(k) in (None, "")]
    problems += [f"{k} is not {t.__name__}" for k, t in REQUIRED.items()
                 if k in book and book[k] not in (None, "") and not isinstance(book[k], t)]
    if isinstance(book.get("price_gbp"), float) and not 0 < book["price_gbp"] < 1000:
        problems.append(f"price out of range: {book['price_gbp']}")
    return problems

Fed a good record and a broken one (empty title, price 0.0), it flagged only the second with ['missing title', 'price out of range: 0.0']. Quarantine bad records instead of dropping them silently, so you can see what changed. The same check works on a Scrape.ly dataset: iterate the items with the client and validate each one.

9. Monitor for site changes

Sites change their HTML without notice. Turn validation into an alarm: if more than a few percent of records fail, or the record count drops sharply against the last run, stop and alert a person.

bad = {book["url"]: problems for book in books if (problems := validate(book))}
if len(bad) > 0.05 * len(books):
    raise RuntimeError(f"{len(bad)} of {len(books)} records failed validation; the layout may have changed")

On Scrape.ly, scheduled automations have Alert me if it fails on by default (email and dashboard), webhooks can call your endpoint on ACTOR.RUN.FAILED, and an automation can start a second Actor after each run to run checks like the one above. With Store Actors, the maintainer fixes selectors when the target site changes, which removes most of this work for big platforms.

10. Store as little personal data as you can

Reviews, comments, profiles and directories contain personal data, and privacy law such as the GDPR applies to it even when it's public. Decide before you scrape which fields you need, drop the rest at collection time, and set a deletion date.

Some Actors make this a switch. The Google Maps Reviews Scraper has a personalData input; set it to false and the reviewer's name, ID, profile URL and photo are dropped, while the review text, rating and date stay. Each run's default dataset also expires under your plan's retention (2 days on the free plan, apart from your latest 10 runs; 31, 60 or 90 days on paid plans), so personal data doesn't linger unless you copy it somewhere permanent.

11. Schedule off-peak

Run big jobs when the target site is quiet: overnight in its own time zone, not yours. It lowers your impact on real users and often makes responses faster. In cron, 0 3 * * * means 03:00 every day.

On Scrape.ly, create a schedule in Automations: hourly, daily, weekly, monthly or a custom cron expression, with a time zone you choose. Schedules follow daylight saving in that time zone, so a 3 a.m. Berlin run stays at 3 a.m. Berlin time all year. Spread several jobs a few minutes apart rather than starting them all on the hour.

12. Document what you collect and why

Write down, for every scraper: the source and the pages it visits, the fields it saves, why you need them, how often it runs, where the data goes and when it's deleted. It takes ten minutes, it's what a privacy review or a site owner's email will ask for, and it makes the scraper maintainable by someone other than you.

On Scrape.ly, save each configuration as a saved input (a task in the API) with a clear name. It keeps the input and run options, including the maximum cost, together, so the scraper that runs is the one you documented. Keep run IDs in your logs; each run keeps its input, log and dataset for its retention period.

FAQ

What are the most important web scraping best practices?

Respect robots.txt and terms, keep your request rate low, retry errors with backoff, and validate your data on every run. Those four prevent most blocks, most legal friction and most silent data failures.

How do I avoid getting blocked while scraping?

Slow down first: fewer parallel requests, pauses between them and backoff on 429. Then use realistic headers, session-consistent proxies and a real browser only where needed. Our guide to web scraping without getting blocked covers each layer.

Is it OK to ignore robots.txt?

Robots.txt isn't a law, but ignoring it removes one of your best arguments that you scraped in good faith, and it usually points at pages you don't need. Honour it unless you have a specific, documented reason and permission from the site.

How fast should a web scraper be?

There's no universal number. One request per second per host is a common, conservative starting point for small sites; large platforms tolerate more. Watch response times and error rates, and slow down when either rises.

How do I keep a scraper running when the website changes?

Validate every run, alert on rising failure rates, and keep selectors in one place so fixes are quick. For big platforms, use a maintained Actor from the Scrape.ly Store, where the maintainer updates the scraper and your input stays the same.