--- title: "12 Web Scraping Best Practices, With Code | Scrape.ly" description: "12 web scraping best practices with tested Python code: robots.txt, rate limits, caching, retries, validation, monitoring, personal data and scheduling." url: https://scrape.ly/blog/web-scraping-best-practices markdown: https://scrape.ly/blog/web-scraping-best-practices.md --- [Guides](https://scrape.ly/blog/category/guides.md) # 12 Web Scraping Best Practices for Reliable, Responsible Scrapers Twelve web scraping best practices that keep scrapers polite, unblocked and accurate: robots.txt, rate limits, caching, retries, validation, monitoring, personal data and more. Each with tested Python code and how to do it on Scrape.ly. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 9 min read ![Pixel-art robot with an orange >\_ on its terminal head reaches for the last empty box on a clipboard checklist whose first four items carry orange ticks, with a web page standing behind it](https://scrape.ly/media/web-scraping-best-practices.png?w=1080) Web scraping best practices are the habits that separate a scraper that runs for a year from one that gets blocked in a week: respect the site's rules, keep your load low, handle errors, and check your data. Most of them take a few lines of code, and they also make your scraper cheaper and your data more trustworthy. Below are 12 web scraping best practices, each with what to do, a tested Python snippet where code helps, and the setting that does it for you on Scrape.ly. All code was run on 23 October 2026 against [books.toscrape.com](https://books.toscrape.com/), a sandbox site built for scraping practice. ## 12 web scraping best practices at a glance | # | Practice | In your own code | On Scrape.ly | | -- | ------------------------------------- | ------------------------ | ------------------------------------------- | | 1 | Respect robots.txt and terms | `RobotFileParser` | `respectRobotsTxtFile` | | 2 | Limit your request rate | A delay per host | `maxConcurrency` | | 3 | Identify your scraper | A descriptive User-Agent | Custom headers | | 4 | Cache and skip unchanged pages | ETag / `If-None-Match` | Named key-value stores | | 5 | Use the official API when it's enough | Check before you build | `/api` pages and Store Actors | | 6 | Rotate proxies responsibly | Your own pool | Managed residential and datacenter proxies | | 7 | Handle errors and retry with backoff | `urllib3` `Retry` | `maxRequestRetries`, maximum charge per run | | 8 | Validate every record | Schema checks | Checks on each dataset | | 9 | Monitor for site changes | Alert on failure rates | Failure alerts and webhooks | | 10 | Store minimal personal data | Drop unneeded fields | `personalData: false`, retention | | 11 | Schedule off-peak | cron | Schedules with time zones | | 12 | Document the scraper | A README | Saved inputs | ## 1. Respect robots.txt and the site's terms Robots.txt tells automated clients which paths a site would rather they didn't visit, and sometimes how fast to go (`Crawl-delay`). It isn't law in itself, but honouring it is the norm for good-faith scrapers, and it keeps you out of login pages, carts and search endpoints you didn't need anyway. Terms of service matter most when you've accepted them, for example by creating an account and logging in. Our guide [is web scraping legal?](https://scrape.ly/blog/is-web-scraping-legal.md) explains why. Python ships a robots.txt parser; the snippet under practice 2 uses it. On Scrape.ly, the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) and the Website Content Crawler both have a `respectRobotsTxtFile` switch that skips disallowed links they discover. ## 2. Limit your request rate The fastest way to get blocked, and to cause real harm, is to hammer a site with parallel requests. Keep one host at a modest pace, and slow down further if responses get slower. This function combines practices 1, 2, 3 and 7: it checks robots.txt, honours `Crawl-delay`, waits at least a second between requests to the same host, identifies itself and retries with backoff: ```python import time from urllib.parse import urljoin, urlparse from urllib.robotparser import RobotFileParser import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry USER_AGENT = "acme-price-monitor/1.0 (+https://example.com/bot; bot@example.com)" session = requests.Session() session.headers["User-Agent"] = USER_AGENT retry = Retry( total=4, backoff_factor=2, # waits 2 s, 4 s, 8 s between retries status_forcelist=[429, 500, 502, 503, 504], respect_retry_after_header=True, ) session.mount("https://", HTTPAdapter(max_retries=retry)) _robots: dict[str, RobotFileParser] = {} _last_request: dict[str, float] = {} def polite_get(url: str, min_delay: float = 1.0) -> requests.Response None: host = urlparse(url).netloc if host not in _robots: parser = RobotFileParser(urljoin(url, "/robots.txt")) parser.read() _robots[host] = parser if not _robots[host].can_fetch(USER_AGENT, url): return None # disallowed: skip it delay = _robots[host].crawl_delay(USER_AGENT) or min_delay wait = _last_request.get(host, 0) + delay time.monotonic() if wait > 0: time.sleep(wait) _last_request[host] = time.monotonic() return session.get(url, timeout=15) ``` Two calls to books.toscrape.com returned 200 a second apart. (That site has no robots.txt; the parser treats a missing file as "everything allowed".) On Scrape.ly, cap parallel requests with `maxConcurrency` on the Web Scraper (default 50) and the Website Content Crawler. For a small site, 2 to 5 is plenty. ## 3. Identify your scraper A User-Agent such as `acme-price-monitor/1.0 (+https://example.com/bot)` tells a site owner who you are and how to reach you, so a problem becomes an email instead of a ban. Pretending to be a person while sending thousands of requests is how scrapers end up blocked wholesale. Set it once on the session, as above. In Scrape.ly Actors that accept headers, such as the Website Content Crawler's `customHttpHeaders`, pass the same header there; in your own Actors, set it on the crawler. ## 4. Cache, and skip pages that haven't changed Don't download the same page twice when once will do. Save raw responses during development so you can rework your parser without new requests. In production, ask the server whether the page changed. Many servers send an `ETag`; send it back and an unchanged page costs a tiny `304 Not Modified` instead of a full download: ```python import requests session = requests.Session() etags: dict[str, str] = {} def get_if_changed(url: str) -> bytes None: headers = {"If-None-Match": etags[url]} if url in etags else {} response = session.get(url, headers=headers, timeout=15) if response.status_code == 304: return None # unchanged since last time: nothing to parse if etag := response.headers.get("ETag"): etags[url] = etag return response.content ``` On books.toscrape.com the first call returned the 51 KB page and the second returned `None`, because the server answered 304. On Scrape.ly, keep the `etags` map between scheduled runs in a named key-value store: named storages outlive runs, while each run's default storage expires with your plan's retention. ## 5. Use the official API when it's enough If a site offers an API with the fields you need, at a volume and price you accept, use it: it's stable, documented and agreed. Scraping is the right tool when the API is missing, lacks fields, caps volume or costs more per record than you can justify. Our [web scraping vs API](https://scrape.ly/blog/web-scraping-vs-api.md) guide compares real prices. Scrape.ly's [API pages](https://scrape.ly/api.md) give you the third option: a maintained scraper for platforms such as Google Maps, Amazon or Reddit that you call like any REST API. ## 6. Rotate proxies responsibly Proxies spread requests across IP addresses so one address doesn't hit a site's rate limit. They're a tool for reliability, not a licence to send more traffic than the site can handle: keep your total rate low (practice 2) whatever your IP count. - Start without proxies or with **datacenter** IPs on sites that don't block them; they're cheaper. - Move to **residential** IPs for sites that block datacenter ranges, and pick a country when content is localised. - Rotate per session rather than per request when a site ties carts, cookies or pagination to an IP. On Scrape.ly, every run can use the managed [proxies](https://scrape.ly/proxies.md) without a separate account: residential from $4 per GB on the free plan down to $2 on Business, datacenter from $1 down to $0.75. The Web Scraper's `proxyRotation` setting offers `RECOMMENDED`, `PER_REQUEST` and `UNTIL_FAILURE`. Our pillar on [web scraping without getting blocked](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) covers fingerprints, headers and browsers, and [Scrape.ly anti-blocking](https://scrape.ly/anti-blocking.md) shows what the platform handles for you. ## 7. Handle errors and retry with backoff Networks fail, servers return 503 under load, and rate limits answer 429. Retry those with growing pauses (the `Retry` object above waits 2, 4 and 8 seconds and honours `Retry-After`), but don't retry 404s, and stop after a few attempts. Log every failed URL so you can rerun just those. The Web Scraper retries transport failures and HTTP 401, 403, 408, 429 and 5xx responses up to `maxRequestRetries` times (default 3). A polite configuration for a small site looks like this: ```json { "startUrls": [{ "url": "https://books.toscrape.com/" }], "pageFunction": "async function pageFunction(context) { return { url: context.request.url, title: context.jQuery('title').text() }; }", "respectRobotsTxtFile": true, "maxConcurrency": 2, "maxRequestRetries": 3, "maxPagesPerCrawl": 500, "proxyRotation": "UNTIL_FAILURE", "proxyConfiguration": { "useScrapelyProxy": true } } ``` Errors can also cost money: a pagination bug that loops forever keeps fetching. Give every run a hard ceiling. With the Python client (`pip install scrapely-client-python`): ```python import os from decimal import Decimal from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") run = client.actor("querystone/web-scraper").call( run_input={ "startUrls": [{"url": "https://books.toscrape.com/"}], "pageFunction": "async function pageFunction(context) { return { url: context.request.url }; }", "maxPagesPerCrawl": 500, "proxyConfiguration": {"useScrapelyProxy": True}, }, max_items=500, max_total_charge_usd=Decimal("0.50"), ) print(run.status) # SUCCEEDED, FAILED, ABORTED or TIMED-OUT ``` When a run reaches its maximum charge, Scrape.ly stops it. With a prepaid balance on top, there's no way to get a surprise bill. ## 8. Validate every record A scraper that "succeeds" with empty prices is worse than one that fails, because nobody notices. Check each record for required fields, types and plausible ranges before you store it: ```python REQUIRED = {"title": str, "price_gbp": float, "rating": int, "url": str} def validate(book: dict) -> list[str]: problems = [f"missing {k}" for k in REQUIRED if book.get(k) in (None, "")] problems += [f"{k} is not {t.__name__}" for k, t in REQUIRED.items() if k in book and book[k] not in (None, "") and not isinstance(book[k], t)] if isinstance(book.get("price_gbp"), float) and not 0 < book["price_gbp"] < 1000: problems.append(f"price out of range: {book['price_gbp']}") return problems ``` Fed a good record and a broken one (empty title, price 0.0), it flagged only the second with `['missing title', 'price out of range: 0.0']`. Quarantine bad records instead of dropping them silently, so you can see what changed. The same check works on a Scrape.ly dataset: iterate the items with the client and validate each one. ## 9. Monitor for site changes Sites change their HTML without notice. Turn validation into an alarm: if more than a few percent of records fail, or the record count drops sharply against the last run, stop and alert a person. ```python bad = {book["url"]: problems for book in books if (problems := validate(book))} if len(bad) > 0.05 * len(books): raise RuntimeError(f"{len(bad)} of {len(books)} records failed validation; the layout may have changed") ``` On Scrape.ly, scheduled [automations](https://scrape.ly/docs/automations.md) have **Alert me if it fails** on by default (email and dashboard), webhooks can call your endpoint on `ACTOR.RUN.FAILED`, and an automation can start a second Actor after each run to run checks like the one above. With Store Actors, the maintainer fixes selectors when the target site changes, which removes most of this work for big platforms. ## 10. Store as little personal data as you can Reviews, comments, profiles and directories contain personal data, and privacy law such as the GDPR applies to it even when it's public. Decide before you scrape which fields you need, drop the rest at collection time, and set a deletion date. Some Actors make this a switch. The [Google Maps Reviews Scraper](https://scrape.ly/store/cornerledger/google-maps-reviews-scraper.md) has a `personalData` input; set it to `false` and the reviewer's name, ID, profile URL and photo are dropped, while the review text, rating and date stay. Each run's default dataset also expires under your plan's retention (2 days on the free plan, apart from your latest 10 runs; 31, 60 or 90 days on paid plans), so personal data doesn't linger unless you copy it somewhere permanent. ## 11. Schedule off-peak Run big jobs when the target site is quiet: overnight in its own time zone, not yours. It lowers your impact on real users and often makes responses faster. In cron, `0 3 * * *` means 03:00 every day. On Scrape.ly, create a schedule in **Automations**: hourly, daily, weekly, monthly or a custom cron expression, with a time zone you choose. Schedules follow daylight saving in that time zone, so a 3 a.m. Berlin run stays at 3 a.m. Berlin time all year. Spread several jobs a few minutes apart rather than starting them all on the hour. ## 12. Document what you collect and why Write down, for every scraper: the source and the pages it visits, the fields it saves, why you need them, how often it runs, where the data goes and when it's deleted. It takes ten minutes, it's what a privacy review or a site owner's email will ask for, and it makes the scraper maintainable by someone other than you. On Scrape.ly, save each configuration as a saved input (a task in the API) with a clear name. It keeps the input and run options, including the maximum cost, together, so the scraper that runs is the one you documented. Keep run IDs in your logs; each run keeps its input, log and dataset for its retention period. ## FAQ ### What are the most important web scraping best practices? Respect robots.txt and terms, keep your request rate low, retry errors with backoff, and validate your data on every run. Those four prevent most blocks, most legal friction and most silent data failures. ### How do I avoid getting blocked while scraping? Slow down first: fewer parallel requests, pauses between them and backoff on 429. Then use realistic headers, session-consistent proxies and a real browser only where needed. Our guide to web scraping without getting blocked covers each layer. ### Is it OK to ignore robots.txt? Robots.txt isn't a law, but ignoring it removes one of your best arguments that you scraped in good faith, and it usually points at pages you don't need. Honour it unless you have a specific, documented reason and permission from the site. ### How fast should a web scraper be? There's no universal number. One request per second per host is a common, conservative starting point for small sites; large platforms tolerate more. Watch response times and error rates, and slow down when either rises. ### How do I keep a scraper running when the website changes? Validate every run, alert on rising failure rates, and keep selectors in one place so fixes are quick. For big platforms, use a maintained Actor from the Scrape.ly Store, where the maintainer updates the scraper and your input stays the same. Scrape.ly Store Web Scraper Polite by configuration: robots.txt, concurrency limits, retries and managed proxies built in. $0.50 per 1,000 results. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art scene in two halves: on the left a small robot walks along a cable linking web pages, on the right a robot with an orange >\_ on its terminal head pulls a data card out of a single web page](https://scrape.ly/media/web-crawling-vs-web-scraping.png?w=750)](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) ### [Web Crawling vs Web Scraping: The Difference, With Code and Real Runs](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) [Web crawling finds pages; web scraping pulls data out of them. See the difference in one Python script, in a real crawl of crawlee.dev that cost $0.0031, and learn which one your project needs.](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a data card out between two routes: a cable from a browser window on the left and a cable from an API socket with curly braces and a Scrape.ly-branded plug on the right](https://scrape.ly/media/web-scraping-vs-api.png?w=750)](https://scrape.ly/blog/web-scraping-vs-api.md) ### [Web Scraping vs API: Which to Use, What Each Costs, and How to Get Both](https://scrape.ly/blog/web-scraping-vs-api.md) [Web scraping vs API: an official API gives you what the platform chooses to expose, on its terms and prices; scraping gets you what any visitor sees. Real 2026 prices for X, Google Places and Reddit, and how to call a scraper like an API.](https://scrape.ly/blog/web-scraping-vs-api.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands thinking between two matching arcade cabinets, one with the Playwright logo and one with the green Selenium logo, next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/playwright-vs-selenium.png?w=750)](https://scrape.ly/blog/playwright-vs-selenium.md) ### [Playwright vs Selenium for Web Scraping: The Same Scraper in Both (Python)](https://scrape.ly/blog/playwright-vs-selenium.md) [Playwright vs Selenium for web scraping: languages, browsers, waiting, network interception, parallel sessions and detection, from the official docs, plus the same Python scraper written in both and tested on 22 October 2026.](https://scrape.ly/blog/playwright-vs-selenium.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.