--- title: "Web Scraping Without Getting Blocked | Scrape.ly" description: "Web scraping without getting blocked: 10 techniques with tested Python, from headers and backoff to proxies, fingerprints and CAPTCHAs, plus real runs." url: https://scrape.ly/blog/web-scraping-without-getting-blocked markdown: https://scrape.ly/blog/web-scraping-without-getting-blocked.md --- [Guides](https://scrape.ly/blog/category/guides.md) # Web Scraping Without Getting Blocked: 10 Techniques That Work Web scraping without getting blocked comes down to ten habits: real headers, pacing, sessions, good IPs, real browsers, clean fingerprints, challenge handling, honeypot checks, robots.txt and caching. Tested Python for each, and how Scrape.ly handles them for you. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 10 min read ![Pixel-art robot with an orange >\_ on its terminal head walks past a raised red-and-white boom barrier, carrying a data card from the web page behind it towards a crate](https://scrape.ly/media/web-scraping-without-getting-blocked.png?w=1080) Web scraping without getting blocked means making your scraper look and behave like an ordinary visitor: browser-like requests, a polite pace, stable sessions, clean IP addresses and a real browser when the page needs one. Most blocks are not mysterious. A site sees a request that no human browser would send, or too many of them from one place, and it says no. Below are the 10 techniques that matter, in the order you should reach for them, with Python we tested on 23 October 2026 against the [books.toscrape.com](https://books.toscrape.com/) sandbox. For each one you'll see what it does on its own and how Scrape.ly handles it, so you can decide what to build and what to skip. ## Web scraping without getting blocked: the 10 techniques at a glance | # | Technique | What it prevents | How Scrape.ly handles it | | -- | ------------------------------ | ------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | 1 | Browser-like headers and TLS | Instant "this is a script" flags | The Website Content Crawler fetches with a Chrome TLS fingerprint; the Scrape.ly browser is real Chromium | | 2 | Slow down and back off | Rate limits, 429s, temporary bans | Concurrency and retry inputs on Actors, concurrency caps per plan | | 3 | Sessions and cookies | Looking like a new visitor on every request | Proxy sessions keep related requests on one IP | | 4 | Proxy rotation | IP bans and per-IP limits | Managed residential and datacenter proxy on every run | | 5 | A real browser when needed | Empty pages from JavaScript sites | Playwright and Puppeteer crawlers, Chrome on demand | | 6 | Clean browser fingerprints | Headless browser detection | The Scrape.ly browser, a hosted Chromium with anti-bot fingerprints | | 7 | Handle CAPTCHAs and challenges | Challenge pages and Turnstile | The Scrape.ly browser solves Cloudflare Turnstile | | 8 | Avoid honeypots | Trap links that only bots follow | Actors follow only the links they're built to follow | | 9 | Respect robots.txt and terms | Legal and ethical trouble | A robots.txt switch on the crawler | | 10 | Cache and don't refetch | Wasted requests that raise your profile | Results stored in datasets, duplicates skipped | ## Why scrapers get blocked Anti-bot systems score every request. Cloudflare, for example, describes a [bot score](https://developers.cloudflare.com/bots/concepts/bot-score/) from 1 to 99 built from "headers, session characteristics, and browser signals", plus a JavaScript detection engine that "identifies headless browsers and other malicious fingerprints". It also fingerprints the TLS handshake: [JA3 and JA4 fingerprints](https://developers.cloudflare.com/bots/additional-configurations/ja3-ja4-fingerprint/) "identify TLS clients based on how they initiate connections". So a site can judge you at four levels: the IP address, the TLS connection, the HTTP request, and the browser that runs its JavaScript. A scraper gets blocked when any level looks wrong. The techniques below fix one level each. ## 1. Send browser-like headers, and a browser-like TLS handshake A bare `requests.get()` announces itself as `python-requests`. Cloudflare's docs are explicit about the extreme case: requests with a missing or empty `User-Agent` "are immediately assigned a bot score of 1", the most certain bot verdict. Send the headers a current browser sends, from one `Session` so cookies and connections persist: ```python import random import time import requests HEADERS = { "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.9", } session = requests.Session() # keeps cookies and reuses connections session.headers.update(HEADERS) ``` Keep the User-Agent current: Chrome 156 was the stable release on 23 October 2026 (per Google's [Chromium Dash](https://chromiumdash.appspot.com/)), and a two-year-old version is itself a signal. Headers don't change the TLS handshake, though. Python's TLS stack doesn't look like Chrome's, and JA3/JA4 fingerprinting sees that before any header is read. [curl_cffi](https://github.com/lexiforest/curl_cffi) sends requests with a browser's TLS and HTTP/2 settings: ```python from curl_cffi import requests r = requests.get("https://books.toscrape.com/", impersonate="chrome", timeout=20) print(r.status_code, r.http_version) # 200 3 (HTTP/2) in our test ``` **On Scrape.ly:** you don't write this. The [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) fetches server-rendered pages "with HTTP and a Chrome TLS fingerprint", and opens Chrome only when a page needs JavaScript. ## 2. Slow down, and back off when the site says so Speed is the most common reason scrapers get blocked. Space requests out with random delays, cap how many run at once, and treat `429 Too Many Requests` and `503` as instructions, not errors. The [Retry-After](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After) header tells you how long to wait. Cloudflare's [error 1015 page](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1015/) warns that retrying quickly "may extend the block". ```python def get(url, retries=4, base_delay=1.0): """GET with exponential backoff and jitter; honours Retry-After on 429/503.""" for attempt in range(retries + 1): resp = session.get(url, timeout=20) if resp.status_code not in (429, 500, 502, 503, 504): return resp if attempt == retries: resp.raise_for_status() retry_after = resp.headers.get("Retry-After") wait = float(retry_after) if retry_after and retry_after.isdigit() else base_delay * 2 ** attempt time.sleep(wait + random.uniform(0, 1)) ``` Between pages, add `time.sleep(random.uniform(1, 3))`. We tested the retry path against a local server that answered `429` with `Retry-After: 2` once: the function waited 2.2 seconds and got the `200` on the second attempt. **On Scrape.ly:** Actors expose these as inputs. The Website Content Crawler has `maxConcurrency` (described in its schema as "useful to avoid overloading the target websites and to avoid getting blocked") and `maxRequestRetries`, and the platform caps concurrency per plan. ## 3. Keep sessions and cookies A real visitor arrives once, gets cookies, and keeps them. A scraper that throws cookies away looks like a new stranger on every page. Cloudflare says it uses its `__cf_bm` cookie to "smooth out the bot score and reduce false positives for actual user sessions", so a session that keeps its cookies is judged on its history, not one request. `requests.Session()` above already does this. The flip side: once a session is flagged, retire it, along with its IP and cookies, instead of retrying with the same identity. **On Scrape.ly:** proxy sessions keep related requests on the same IP (`newUrl(sessionId)` in the SDK), and the Website Content Crawler rotates the whole session, "IP address + browser configuration", when it hits anti-scraping measures, up to `maxSessionRotations` times. ## 4. Rotate proxies, and pick the right pool Every site can count requests per IP. Spreading traffic across many IPs keeps each one under the limit, and gets you around an IP ban when one happens. The pool matters too: datacenter IPs are cheap and fast, residential IPs come from home connections and look like ordinary visitors. We compare them in detail in [residential vs datacenter proxies](https://scrape.ly/blog/residential-vs-datacenter-proxies.md). Inside a Scrape.ly Actor, the managed proxy needs no credentials. This is the Python SDK call from the proxy docs: ```python import asyncio from scrapely import Actor async def main(): proxy_configuration = await Actor.create_proxy_configuration( groups=["RESIDENTIAL"], country_code="US", ) proxy_url = await proxy_configuration.new_url() Actor.log.info(f"Using {proxy_url}") if __name__ == "__main__": asyncio.run(Actor.main(main)) ``` **On Scrape.ly:** residential costs $4 per GB on the Free plan down to $2 on Business, datacenter $1 down to $0.75, billed as run usage. Each run gets its own credentials that only work inside that run, so there's no proxy password to leak. See [proxies](https://scrape.ly/proxies.md) for the full price list. ## 5. Use a real browser when the page needs one If the data is written into the page by JavaScript, an HTTP client sees an empty shell and you'll think you're blocked when you aren't. Use Playwright for those pages, and block images, fonts and stylesheets so you don't pay for bandwidth you don't need. Don't use a browser for everything, though: it's slower, heavier and easier to fingerprint than a well-formed HTTP request. **On Scrape.ly:** templates give you a `PlaywrightCrawler` or `PuppeteerCrawler` ready to deploy, and the Website Content Crawler decides per page: HTTP first, Chrome only when the HTML is "only a script". ## 6. Fix the browser fingerprint A default headless browser gives itself away. In our test, headless Chromium from Playwright 1.64 failed or warned on 13 checks at [bot.sannysoft.com](https://bot.sannysoft.com/), including `navigator.webdriver` being `true` and "HeadlessChrome" in the User-Agent. With the stealth plugin loaded, every check passed. Our [Playwright Stealth guide](https://scrape.ly/blog/playwright-stealth.md) has the setup in Node and Python, and the limits: the stealth packages patch what JavaScript can see, not your IP or TLS handshake. **On Scrape.ly:** the [Scrape.ly browser](https://scrape.ly/docs/browser.md) is a hosted Chromium with anti-bot fingerprinting. You turn it on per Actor and rebuild, and your Playwright or Puppeteer code connects to it unchanged. You choose the operating system (Windows, macOS or Android) and the exit country, state or city; time zone and language follow the location. ## 7. Handle CAPTCHAs and challenge pages Challenge pages are what you see when a site isn't sure about you. Cloudflare says its challenges don't use visual puzzles at all; they run JavaScript checks and sometimes ask for a click ([challenges docs](https://developers.cloudflare.com/cloudflare-challenges/)). A plain HTTP client can't pass them. The right first response is to detect them and slow down, because a challenge usually means one of techniques 1 to 6 is off. We go through what Cloudflare checks, and what to do, in [how to bypass Cloudflare when scraping](https://scrape.ly/blog/bypass-cloudflare-scraping.md). **On Scrape.ly:** the Scrape.ly browser solves Cloudflare Turnstile challenges, on by default, at $0.45 per browser hour at list price plus its traffic. Store Actors built for hard sites handle challenges themselves: our eBay Scraper run below logged four challenge solves. ## 8. Stay out of honeypots Some sites plant links that no human can see, hidden with `display: none` or the `hidden` attribute. A person never clicks them, so any client that follows one has identified itself. Filter them before you queue links: ```python from urllib.parse import urljoin from bs4 import BeautifulSoup def visible_links(html, base): soup = BeautifulSoup(html, "html.parser") links = [] for a in soup.select("a[href]"): style = (a.get("style") or "").replace(" ", "").lower() if a.has_attr("hidden") or "display:none" in style or "visibility:hidden" in style: continue # likely a honeypot: no human can click it links.append(urljoin(base, a["href"])) return links ``` This catches inline hiding. Links hidden by a CSS class need a browser check (`locator.is_visible()` in Playwright). Better still, follow only the links you need, by pattern, rather than every link on the page. **On Scrape.ly:** crawlers follow only links that match the start URL path or your include globs, so a stray trap link elsewhere on the site isn't queued. ## 9. Respect robots.txt and the site's terms [robots.txt](https://www.rfc-editor.org/rfc/rfc9309.html) is the site telling crawlers where they're welcome. Checking it costs one request: ```python from urllib import robotparser from urllib.parse import urljoin def allowed(url, agent="*"): rp = robotparser.RobotFileParser(urljoin(url, "/robots.txt")) rp.read() # a missing robots.txt (404) means everything is allowed return rp.can_fetch(agent, url) ``` This isn't only about manners. Scraping paths a site has asked you to avoid, or ignoring its terms, is how a technical problem becomes a legal one. Stick to public pages, don't collect personal data you don't need, and read our guide on [whether web scraping is legal](https://scrape.ly/blog/is-web-scraping-legal.md). We cover the wider set of habits in [web scraping best practices](https://scrape.ly/blog/web-scraping-best-practices.md). **On Scrape.ly:** the Website Content Crawler has a `respectRobotsTxtFile` switch, on in its form by default. Store Actors collect publicly available data that any visitor can see. ## 10. Cache, and don't fetch what you already have Every request you don't make is one that can't get you blocked. Store what you've scraped, and when you revisit a page, ask whether it changed. If the server sends an [ETag](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/ETag), send it back: ```python etags = {} def get_cached(url): """Send If-None-Match so unchanged pages come back as an empty 304.""" headers = {"If-None-Match": etags[url]} if url in etags else {} resp = session.get(url, headers=headers, timeout=20) if "ETag" in resp.headers: etags[url] = resp.headers["ETag"] return resp ``` **On Scrape.ly:** results land in a dataset you can re-read through the API as often as you like, schedules run only as often as you set them, and the Website Content Crawler doesn't save two pages that share a canonical URL. ## What our test printed Techniques 1, 2, 8, 9 and 10 together, run against books.toscrape.com on 23 October 2026 (macOS, Python 3.12, requests and Beautiful Soup): three listing pages with 1 to 3 second pauses, a robots.txt check, the honeypot filter on a test snippet with two hidden links, and the ETag cache on the home page. ```plaintext robots allows: True pages: 3 books: 60 first: A Light in the Attic honeypot test: ['https://books.toscrape.com/a'] cache: 200 51294 -> 304 0 ``` The second request for the home page came back as `304 Not Modified` with zero bytes instead of 51 KB. ## What it looks like in real runs Three runs from the Scrape.ly Store show the same techniques working without any code from you: - **Website Content Crawler, 22 October 2026:** 10 Python docs pages in 7 seconds over HTTP with a Chrome TLS fingerprint and the residential proxy. Residential traffic was 0.14 MB; the whole run cost $0.0058. - **eBay Scraper, 22 October 2026:** 10 listings in 35 seconds through the datacenter pool. The log shows the anti-bot handling: `Anti-bot solver source: platform.` and then `Saved 10 listings. Challenge solves: 4`. Total $0.0466. - **Google AI Overviews Scraper, 23 October 2026:** the first browser session landed on a block page, the Actor dropped it and the second loaded the results: ```plaintext 20261023T03:02:38.040Z [scrapely.actor] INFO browser_session attempt=1 state=sorry latency_ms=4644 connect=1574 commit=2819 ready=4533 20261023T03:02:41.983Z [scrapely.actor] INFO browser_session attempt=2 state=serp latency_ms=3835 connect=1508 commit=2711 ready=3707 20261023T03:02:49.042Z [scrapely.actor] INFO Done: 3 AI Overviews, 0 queries without one, 0 failed, 33 queries in 16 s. (deferred overviews: 2, deferred without overview: 0, partial: 0) ``` That run cost $0.0259, of which $0.021 was Scrape.ly browser time and traffic. Scrape.ly Store Crawl a site without building the stack Chrome TLS fingerprint, residential proxy, retries and robots.txt built in: $0.0005 per page, on $5 of free monthly credit. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) ## How Scrape.ly handles blocking for you Doing all ten yourself is a project. Scrape.ly splits the work into two layers, described on the [anti-blocking page](https://scrape.ly/anti-blocking.md): - **The managed proxy** for HTTP traffic: residential by default or datacenter, targeted by country, with sessions, and nothing to configure. You can bring your own proxy URLs too; that traffic is never billed as proxy usage. - **The Scrape.ly browser** for sites that block headless browsers: a hosted Chromium with anti-bot fingerprints, Cloudflare Turnstile solving and its own residential proxy, at $0.45 per browser hour at list price. On top of both sit 568 ready-made Actors in the [Scrape.ly Store](https://scrape.ly/store.md), each built for a specific site or job, so keeping up with that site's defenses is the maintainer's work, not yours. Every run has an optional maximum charge, and proxy and browser usage count toward it, so a run that keeps hitting walls can't run up a bill. ## FAQ ### How do I bypass an IP ban when scraping? First stop and find out why you were banned: usually too many requests, too fast, from one IP. Slow down and add backoff, then spread requests over rotating proxies so no single IP crosses the limit. If datacenter IPs are banned as a range, switch to residential IPs. Don't use rotation to keep hammering a site that has clearly asked you to stop. ### Can websites tell that I'm scraping? They can tell when your traffic doesn't look like a browser's: missing or odd headers, a non-browser TLS handshake, headless browser properties, or request rates no person could produce. Each technique above removes one of those tells. ### Do I need proxies for web scraping? For a few hundred pages at a polite pace, often not. For anything larger, or sites that limit per IP, yes. On Scrape.ly every run can use the managed proxy, billed per GB. ### Is a headless browser enough to avoid blocks? No. A default headless browser is easier to detect than a good HTTP client, because it exposes properties such as `navigator.webdriver`. You need a browser with clean fingerprints and a good IP, which is what the Scrape.ly browser provides. ### Is it legal to scrape a site that tries to block scrapers? Blocking is a technical measure, not a law, but it can signal that a site doesn't want automated access, and its terms may say so. Scrape public data only, respect robots.txt and rate limits, and avoid personal data. This is not legal advice; see our guide on whether web scraping is legal. About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head reaches for the last empty box on a clipboard checklist whose first four items carry orange ticks, with a web page standing behind it](https://scrape.ly/media/web-scraping-best-practices.png?w=750)](https://scrape.ly/blog/web-scraping-best-practices.md) ### [12 Web Scraping Best Practices for Reliable, Responsible Scrapers](https://scrape.ly/blog/web-scraping-best-practices.md) [Twelve web scraping best practices that keep scrapers polite, unblocked and accurate: robots.txt, rate limits, caching, retries, validation, monitoring, personal data and more. Each with tested Python code and how to do it on Scrape.ly.](https://scrape.ly/blog/web-scraping-best-practices.md) - [![Pixel-art scene in two halves: on the left a small robot walks along a cable linking web pages, on the right a robot with an orange >\_ on its terminal head pulls a data card out of a single web page](https://scrape.ly/media/web-crawling-vs-web-scraping.png?w=750)](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) ### [Web Crawling vs Web Scraping: The Difference, With Code and Real Runs](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) [Web crawling finds pages; web scraping pulls data out of them. See the difference in one Python script, in a real crawl of crawlee.dev that cost $0.0031, and learn which one your project needs.](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a data card out between two routes: a cable from a browser window on the left and a cable from an API socket with curly braces and a Scrape.ly-branded plug on the right](https://scrape.ly/media/web-scraping-vs-api.png?w=750)](https://scrape.ly/blog/web-scraping-vs-api.md) ### [Web Scraping vs API: Which to Use, What Each Costs, and How to Get Both](https://scrape.ly/blog/web-scraping-vs-api.md) [Web scraping vs API: an official API gives you what the platform chooses to expose, on its terms and prices; scraping gets you what any visitor sees. Real 2026 prices for X, Google Places and Reddit, and how to call a scraper like an API.](https://scrape.ly/blog/web-scraping-vs-api.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.