--- title: "How to Bypass Cloudflare When Scraping | Scrape.ly" description: "How to bypass Cloudflare when scraping public pages: what Cloudflare checks, why simple requests fail, and the options that work, with tested code." url: https://scrape.ly/blog/bypass-cloudflare-scraping markdown: https://scrape.ly/blog/bypass-cloudflare-scraping.md --- [Guides](https://scrape.ly/blog/category/guides.md) # How to Bypass Cloudflare When Scraping (Responsibly) What Cloudflare actually checks (bot scores, TLS fingerprints, challenge pages, Turnstile), from its own docs, why a plain HTTP request fails, and the responsible ways to scrape public pages behind it, up to a hosted browser that solves Turnstile. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 8 min read ![Pixel-art robot with an orange >\_ on its terminal head walks out through an open iron gate whose arch carries the orange Cloudflare cloud logo, carrying a data card from the web page inside towards a crate](https://scrape.ly/media/bypass-cloudflare-scraping.png?w=1080) To bypass Cloudflare when scraping, you don't break anything: you make your scraper look and behave like the ordinary browser Cloudflare is built to let through. That means a real browser engine, clean fingerprints, a good IP address and a polite pace. When those are right, most public pages load like they would for any visitor. This guide is about public pages: product listings, articles, docs, prices, anything a visitor can see without logging in. It is not about getting into accounts, paywalls or private areas, and not about overloading a site. Below is what Cloudflare checks, described from its own documentation as of 23 October 2026, why simple requests fail, and the options in order of effort, ending with a hosted browser that solves Turnstile challenges for you. ## First: scrape responsibly Cloudflare's bot protection exists because a lot of automated traffic is abusive: credential stuffing, scalping, attacks. Your scraper should never look like that, in behavior as well as fingerprint. - **Public pages only.** If a page needs a login, it's out of scope. - **Check for an official route first.** An API, a data export or an RSS feed is faster and more stable than scraping. - **Respect robots.txt, the site's terms and its rate limits.** Cloudflare's own bar for a [verified bot](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) is a client that "obeys `robots.txt` and crawl directives, maintains reasonable request rates", and identifies itself honestly. If you run a large crawler for search, monitoring or similar, you can apply to be one. - **A block is information.** If a site keeps challenging you at a gentle pace, it may simply not want automated visitors on that path. Our guide on [whether web scraping is legal](https://scrape.ly/blog/is-web-scraping-legal.md) covers the legal side. ## What Cloudflare checks Cloudflare sits between the visitor and the website, so it sees every request before the site does. Its docs describe several layers: | Layer | What Cloudflare documents | What a default scraper shows | | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | Bot score | A score from 1 to 99 from heuristics, machine learning on "headers, session characteristics, and browser signals", and JavaScript detections | Script-like headers; no cookies; a missing User-Agent scores 1 | | TLS fingerprint | JA3 and JA4 fingerprints "identify TLS clients based on how they initiate connections" | A handshake that doesn't match any browser | | JavaScript detections | An invisible script that "identifies headless browsers and other malicious fingerprints" | Nothing runs it (HTTP client) or it reveals automation (default headless browser) | | Challenge pages | Full-page interstitials: non-interactive, managed or interactive | A `403` HTML page instead of the content | | Turnstile | Small JavaScript checks: proof-of-work, proof-of-space, probing web APIs, browser quirks and human behavior | Can't run them, or fails them | | Rate limiting | Site-defined limits; error 1015 "You are being rate limited" | Bursts of requests from one IP | A few details from the docs matter for scrapers: - **Bot scores** come from several engines. The heuristics engine gives requests "with a missing or empty `User-Agent` header" a score of 1 immediately, and Cloudflare uses its `__cf_bm` cookie to "smooth out the bot score" over a user's session ([bot scores](https://developers.cloudflare.com/bots/concepts/bot-score/)). Granular scores and JA3/JA4 fingerprints are Enterprise Bot Management features, so what you run into depends on the site's plan. - **Bot Fight Mode**, the free tier, "issues computationally expensive challenges" to traffic matching known bot patterns, and JavaScript detections are always on with it ([Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/)). - **Challenge pages** hold the request while the browser runs checks. A non-interactive one "typically takes less than five seconds"; a managed one may ask for a click if Cloudflare "detects non-human attributes" ([challenge pages](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/)). - **No picture puzzles.** "Cloudflare does not use CAPTCHA puzzles or visual tests like selecting objects or typing distorted characters" ([challenges](https://developers.cloudflare.com/cloudflare-challenges/)). [Turnstile](https://developers.cloudflare.com/turnstile/), its CAPTCHA replacement, can also be embedded on sites that don't use Cloudflare's CDN, so you can meet it anywhere. ## Why simple requests fail on Cloudflare scraping A `requests.get()` fails on several of those layers at once. Its TLS handshake is Python's, not Chrome's. Its headers say `python-requests`. And it can't run JavaScript, so it can never complete a challenge: it just receives the challenge page as HTML, usually with a `403`. You can at least tell that's what happened. Cloudflare marks every challenge response with a `cf-mitigated: challenge` header ([docs](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/)). Check for it, and slow down instead of retrying fast: ```python import time import requests def is_challenge(resp): """Cloudflare marks every challenge page with cf-mitigated: challenge.""" return resp.headers.get("cf-mitigated") == "challenge" def fetch(session, url, max_challenges=2): for attempt in range(max_challenges + 1): resp = session.get(url, timeout=20) if not is_challenge(resp): return resp time.sleep(30 * (attempt + 1)) # challenged: slow down instead of retrying fast return None # still challenged: this client needs a real browser ``` We tested it on 23 October 2026 against books.toscrape.com (no challenge, returns the page) and a local server that mimics a challenge response (detected). Two other Cloudflare errors are worth recognizing: [1020 "Access denied"](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1020/) is a block rule the site owner set, returned as a `403`, and [1010](https://developers.cloudflare.com/support/troubleshooting/http-status-codes/cloudflare-1xxx-errors/error-1010/) means the owner banned your browser's signature. Neither is a puzzle to solve; they're a "no". ## Your options, from lightest to strongest ### 1. Slow down, or use the official route If you're seeing error 1015 or occasional challenges, your pace is the problem. Cloudflare's advice on 1015 is to wait, because retrying quickly "may extend the block". Add delays, cut concurrency and cache what you've already fetched. The techniques are in our pillar guide, [web scraping without getting blocked](https://scrape.ly/blog/web-scraping-without-getting-blocked.md). ### 2. Use a real browser, not an HTTP client Challenges and Turnstile run JavaScript, so you need a browser engine. Playwright or Puppeteer will execute the checks, but a default headless browser fails many of them: in our test, headless Chromium from Playwright 1.64 reported `navigator.webdriver` as `true` and "HeadlessChrome" in its User-Agent. Stealth plugins patch those properties; our [Playwright Stealth guide](https://scrape.ly/blog/playwright-stealth.md) shows the setup and the results, and why patching JavaScript properties doesn't change your IP or TLS handshake. ### 3. Use residential IPs Anti-bot systems weigh where a request comes from, not only what it looks like. Datacenter IP ranges carry a lot of automated traffic, so they tend to draw more scrutiny than home connections. Residential proxies route your browser through ordinary consumer connections; we compare the two in [residential vs datacenter proxies](https://scrape.ly/blog/residential-vs-datacenter-proxies.md). ### 4. Use a hosted browser built for this Putting 2 and 3 together yourself, and keeping it working as detection changes, is a maintenance job. A hosted anti-bot browser does it for you. That's what the Scrape.ly browser is. ## Bypass Cloudflare with the Scrape.ly browser The [Scrape.ly browser](https://scrape.ly/docs/browser.md) is a hosted Chromium your Actor's Playwright or Puppeteer code runs on. It has anti-bot fingerprinting, routes traffic through its own residential proxy, and solves Cloudflare Turnstile challenges, on by default. **1. Turn it on.** In the dashboard, open your Actor, go to **Settings** → **Scrapely Browser** and turn on **Use Scrapely browser**. **2. Rebuild.** The setting applies at build time, so start a new build (`scrapely push` or **Builds** → **Build**). The build log confirms it: ```plaintext [Scrapely Build] Browser patch: on (remote browser) ``` **3. Run your code unchanged.** Every browser launch, including raw `chromium.launch()` and `puppeteer.launch()`, now connects to the Scrape.ly browser. To pick where it exits and how it looks, add a browser configuration. Turnstile solving is the `solveCloudflare` option, `true` unless you turn it off: ```javascript import { Actor, PlaywrightCrawler } from 'scrapely'; await Actor.main(async () => { const browserConfiguration = await Actor.createBrowserConfiguration({ countryCode: 'US', state: 'California', os: 'windows', solveCloudflare: true, }); const crawler = new PlaywrightCrawler({ browserConfiguration, async requestHandler({ page }) { await Actor.pushData({ title: await page.title() }); }, }); await crawler.run(['https://example.com']); }); ``` If your code connects over CDP itself, every run gets a `CDP_URL` environment variable: ```python import os from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser = await p.chromium.connect_over_cdp(os.environ["CDP_URL"]) ``` Time zone and language follow the exit location unless you set them, so the fingerprint stays consistent with the IP. You can pick Windows, macOS or Android as the operating system. Firefox, WebKit and Camoufox launches are redirected to the Scrape.ly browser too. **What it costs.** $0.45 per browser hour at list price, billed in 30-second blocks plus the browser's proxy traffic per MB, on top of the run's compute. That traffic is never billed again as proxy GB. For scale: a Store run on 23 October 2026 that used the Scrape.ly browser for 13.5 seconds and 4.3 MB of traffic was charged $0.021 for it. Browser usage counts toward the run's maximum charge, so a run can never spend more than you allow. See [anti-blocking](https://scrape.ly/anti-blocking.md) for how the browser and the managed proxy fit together. ## When you don't need to write a scraper at all For popular sites, someone has already solved this. The [Scrape.ly Store](https://scrape.ly/store.md) has 568 ready-made Actors, and the maintainer keeps each one working as the site changes. For crawling public docs, blogs and help centers into clean text, the [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) fetches with a Chrome TLS fingerprint, opens Chrome only when a page needs JavaScript, rotates its session on anti-scraping measures and can respect robots.txt. Scrape.ly Store Public pages to clean text, no setup Chrome TLS fingerprint, residential proxy, retries and session rotation built in: $0.0005 per page, on $5 of free monthly credit. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) ## What to watch out for - **Challenge loops.** If every page challenges you even in a real browser, something in your setup can be inconsistent: a time zone that doesn't match the IP's country, a reused flagged session, or too much speed. The Scrape.ly browser keeps time zone and language matched to the exit location by default. - **Blocks that aren't challenges.** Errors 1020 and 1010 are rules the site owner set. Respect them. - **Turnstile on forms.** Solving a challenge to read a public page is different from submitting forms in bulk. Don't automate sign-ups, logins or checkouts. - **Cost of browsers.** A browser is heavier than an HTTP request. Use it only for the pages that need it, block images and fonts, and set a maximum charge per run. ## FAQ ### Can I bypass Cloudflare with Python requests? Usually not. Requests can't run JavaScript, so it can't complete a challenge page or Turnstile, and its TLS handshake doesn't match a browser's. On sites with only rate limiting, slowing down may be enough. For challenges you need a browser engine with clean fingerprints. ### Is it legal to bypass Cloudflare when scraping? Reading public pages that any visitor can see is generally lower risk than accessing anything behind a login, but the site's terms, copyright and privacy law still apply, and getting around access controls can raise legal questions in some places. This is not legal advice. Stick to public data, respect robots.txt and rate limits, and read our guide on whether web scraping is legal. ### Does Cloudflare use CAPTCHAs? Not picture puzzles. Cloudflare says its challenges don't use "CAPTCHA puzzles or visual tests". They run JavaScript checks in the browser and sometimes ask for a click on a checkbox. Turnstile is its CAPTCHA replacement for forms and pages. ### What is the cf_clearance cookie? It's the clearance cookie Cloudflare issues after a visitor passes verification, for example through Turnstile pre-clearance, so later requests from that visitor aren't challenged again. It lives in the browser that passed, which is one reason to keep sessions and cookies instead of starting fresh on every page. ### What is the best way to handle Cloudflare scraping at scale? Run a real browser with consistent fingerprints and residential IPs, keep the pace polite, and let a service maintain it. On Scrape.ly that's the Scrape.ly browser: turn it on per Actor, rebuild, and your Playwright or Puppeteer code runs on it unchanged. About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head, disguised in a dark fedora and a raised collar, pulls a data card out of a browser window showing the Playwright logo and drops it into a crate](https://scrape.ly/media/playwright-stealth.png?w=750)](https://scrape.ly/blog/playwright-stealth.md) ### [Playwright Stealth: How It Works, Tested Setup, and When It Is Not Enough](https://scrape.ly/blog/playwright-stealth.md) [Playwright stealth means patching a headless browser so it hides the signs of automation. The Node and Python packages, who maintains them, what they patch, tested setup code with real bot.sannysoft.com results, and where stealth stops working.](https://scrape.ly/blog/playwright-stealth.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands thinking between a house with lit windows on the left and a server rack of the same height on the right, a cable with data packets running to it from each](https://scrape.ly/media/residential-vs-datacenter-proxies.png?w=750)](https://scrape.ly/blog/residential-vs-datacenter-proxies.md) ### [Residential vs Datacenter Proxies for Web Scraping: Cost, Speed and When to Use Each](https://scrape.ly/blog/residential-vs-datacenter-proxies.md) [Residential vs datacenter proxies: residential IPs come from home internet connections and look like ordinary visitors; datacenter IPs come from servers and are faster and cheaper. How they compare, when to use which, real per-GB prices and two real runs.](https://scrape.ly/blog/residential-vs-datacenter-proxies.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser window showing the green Selenium logo, which is cabled to a monitor showing the blue and yellow Python logo](https://scrape.ly/media/selenium-web-scraping.png?w=750)](https://scrape.ly/blog/selenium-web-scraping.md) ### [Selenium Web Scraping in Python: Step-by-Step Guide](https://scrape.ly/blog/selenium-web-scraping.md) [Selenium web scraping in Python, step by step: install, headless Chrome, locators, explicit waits, clicking through pages, infinite scroll, screenshots and CSV/JSON export, tested with Selenium 4.51 on 23 October 2026. Plus when Playwright fits better and how to run it at scale.](https://scrape.ly/blog/selenium-web-scraping.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.