--- title: "CAPTCHA Bypass for Web Scraping | Scrape.ly" description: "Captcha bypass for web scraping, done responsibly: why CAPTCHAs appear, the main types, how to prevent them, solving services, and tested detection code." url: https://scrape.ly/blog/captcha-solving-for-scraping markdown: https://scrape.ly/blog/captcha-solving-for-scraping.md --- [Guides](https://scrape.ly/blog/category/guides.md) # CAPTCHA Bypass for Web Scraping: Why CAPTCHAs Appear and How Scrapers Handle Them CAPTCHA bypass for scrapers, done responsibly: the CAPTCHA types you'll meet, why they appear, how proxies, fingerprints and pacing prevent most of them, tested detection code, what solving services cost, and a hosted browser that solves Cloudflare Turnstile for you. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 26, 2026 - Reading time 9 min read ![Pixel-art robot with an orange >\_ on its terminal head walks away from a solved CAPTCHA panel, its grid of sky-and-road picture tiles with traffic lights, three tiles picked in orange and its checkbox ticked green, carrying a data card towards a crate](https://scrape.ly/media/captcha-solving-for-scraping.png?w=1080) CAPTCHA bypass in web scraping means getting the public data you came for without tripping the checks that put a CAPTCHA in front of you, and handling the challenge cleanly when one shows up anyway. In practice, most of the work is prevention: a CAPTCHA is a site's response to traffic that looks risky, so a scraper that paces itself, uses good IPs and runs a believable browser sees far fewer of them. This guide covers public pages only: listings, prices, articles, docs, anything a visitor sees without logging in. It is not about getting into accounts, creating sign-ups or pushing forms through in bulk. Below: the CAPTCHA types you'll meet, why they appear, how to prevent them, tested code that detects them, what solving services are and cost, and how Scrape.ly handles challenges for you. ## The CAPTCHA types scrapers run into | Type | What the visitor sees | What decides it | What a basic scraper gets | | ------------------------- | --------------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------- | | Text or image CAPTCHA | Distorted letters or a picture to read | A fixed puzzle on the page | A form it can't fill in | | reCAPTCHA v2 | "I'm not a robot" checkbox, sometimes an image grid | Google's risk analysis, then the puzzle | An iframe and a `g-recaptcha` widget instead of content | | reCAPTCHA v3 | Nothing | A score from 0.0 (bot) to 1.0 (human) per action | Pages that silently fail, block or degrade | | hCaptcha | Checkbox or image task | hCaptcha's risk check | An `h-captcha` widget in the form | | Cloudflare Turnstile | Usually nothing, sometimes a checkbox | Small JavaScript checks in the browser | A `cf-turnstile` widget it can't run | | Cloudflare challenge page | "Just a moment…" interstitial | Cloudflare's bot score and JavaScript detections | A `403` with `cf-mitigated: challenge` | Two trends shape this table. First, the visible puzzle is becoming the exception. Google says reCAPTCHA v3 "will never interrupt your users" and returns a score instead ([reCAPTCHA v3 docs](https://developers.google.com/recaptcha/docs/v3)). Cloudflare says Turnstile runs "a series of small non-interactive JavaScript challenges", including proof-of-work and probing for web APIs, to "avoid showing a visual or interactive puzzle" ([Turnstile docs](https://developers.cloudflare.com/turnstile/)). Second, the decision happens before any puzzle: by the time you see a checkbox, the site has already judged your traffic. ## Why CAPTCHAs appear A CAPTCHA is a risk decision. The site, or the service it uses, scores each request on signals like these: - **Where the request comes from.** IP ranges that belong to cloud providers carry a lot of automated traffic, so they start with less trust than a home connection. - **What the client looks like.** An HTTP library's TLS handshake and headers don't match any browser. A default headless browser reports `navigator.webdriver` as `true` and "HeadlessChrome" in its User-Agent. - **How it behaves.** Bursts of requests, no cookies, perfectly regular timing and jumping straight to deep URLs all read as automation. - **What it's doing.** Sites put CAPTCHAs on sign-ups, logins and checkouts far more than on product pages. Google's own examples include "filtering scrapers" on a homepage and stopping "credential stuffing attacks" on logins. The scores are tuned per site. Google suggests a default threshold of 0.5 for reCAPTCHA v3 and tells site owners to adjust it to their traffic. That's why the same scraper can be fine on one site and challenged on every page of another. ## CAPTCHA bypass starts with prevention The cheapest CAPTCHA is the one that never appears. Each technique below lowers your risk score; together they make challenges rare on most public pages. ### 1. Slow down and spread the load Rate is the first thing every system watches. Cap concurrency per site, add jittered delays, and back off when you see a `429` or a challenge instead of retrying instantly. Cache what you've already fetched. Our pillar guide to [web scraping without getting blocked](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) has tested code for backoff and caching. ### 2. Use IPs with a good reputation Spread requests over several IPs so no single address carries all your traffic, and use residential IPs on sites that distrust datacenter ranges. [Rotating proxies for web scraping](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) explains per-request rotation, sticky sessions and which pool to pick. ### 3. Look like a real browser, consistently If a page needs JavaScript, use a real browser engine, and make its signals agree with each other: the User-Agent with the client hints, the language with the proxy's country, the time zone with both. Stealth plugins patch the obvious tells; our [Puppeteer stealth](https://scrape.ly/blog/puppeteer-stealth.md) test shows what they fix and the new mismatches they can introduce. ### 4. Keep sessions and cookies After you pass a check once, many systems remember it in a cookie. Cloudflare's `cf_clearance` cookie is the best-known example. Throwing cookies away on every request means getting checked again on every request. Keep one cookie jar per session and pair it with a sticky IP. ### 5. Fetch less Check for an official API, a sitemap, an RSS feed or a data export. Skip pages you don't need. Every request you don't make is one that can't be challenged. ## Detect a CAPTCHA before it breaks your scraper Even with good prevention, you'll hit a challenge now and then. The worst outcome is not the challenge itself; it's a scraper that saves the CAPTCHA page as if it were data, or retries it in a tight loop. Detect it and slow down: ```python import re import time import requests MARKERS = { "reCAPTCHA": re.compile(r"google\.com/recaptcha|g-recaptcha|grecaptcha", re.I), "hCaptcha": re.compile(r"hcaptcha\.com|h-captcha", re.I), "Turnstile": re.compile(r"challenges\.cloudflare\.com/turnstile|cf-turnstile", re.I), } def captcha_kind(resp): """Name the CAPTCHA or challenge in a response, or None if the page looks normal.""" if resp.headers.get("cf-mitigated") == "challenge": return "Cloudflare challenge page" for kind, pattern in MARKERS.items(): if pattern.search(resp.text): return kind return None def polite_get(session, url, max_tries=3): for attempt in range(max_tries): resp = session.get(url, timeout=20) kind = captcha_kind(resp) if kind is None: return resp wait = 30 * (attempt + 1) print(f"{kind} on {url}: pausing {wait} s") time.sleep(wait) raise RuntimeError(f"Still challenged on {url}; slow down or switch to a real browser") ``` We ran `captcha_kind` on 26 October 2026 against three public pages, with `requests` 2.34.2 on Python 3.12: ```plaintext 200 https://books.toscrape.com/ -> None 200 https://www.google.com/recaptcha/api2/demo -> reCAPTCHA 200 https://accounts.hcaptcha.com/demo -> hCaptcha ``` A Turnstile widget snippet and a response carrying `cf-mitigated: challenge` were detected as `Turnstile` and `Cloudflare challenge page`. Note the status codes: both CAPTCHA demo pages answered `200`. Status codes alone won't tell you that you've been challenged, which is why the check reads the body and headers. For Cloudflare, the `cf-mitigated: challenge` header is the documented marker of a [challenge response](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/). When the challenge persists at a gentle pace, treat it as an answer. Pause the job, reduce the rate, or move that site to a real browser with a clean fingerprint and a residential IP. If it still won't load, the site doesn't want automated visitors on that path. ## CAPTCHA solving services When a CAPTCHA appears every time, regardless of how clean your traffic is, some scrapers turn to solving services. They work in roughly the same way: your code sends the CAPTCHA's site key and the page URL to the service's API, the service returns a token, and your browser submits that token where the widget would have. The tokens are short-lived; Google notes that "reCAPTCHA tokens expire after two minutes". Prices are per thousand solves. 2Captcha's [pricing page](https://2captcha.com/pricing) lists reCAPTCHA v2 at $1 to $2.99 per 1,000, Cloudflare Turnstile at $1.45 per 1,000 and simple image CAPTCHAs from $0.50 per 1,000, on 26 October 2026. Before you add one, weigh the downsides: - **Terms.** A CAPTCHA is an explicit signal that the site wants a human at that point. Solving it with a third party can break the site's terms. Read them first. - **Scope.** Never use a solver to create accounts, log in, post content or buy in bulk. That is the abuse CAPTCHAs exist to stop, and it's out of scope for responsible scraping. - **Cost and speed.** Every solve adds a fee and waiting time on top of the page load. At scale, prevention is far cheaper. - **Fragility.** A solved token doesn't fix the signals that triggered the CAPTCHA. If your IP and fingerprint still look automated, the next page challenges you again. For public pages, the better route is a browser setup that rarely gets challenged and passes the invisible checks on its own. That's what a hosted anti-bot browser does. ## How Scrape.ly handles CAPTCHAs Scrape.ly takes two approaches, depending on whether you run a ready-made scraper or your own code. **Ready-made Actors handle challenges themselves.** Store Actors built for sites with bot checks clear them inside the run, so you don't need your own proxy or solver. The [eBay Scraper](https://scrape.ly/store/shelfbright/ebay-scraper.md) README says it plainly: "the actor clears the challenge itself". In our reused run from 22 October 2026, it saved 10 keyboard listings in 35 seconds and its log reported the challenges along the way: ```plaintext [scrapely.actor] INFO Antibot solver source: platform. [scrapely.proxy_configuration] INFO Platform proxy access check succeeded. [scrapely.actor] INFO Reading 10 listings (10 found so far). [scrapely.actor] INFO Saved 10 listings. Challenge solves: 4 ``` Those four solves cost nothing extra: the eBay Scraper charges $4.47 per 1,000 listings, platform usage included. The Scrape.ly Store has 570+ Actors like it, each maintained by its publisher as the site changes. **Your own code runs on the Scrape.ly browser.** When you write a Playwright or Puppeteer scraper, turn on the [Scrape.ly browser](https://scrape.ly/docs/browser.md) for the Actor (**Settings** → **Scrapely Browser**) and rebuild. Every browser launch then connects to a hosted Chromium with anti-bot fingerprints and its own residential proxy, and it solves Cloudflare Turnstile challenges by default. You set the location and the operating system with a browser configuration: ```javascript import { Actor, PlaywrightCrawler } from 'scrapely'; await Actor.main(async () => { const browserConfiguration = await Actor.createBrowserConfiguration({ countryCode: 'US', state: 'California', os: 'windows', solveCloudflare: true, }); const crawler = new PlaywrightCrawler({ browserConfiguration, async requestHandler({ page }) { await Actor.pushData({ title: await page.title() }); }, }); await crawler.run(['https://example.com']); }); ``` `solveCloudflare` is `true` unless you turn it off. Time zone and language follow the exit location, so the fingerprint agrees with the IP, the consistency problem that triggers so many challenges in the first place. The browser costs $0.45 per browser hour at list price, billed in 30-second blocks plus its traffic per MB, with no fee per solve. Its usage counts toward the run's maximum charge, so a stubborn page can't run up your bill. [What is a cloud browser](https://scrape.ly/blog/what-is-a-cloud-browser.md) explains how this kind of hosted browser works, and the [anti-blocking page](https://scrape.ly/anti-blocking.md) shows how it fits with the managed proxy. Scrape.ly Store Let the Actor clear the bot check Public eBay listings with the challenge handled inside the run: $4.47 per 1,000 listings, platform usage included, on $5 of free monthly credit. [Try it free](https://scrape.ly/store/shelfbright/ebay-scraper.md) [Browse the Store](https://scrape.ly/store.md) ## What to watch out for - **Challenge loops.** If every page challenges you even in a real browser, look for an inconsistency: a time zone that doesn't match the IP, a flagged session you keep reusing, or too much speed. The Scrape.ly browser keeps location, language and time zone aligned by default. - **Silent scoring.** With reCAPTCHA v3 and Turnstile there may be nothing to solve. A low score shows up as missing content or a blocked request, so validate the data you save, not just the status code. - **Blocks that aren't puzzles.** A `403` from a block rule or an IP ban is the site owner's decision. Respect it. - **Personal data.** Getting past a CAPTCHA doesn't change what you're allowed to collect. Stick to public, non-personal data, and see [is web scraping legal](https://scrape.ly/blog/is-web-scraping-legal.md) for the legal background. - **Cost.** Browsers are heavier than HTTP requests. Use them for the pages that need them and keep plain HTTP for the rest, the way the [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) opens Chrome only when a page needs JavaScript. ## FAQ ### Is CAPTCHA bypass legal? Reducing how often you're challenged, by pacing requests and using a normal browser, is ordinary engineering. Solving a CAPTCHA with a third-party service can break a site's terms, and getting around access controls can raise legal questions in some places. This is not legal advice: stick to public data, read the site's terms and robots.txt, and never automate logins, sign-ups or purchases. ### Can Python requests solve a CAPTCHA? No. reCAPTCHA, hCaptcha and Turnstile all run JavaScript in the browser, and `requests` can't execute it. What `requests` can do is detect the CAPTCHA, as in the code above, and stop. For pages that challenge every HTTP client, you need a real browser. ### Why do I get a CAPTCHA on every request? Usually because several risk signals stack up: a datacenter IP, an HTTP client's headers, no cookies and a fast request rate. Fix them in order of cost: slow down, keep cookies, switch to residential IPs, then move to a real browser with a consistent fingerprint. ### Does Cloudflare use CAPTCHAs? Not picture puzzles. Cloudflare's challenge pages and Turnstile run small JavaScript checks in the browser and sometimes ask for a click. Our guide to [bypassing Cloudflare when scraping](https://scrape.ly/blog/bypass-cloudflare-scraping.md) explains what it checks. ### How much does a CAPTCHA solving service cost? On 26 October 2026, 2Captcha listed $1 to $2.99 per 1,000 reCAPTCHA v2 solves and $1.45 per 1,000 Turnstile solves. On the Scrape.ly browser, Turnstile solving has no per-solve fee: you pay $0.45 per browser hour at list price plus traffic. ### What is a reCAPTCHA v3 score? A number from 0.0 to 1.0 that Google returns for each action: 1.0 is very likely human, 0.0 very likely a bot. The site decides what to do with it, and Google suggests starting with a threshold of 0.5. About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a monitor showing the Java coffee cup logo with orange steam above lines of code, and drops it into a crate](https://scrape.ly/media/java-web-scraping.png?w=750)](https://scrape.ly/blog/java-web-scraping.md) ### [Java Web Scraping: Jsoup, HttpClient, Playwright and Selenium (Tested Code)](https://scrape.ly/blog/java-web-scraping.md) [A practical guide to Java web scraping: Jsoup for HTML, java.net.http for polite crawling with retries, virtual threads for speed, Playwright and Selenium for JavaScript pages, and calling cloud scrapers from Java. Every snippet tested on JDK 25 on 26 October 2026.](https://scrape.ly/blog/java-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head raises a wrench beside a workbench where a half-built scraper machine with a small screen, a mast and a hanging claw stands among loose parts: a gear, a chip and bolts, a cardboard box of gears at its side](https://scrape.ly/media/how-to-build-a-web-scraper.png?w=750)](https://scrape.ly/blog/how-to-build-a-web-scraper.md) ### [How to Build a Web Scraper: 8 Steps From First Request to the Cloud](https://scrape.ly/blog/how-to-build-a-web-scraper.md) [How to build a web scraper that keeps working: define the data, inspect the site, fetch politely, parse and validate, follow pagination, store in SQLite with price history, schedule it and scale it. Tested Python code, plus when a ready-made cloud scraper beats building your own.](https://scrape.ly/blog/how-to-build-a-web-scraper.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser page whose title bar has been lifted off and floats above it, a dotted line where it sat, and drops it into a crate](https://scrape.ly/media/what-is-a-headless-browser.png?w=750)](https://scrape.ly/blog/what-is-a-headless-browser.md) ### [What Is a Headless Browser? Uses, Tools and Detection](https://scrape.ly/blog/what-is-a-headless-browser.md) [A headless browser is a real browser without a visible window, driven from code. Headless vs headed, how Chrome's headless mode works today, tested Playwright, Puppeteer and Selenium examples, how sites detect headless browsers, and when you need one.](https://scrape.ly/blog/what-is-a-headless-browser.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.