CAPTCHA Bypass for Web Scraping: Why CAPTCHAs Appear and How Scrapers Handle Them

CAPTCHA bypass in web scraping means getting the public data you came for without tripping the checks that put a CAPTCHA in front of you, and handling the challenge cleanly when one shows up anyway. In practice, most of the work is prevention: a CAPTCHA is a site's response to traffic that looks risky, so a scraper that paces itself, uses good IPs and runs a believable browser sees far fewer of them.

This guide covers public pages only: listings, prices, articles, docs, anything a visitor sees without logging in. It is not about getting into accounts, creating sign-ups or pushing forms through in bulk. Below: the CAPTCHA types you'll meet, why they appear, how to prevent them, tested code that detects them, what solving services are and cost, and how Scrape.ly handles challenges for you.

The CAPTCHA types scrapers run into

Two trends shape this table. First, the visible puzzle is becoming the exception. Google says reCAPTCHA v3 "will never interrupt your users" and returns a score instead (reCAPTCHA v3 docs). Cloudflare says Turnstile runs "a series of small non-interactive JavaScript challenges", including proof-of-work and probing for web APIs, to "avoid showing a visual or interactive puzzle" (Turnstile docs). Second, the decision happens before any puzzle: by the time you see a checkbox, the site has already judged your traffic.

Why CAPTCHAs appear

A CAPTCHA is a risk decision. The site, or the service it uses, scores each request on signals like these:

• Where the request comes from. IP ranges that belong to cloud providers carry a lot of automated traffic, so they start with less trust than a home connection.
• What the client looks like. An HTTP library's TLS handshake and headers don't match any browser. A default headless browser reports navigator.webdriver as true and "HeadlessChrome" in its User-Agent.
• How it behaves. Bursts of requests, no cookies, perfectly regular timing and jumping straight to deep URLs all read as automation.
• What it's doing. Sites put CAPTCHAs on sign-ups, logins and checkouts far more than on product pages. Google's own examples include "filtering scrapers" on a homepage and stopping "credential stuffing attacks" on logins.

The scores are tuned per site. Google suggests a default threshold of 0.5 for reCAPTCHA v3 and tells site owners to adjust it to their traffic. That's why the same scraper can be fine on one site and challenged on every page of another.

CAPTCHA bypass starts with prevention

The cheapest CAPTCHA is the one that never appears. Each technique below lowers your risk score; together they make challenges rare on most public pages.

1. Slow down and spread the load

Rate is the first thing every system watches. Cap concurrency per site, add jittered delays, and back off when you see a 429 or a challenge instead of retrying instantly. Cache what you've already fetched. Our pillar guide to web scraping without getting blocked has tested code for backoff and caching.

2. Use IPs with a good reputation

Spread requests over several IPs so no single address carries all your traffic, and use residential IPs on sites that distrust datacenter ranges. Rotating proxies for web scraping explains per-request rotation, sticky sessions and which pool to pick.

3. Look like a real browser, consistently

If a page needs JavaScript, use a real browser engine, and make its signals agree with each other: the User-Agent with the client hints, the language with the proxy's country, the time zone with both. Stealth plugins patch the obvious tells; our Puppeteer stealth test shows what they fix and the new mismatches they can introduce.

4. Keep sessions and cookies

After you pass a check once, many systems remember it in a cookie. Cloudflare's cf_clearance cookie is the best-known example. Throwing cookies away on every request means getting checked again on every request. Keep one cookie jar per session and pair it with a sticky IP.

5. Fetch less

Check for an official API, a sitemap, an RSS feed or a data export. Skip pages you don't need. Every request you don't make is one that can't be challenged.

Detect a CAPTCHA before it breaks your scraper

Even with good prevention, you'll hit a challenge now and then. The worst outcome is not the challenge itself; it's a scraper that saves the CAPTCHA page as if it were data, or retries it in a tight loop. Detect it and slow down:

import re
import time

import requests

MARKERS = {
    "reCAPTCHA": re.compile(r"google\.com/recaptcha|g-recaptcha|grecaptcha", re.I),
    "hCaptcha": re.compile(r"hcaptcha\.com|h-captcha", re.I),
    "Turnstile": re.compile(r"challenges\.cloudflare\.com/turnstile|cf-turnstile", re.I),
}


def captcha_kind(resp):
    """Name the CAPTCHA or challenge in a response, or None if the page looks normal."""
    if resp.headers.get("cf-mitigated") == "challenge":
        return "Cloudflare challenge page"
    for kind, pattern in MARKERS.items():
        if pattern.search(resp.text):
            return kind
    return None


def polite_get(session, url, max_tries=3):
    for attempt in range(max_tries):
        resp = session.get(url, timeout=20)
        kind = captcha_kind(resp)
        if kind is None:
            return resp
        wait = 30 * (attempt + 1)
        print(f"{kind} on {url}: pausing {wait} s")
        time.sleep(wait)
    raise RuntimeError(f"Still challenged on {url}; slow down or switch to a real browser")

We ran captcha_kind on 26 October 2026 against three public pages, with requests 2.34.2 on Python 3.12:

200 https://books.toscrape.com/ -> None
200 https://www.google.com/recaptcha/api2/demo -> reCAPTCHA
200 https://accounts.hcaptcha.com/demo -> hCaptcha

A Turnstile widget snippet and a response carrying cf-mitigated: challenge were detected as Turnstile and Cloudflare challenge page. Note the status codes: both CAPTCHA demo pages answered 200. Status codes alone won't tell you that you've been challenged, which is why the check reads the body and headers. For Cloudflare, the cf-mitigated: challenge header is the documented marker of a challenge response.

When the challenge persists at a gentle pace, treat it as an answer. Pause the job, reduce the rate, or move that site to a real browser with a clean fingerprint and a residential IP. If it still won't load, the site doesn't want automated visitors on that path.

CAPTCHA solving services

When a CAPTCHA appears every time, regardless of how clean your traffic is, some scrapers turn to solving services. They work in roughly the same way: your code sends the CAPTCHA's site key and the page URL to the service's API, the service returns a token, and your browser submits that token where the widget would have. The tokens are short-lived; Google notes that "reCAPTCHA tokens expire after two minutes".

Prices are per thousand solves. 2Captcha's pricing page lists reCAPTCHA v2 at $1 to $2.99 per 1,000, Cloudflare Turnstile at $1.45 per 1,000 and simple image CAPTCHAs from $0.50 per 1,000, on 26 October 2026.

Before you add one, weigh the downsides:

• Terms. A CAPTCHA is an explicit signal that the site wants a human at that point. Solving it with a third party can break the site's terms. Read them first.
• Scope. Never use a solver to create accounts, log in, post content or buy in bulk. That is the abuse CAPTCHAs exist to stop, and it's out of scope for responsible scraping.
• Cost and speed. Every solve adds a fee and waiting time on top of the page load. At scale, prevention is far cheaper.
• Fragility. A solved token doesn't fix the signals that triggered the CAPTCHA. If your IP and fingerprint still look automated, the next page challenges you again.

For public pages, the better route is a browser setup that rarely gets challenged and passes the invisible checks on its own. That's what a hosted anti-bot browser does.

How Scrape.ly handles CAPTCHAs

Scrape.ly takes two approaches, depending on whether you run a ready-made scraper or your own code.

Ready-made Actors handle challenges themselves. Store Actors built for sites with bot checks clear them inside the run, so you don't need your own proxy or solver. The eBay Scraper README says it plainly: "the actor clears the challenge itself". In our reused run from 22 October 2026, it saved 10 keyboard listings in 35 seconds and its log reported the challenges along the way:

[scrapely.actor] INFO  Anti-bot solver source: platform.
[scrapely.proxy_configuration] INFO  Platform proxy access check succeeded.
[scrapely.actor] INFO  Reading 10 listings (10 found so far).
[scrapely.actor] INFO  Saved 10 listings. Challenge solves: 4

Those four solves cost nothing extra: the eBay Scraper charges $4.47 per 1,000 listings, platform usage included. The Scrape.ly Store has 570+ Actors like it, each maintained by its publisher as the site changes.

Your own code runs on the Scrape.ly browser. When you write a Playwright or Puppeteer scraper, turn on the Scrape.ly browser for the Actor (Settings → Scrapely Browser) and rebuild. Every browser launch then connects to a hosted Chromium with anti-bot fingerprints and its own residential proxy, and it solves Cloudflare Turnstile challenges by default. You set the location and the operating system with a browser configuration:

import { Actor, PlaywrightCrawler } from 'scrapely';

await Actor.main(async () => {
    const browserConfiguration = await Actor.createBrowserConfiguration({
        countryCode: 'US',
        state: 'California',
        os: 'windows',
        solveCloudflare: true,
    });

    const crawler = new PlaywrightCrawler({
        browserConfiguration,
        async requestHandler({ page }) {
            await Actor.pushData({ title: await page.title() });
        },
    });
    await crawler.run(['https://example.com']);
});

solveCloudflare is true unless you turn it off. Time zone and language follow the exit location, so the fingerprint agrees with the IP, the consistency problem that triggers so many challenges in the first place. The browser costs $0.45 per browser hour at list price, billed in 30-second blocks plus its traffic per MB, with no fee per solve. Its usage counts toward the run's maximum charge, so a stubborn page can't run up your bill. What is a cloud browser explains how this kind of hosted browser works, and the anti-blocking page shows how it fits with the managed proxy.

What to watch out for

• Challenge loops. If every page challenges you even in a real browser, look for an inconsistency: a time zone that doesn't match the IP, a flagged session you keep reusing, or too much speed. The Scrape.ly browser keeps location, language and time zone aligned by default.
• Silent scoring. With reCAPTCHA v3 and Turnstile there may be nothing to solve. A low score shows up as missing content or a blocked request, so validate the data you save, not just the status code.
• Blocks that aren't puzzles. A 403 from a block rule or an IP ban is the site owner's decision. Respect it.
• Personal data. Getting past a CAPTCHA doesn't change what you're allowed to collect. Stick to public, non-personal data, and see is web scraping legal for the legal background.
• Cost. Browsers are heavier than HTTP requests. Use them for the pages that need them and keep plain HTTP for the rest, the way the Website Content Crawler opens Chrome only when a page needs JavaScript.

FAQ

Is CAPTCHA bypass legal?

Reducing how often you're challenged, by pacing requests and using a normal browser, is ordinary engineering. Solving a CAPTCHA with a third-party service can break a site's terms, and getting around access controls can raise legal questions in some places. This is not legal advice: stick to public data, read the site's terms and robots.txt, and never automate logins, sign-ups or purchases.

Can Python requests solve a CAPTCHA?

No. reCAPTCHA, hCaptcha and Turnstile all run JavaScript in the browser, and requests can't execute it. What requests can do is detect the CAPTCHA, as in the code above, and stop. For pages that challenge every HTTP client, you need a real browser.

Why do I get a CAPTCHA on every request?

Usually because several risk signals stack up: a datacenter IP, an HTTP client's headers, no cookies and a fast request rate. Fix them in order of cost: slow down, keep cookies, switch to residential IPs, then move to a real browser with a consistent fingerprint.

Does Cloudflare use CAPTCHAs?

Not picture puzzles. Cloudflare's challenge pages and Turnstile run small JavaScript checks in the browser and sometimes ask for a click. Our guide to bypassing Cloudflare when scraping explains what it checks.

How much does a CAPTCHA solving service cost?

On 26 October 2026, 2Captcha listed $1 to $2.99 per 1,000 reCAPTCHA v2 solves and $1.45 per 1,000 Turnstile solves. On the Scrape.ly browser, Turnstile solving has no per-solve fee: you pay $0.45 per browser hour at list price plus traffic.

What is a reCAPTCHA v3 score?

A number from 0.0 to 1.0 that Google returns for each action: 1.0 is very likely human, 0.0 very likely a bot. The site decides what to do with it, and Google suggests starting with a threshold of 0.5.