HTTP client

scrapely.http sends requests that look like a real browser and deals with blocks for you. Use it for scraping where you would otherwise reach for curl_cffi, httpx or requests. For pages that don't need JavaScript, it uses far fewer resources than a browser. This page covers how the client works, requests and responses, headers and cookies, sessions, options, profiles, stats and migrating from curl_cffi.

scrapely.http is part of the SDK from version 0.1.38.

How it works

• One coherent fingerprint per request. The TLS ClientHello, the HTTP/2 settings, the User-Agent, the client hints (sec-ch-ua*) and the header order all come from one current browser profile. The default desktop Chrome profiles (153 to 156) send the same TLS (JA4), HTTP/2 and header order as real Chrome 156, with each version's own User-Agent, client hints and Accept.
• Identities. Each identity is a profile, a proxy session and a cookie jar. Concurrent requests never share an identity. When a site blocks one, the whole identity is retired, and the request is retried on a fresh one with a new IP, a new profile and an empty jar.
• Block handling. The client detects Cloudflare, PerimeterX, DataDome and Akamai challenges, 429 and 511 captcha pages and plain 403s. It is careful with markers that real pages also carry, such as __CF$cv$params, _pxAppId, px-captcha and DataDome cookies.
• Accept-Language matches the exit country of each identity.
  • Proxy with a country (country_code="DE"): each identity gets a language setup that is common there, such as de-DE,de;q=0.9, de-DE,de;q=0.9,en;q=0.8 or de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7.
  • Residential platform proxy without a country: each identity picks an exit country, weighted like real web traffic (about a third US, then GB, DE, FR, CA, BR and others). It asks the proxy for that country and sends that country's language, so the language always matches the IP.
  • locales: each identity uses one of them, the way a browser has one language setup. With a residential platform proxy without a country, the exit is in that locale's country (de-AT → AT, de → DE).
  • No proxy, your own proxy URLs, or DATACENTER (which exits in the US): en-US,en;q=0.9.
• URLs. A URL with a query and no path (https://host?q=1) is sent as https://host/?q=1, as browsers do.

Blocks and rotation

• Instant rotation. A 429, a block page, a captcha, a challenge, a 511, a 5xx or a network error moves the request to a brand-new identity at once. It gets a new proxy session (new IP), a new browser profile and an empty cookie jar, and the profile is a different one from the blocked identity's when there is a choice. The client does not wait and ignores Retry-After, because the block was tied to the old identity.
• Cap. A request rotates at most max_rotations times (default 6, so 7 attempts). After that it returns the last response, with .blocked set, or raises the last network error.
• Fixed proxy. A fresh identity gets a new exit IP only with proxy_configuration. With a fixed proxy URL, or with no proxy, a block rotates the profile and jar once. Network errors and 5xx keep the full cap.
• Polite mode (opt in). polite=True, or backoff=BackoffPolicy(...), brings back the waits:
  • a 429 waits for Retry-After, capped at 60 s, or else 1, 2, 4 … s per host plus jitter;
  • 5xx and network errors back off;
  • a PerimeterX 403 first gets one retry on the same identity;
  • a host that blocked more than 70% of its last 20 attempts gets one request at a time, 2–5 s apart.

Requirements

It needs the wreq package (Python 3.11+). The SDK depends on it, so it comes with every Actor build. If wreq can't be imported and curl_cffi is installed, the client uses curl_cffi and logs one warning.

Making requests

from scrapely import Actor
from scrapely.http import AsyncClient

async with Actor:
    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
    async with AsyncClient(proxy_configuration=proxy) as http:
        page = await http.get("https://www.example.com/search", params={"q": "laptop"})
        if page.blocked:                       # every attempt was blocked
            Actor.log.warning(f"blocked by {page.blocked}")
        api = await http.get("https://www.example.com/api/items", kind="json",
                             headers={"referer": "https://www.example.com/search"})
        items = api.json()
        r = await http.post("https://www.example.com/api/cart", json={"id": 1}, kind="json")

See Proxy configuration for the options of Actor.create_proxy_configuration().

Responses

The response has .status_code, .ok, .text, .content, .json(), .headers (case-insensitive), .url, .cookies and .raise_for_status(). It also has:

• .blocked: the block reason when every attempt was blocked, else None.
• .profile: the profile used, for example chrome156-win.
• .attempts.

A request that still fails with a network error after its retries raises scrapely.http.TransportError (alias RequestsError).

Request kinds

kind sets the headers that only depend on the type of request:

For "xhr" and "json", the referer defaults to the target's origin. A referer header from another site is cut down to its origin, the way browsers do it.

Headers and cookies

Headers you may set

You can set accept-language, cookie, referer, origin, content-type, authorization, x-* and your own app headers. accept works only with kind="xhr" or "json", because a page load always sends the browser's own Accept. Each header is placed where the browser would put it, not appended at the end.

user-agent, sec-ch-ua*, sec-fetch-*, accept-encoding, priority and upgrade-insecure-requests belong to the profile. If you pass them, they are ignored, with one warning per name. Pass allow_header_override=True to send them anyway; that breaks the fingerprint's consistency.

Cookies

A Cookie header, cookies={...} or http.cookies.set(name, value, domain=".example.com") is stored in the identity's jar, like document.cookie. Cookies set with http.cookies.set reach every new identity too. Cookies the site sets stay with the identity that received them, and identities are never saved across runs.

Sessions for multi-step flows

For a flow such as homepage, then search, then API with the same cookies, use a session. A session pins one identity:

async with http.session() as s:
    await s.get("https://www.example.com/")
    r = await s.get("https://www.example.com/api/search?q=x", kind="json")

If one of its requests is blocked, the session retires the identity and continues on a fresh one. That fresh identity starts with an empty jar. s.accept_language and s.country hold the current identity's language and exit country.

Options

Options are the same for AsyncClient and the synchronous Client:

fingerprint_options also accepts the camelCase form used by Crawlee-style code:

AsyncClient(fingerprintOptions={"fingerprintGeneratorOptions": {
    "browsers": [{"name": "chrome", "minVersion": 150}],
    "operatingSystems": ["windows", "macos"],
    "devices": ["desktop"],
    "locales": ["en-GB"],
}})

Profiles

The profiles are stored in scrapely/http/profiles.json.

• Default. Desktop Chrome 156, 155, 154 and 153 on Windows, macOS and Linux. Versions are weighted 55 / 30 / 10 / 5 and operating systems 65 / 27 / 8, like real traffic. These are verified profiles.
• Opt in, verified. Firefox 153 ESR and 152 (Windows, macOS, Linux). Their TLS and HTTP/2 equal real Firefox 153.6 ESR and 152 builds. Ask for them by name.
• Opt in, unverified. Edge 148, Safari 26.4 (macOS), Safari 26.2 (iOS) and Chrome 156, 155 and 154 (Android). Ask for them by name. With strict=True, only verified profiles are used.
• When nothing matches. If no profile matches your options, the client relaxes them in this order: version range, devices, operating systems, browsers. It logs a warning, and it never uses a browser it can't impersonate end to end.
• Updates. New profiles come with SDK releases. Rebuild the Actor to pick them up.

Stats

• http.stats() returns the counters: requests, blocks per vendor, retries, identities, bytes, and ok/blocked counts per profile.
• On close, the client logs one line, for example: HTTP: 120 requests, 3 blocked (cloudflare 2, 429 1), 3 rotations (cloudflare 2, 429 1), 5 identities, profiles ok/blocked chrome156-win 80/2 …. stats() and HTTP_STATS also hold rotations, rotationsBy (per reason or vendor) and rotationsByHost.
• On close, the totals for the run also go to the default key-value store as HTTP_STATS.
• AsyncSession and Session (the curl_cffi-style classes below) are often opened by the dozen. They log the run total once, when the process exits, instead of one line per session.

Migrating from curl_cffi

Change the import; AsyncSession and Session accept impersonate=, proxy=, proxies=, timeout=, headers=, verify= and cookies=:

from scrapely.http import AsyncSession      # was: from curl_cffi.requests import AsyncSession

async with AsyncSession(impersonate="chrome", proxy=proxy_url, timeout=30) as s:
    r = await s.get(url)

A session keeps one identity, and so one jar and one proxy, like a curl_cffi session. On a block it switches to a fresh identity. The version in impersonate is ignored, so "chrome99" still sends the current Chrome profile, because old versions get scored as bots. "safari_ios" and "chrome_android" choose the mobile profiles.

Synchronous code

Client has the same API for thread-based Actors:

from scrapely.http import Client

with Client(proxy=proxy_url) as http:
    r = http.get(url)

What's next

• Environment & proxy covers the platform proxy and the Scrape.ly browser for pages that need JavaScript.
• Storages covers the default key-value store where HTTP_STATS lands.
• The API reference lists every signature.