---
title: "HTTP client · Scrape.ly Docs"
description: "Browser-like HTTP requests with identities, block detection, and retries."
url: https://scrape.ly/docs/sdk-python-http-client
markdown: https://scrape.ly/docs/sdk-python-http-client.md
---

# HTTP client

> Browser-like HTTP requests with identities, block detection, and retries.

`scrapely.http` sends requests that look like a real browser and deals with blocks for you. Use it for scraping where you would otherwise reach for `curl_cffi`, `httpx` or `requests`. For pages that don't need JavaScript, it uses far fewer resources than a browser. This page covers how the client works, requests and responses, headers and cookies, sessions, options, profiles, stats and migrating from curl_cffi.

`scrapely.http` is part of the SDK from version 0.1.38.

## How it works

- **One coherent fingerprint per request.** The TLS ClientHello, the HTTP/2 settings, the User-Agent, the client hints (`sec-ch-ua*`) and the header order all come from one current browser profile. The default desktop Chrome profiles (153 to 156) send the same TLS (JA4), HTTP/2 and header order as real Chrome 156, with each version's own User-Agent, client hints and Accept.
- **Identities.** Each identity is a profile, a proxy session and a cookie jar. Concurrent requests never share an identity. When a site blocks one, the whole identity is retired, and the request is retried on a fresh one with a new IP, a new profile and an empty jar.
- **Block handling.** The client detects Cloudflare, PerimeterX, DataDome and Akamai challenges, `429` and `511` captcha pages and plain `403`s. It is careful with markers that real pages also carry, such as `__CF$cv$params`, `_pxAppId`, `px-captcha` and DataDome cookies.
- **Accept-Language matches the exit country of each identity.**
  - **Proxy with a country** (`country_code="DE"`): each identity gets a language setup that is common there, such as `de-DE,de;q=0.9`, `de-DE,de;q=0.9,en;q=0.8` or `de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7`.
  - **Residential platform proxy without a country:** each identity picks an exit country, weighted like real web traffic (about a third US, then GB, DE, FR, CA, BR and others). It asks the proxy for that country and sends that country's language, so the language always matches the IP.
  - **`locales`:** each identity uses one of them, the way a browser has one language setup. With a residential platform proxy without a country, the exit is in that locale's country (`de-AT` → AT, `de` → DE).
  - **No proxy, your own proxy URLs, or `DATACENTER`** (which exits in the US): `en-US,en;q=0.9`.
- **URLs.** A URL with a query and no path (`https://host?q=1`) is sent as `https://host/?q=1`, as browsers do.

### Blocks and rotation

- **Instant rotation.** A `429`, a block page, a captcha, a challenge, a `511`, a `5xx` or a network error moves the request to a brand-new identity at once. It gets a new proxy session (new IP), a new browser profile and an empty cookie jar, and the profile is a different one from the blocked identity's when there is a choice. The client does not wait and ignores `Retry-After`, because the block was tied to the old identity.
- **Cap.** A request rotates at most `max_rotations` times (default 6, so 7 attempts). After that it returns the last response, with `.blocked` set, or raises the last network error.
- **Fixed proxy.** A fresh identity gets a new exit IP only with `proxy_configuration`. With a fixed `proxy` URL, or with no proxy, a block rotates the profile and jar once. Network errors and `5xx` keep the full cap.
- **Polite mode (opt in).** `polite=True`, or `backoff=BackoffPolicy(...)`, brings back the waits:
  - a `429` waits for `Retry-After`, capped at 60 s, or else 1, 2, 4 … s per host plus jitter;
  - `5xx` and network errors back off;
  - a PerimeterX `403` first gets one retry on the same identity;
  - a host that blocked more than 70% of its last 20 attempts gets one request at a time, 2–5 s apart.

### Requirements

It needs the `wreq` package (Python 3.11+). The SDK depends on it, so it comes with every Actor build. If `wreq` can't be imported and `curl_cffi` is installed, the client uses `curl_cffi` and logs one warning.

## Making requests

```python
from scrapely import Actor
from scrapely.http import AsyncClient

async with Actor:
    proxy = await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="DE")
    async with AsyncClient(proxy_configuration=proxy) as http:
        page = await http.get("https://www.example.com/search", params={"q": "laptop"})
        if page.blocked:                       # every attempt was blocked
            Actor.log.warning(f"blocked by {page.blocked}")
        api = await http.get("https://www.example.com/api/items", kind="json",
                             headers={"referer": "https://www.example.com/search"})
        items = api.json()
        r = await http.post("https://www.example.com/api/cart", json={"id": 1}, kind="json")
```

See [Proxy configuration](https://scrape.ly/docs/sdk-python-environment.md#proxy-configuration) for the options of `Actor.create_proxy_configuration()`.

### Responses

The response has `.status_code`, `.ok`, `.text`, `.content`, `.json()`, `.headers` (case-insensitive), `.url`, `.cookies` and `.raise_for_status()`. It also has:

- `.blocked`: the block reason when every attempt was blocked, else `None`.
- `.profile`: the profile used, for example `chrome156-win`.
- `.attempts`.

A request that still fails with a network error after its retries raises `scrapely.http.TransportError` (alias `RequestsError`).

### Request kinds

`kind` sets the headers that only depend on the type of request:

| `kind` | Request |
| --- | --- |
| `"navigate"` (default) | A page load. |
| `"xhr"` | A `fetch()` from the page. It sends `accept: */*` and `sec-fetch-mode: cors` and drops the navigation-only headers. |
| `"json"` | Like `"xhr"`, with an `application/json` accept. |

For `"xhr"` and `"json"`, the referer defaults to the target's origin. A `referer` header from another site is cut down to its origin, the way browsers do it.

## Headers and cookies

### Headers you may set

You can set `accept-language`, `cookie`, `referer`, `origin`, `content-type`, `authorization`, `x-*` and your own app headers. `accept` works only with `kind="xhr"` or `"json"`, because a page load always sends the browser's own Accept. Each header is placed where the browser would put it, not appended at the end.

`user-agent`, `sec-ch-ua*`, `sec-fetch-*`, `accept-encoding`, `priority` and `upgrade-insecure-requests` belong to the profile. If you pass them, they are ignored, with one warning per name. Pass `allow_header_override=True` to send them anyway; that breaks the fingerprint's consistency.

### Cookies

A `Cookie` header, `cookies={...}` or `http.cookies.set(name, value, domain=".example.com")` is stored in the identity's jar, like `document.cookie`. Cookies set with `http.cookies.set` reach every new identity too. Cookies the site sets stay with the identity that received them, and identities are never saved across runs.

## Sessions for multi-step flows

For a flow such as homepage, then search, then API with the same cookies, use a session. A session pins one identity:

```python
async with http.session() as s:
    await s.get("https://www.example.com/")
    r = await s.get("https://www.example.com/api/search?q=x", kind="json")
```

If one of its requests is blocked, the session retires the identity and continues on a fresh one. That fresh identity starts with an empty jar. `s.accept_language` and `s.country` hold the current identity's language and exit country.

## Options

Options are the same for `AsyncClient` and the synchronous `Client`:

| Option | Default | Meaning |
| --- | --- | --- |
| `proxy_configuration` | `None` | A `ProxyConfiguration`. Each identity gets its own proxy session from `new_url(session_id)`. |
| `proxy` / `proxies` | `None` | A fixed proxy URL (or a `{"https": url}` dict). Identities then differ only in profile and jar. |
| `fingerprint_options` | Chrome desktop | `browsers` (`chrome`, `edge`, `firefox`, `safari`, or `{"name": "chrome", "min_version": 150}`), `operating_systems` (`windows`, `macos`, `linux`, `android`, `ios`), `devices` (`desktop`, `mobile`), `locales`, `strict`, `http_version`. |
| `max_rotations` | `6` | Fresh identities per request after the first attempt. `max_retries` is accepted as an older name. Both can also be passed per request. |
| `polite` / `backoff` | off | Opt in to `Retry-After`, per-host backoff, the PerimeterX same-identity retry and the slow mode. `backoff=BackoffPolicy(honour_retry_after=..., max_retry_after=60, backoff_base=1, backoff_cap=30, jitter=1, circuit_breaker=True, px_same_identity_retries=1)` tunes them. |
| `timeout` | `30` | Seconds per attempt. |
| `max_concurrency` | `8` | Identity pool size, which is also the number of requests in flight. |
| `max_usage` / `max_age` | `50` / `1800` | An identity is retired after this many requests or seconds. |
| `sticky` | `True` | `False` uses a new identity for every request. |
| `min_delay` / `max_delay` | `0` | Random pause between two requests of one identity, in seconds. |
| `is_blocked` | `None` | `callable(response)`. A truthy value (or a reason string) counts as a block. It can also be passed per request. |
| `headers` | `None` | Default caller headers for every request. |
| `verify` | `True` | Verify TLS certificates. |
| `backend` | `"auto"` | `"wreq"`, or `"curl_cffi"` for sites that need HTTP/3. |

`fingerprint_options` also accepts the camelCase form used by Crawlee-style code:

```python
AsyncClient(fingerprintOptions={"fingerprintGeneratorOptions": {
    "browsers": [{"name": "chrome", "minVersion": 150}],
    "operatingSystems": ["windows", "macos"],
    "devices": ["desktop"],
    "locales": ["en-GB"],
}})
```

## Profiles

The profiles are stored in `scrapely/http/profiles.json`.

- **Default.** Desktop Chrome 156, 155, 154 and 153 on Windows, macOS and Linux. Versions are weighted 55 / 30 / 10 / 5 and operating systems 65 / 27 / 8, like real traffic. These are verified profiles.
- **Opt in, verified.** Firefox 153 ESR and 152 (Windows, macOS, Linux). Their TLS and HTTP/2 equal real Firefox 153.6 ESR and 152 builds. Ask for them by name.
- **Opt in, unverified.** Edge 148, Safari 26.4 (macOS), Safari 26.2 (iOS) and Chrome 156, 155 and 154 (Android). Ask for them by name. With `strict=True`, only verified profiles are used.
- **When nothing matches.** If no profile matches your options, the client relaxes them in this order: version range, devices, operating systems, browsers. It logs a warning, and it never uses a browser it can't impersonate end to end.
- **Updates.** New profiles come with SDK releases. Rebuild the Actor to pick them up.

## Stats

- `http.stats()` returns the counters: requests, blocks per vendor, retries, identities, bytes, and ok/blocked counts per profile.
- On close, the client logs one line, for example: `HTTP: 120 requests, 3 blocked (cloudflare 2, 429 1), 3 rotations (cloudflare 2, 429 1), 5 identities, profiles ok/blocked chrome156-win 80/2 …`. `stats()` and `HTTP_STATS` also hold `rotations`, `rotationsBy` (per reason or vendor) and `rotationsByHost`.
- On close, the totals for the run also go to the default key-value store as `HTTP_STATS`.
- `AsyncSession` and `Session` (the curl_cffi-style classes below) are often opened by the dozen. They log the run total once, when the process exits, instead of one line per session.

> [!NOTE]
> Always close the client, with `async with` or `await http.aclose()`.

## Migrating from curl_cffi

Change the import; `AsyncSession` and `Session` accept `impersonate=`, `proxy=`, `proxies=`, `timeout=`, `headers=`, `verify=` and `cookies=`:

```python
from scrapely.http import AsyncSession      # was: from curl_cffi.requests import AsyncSession

async with AsyncSession(impersonate="chrome", proxy=proxy_url, timeout=30) as s:
    r = await s.get(url)
```

A session keeps one identity, and so one jar and one proxy, like a curl_cffi session. On a block it switches to a fresh identity. The version in `impersonate` is ignored, so `"chrome99"` still sends the current Chrome profile, because old versions get scored as bots. `"safari_ios"` and `"chrome_android"` choose the mobile profiles.

## Synchronous code

`Client` has the same API for thread-based Actors:

```python
from scrapely.http import Client

with Client(proxy=proxy_url) as http:
    r = http.get(url)
```

## What's next

- [Environment & proxy](https://scrape.ly/docs/sdk-python-environment.md) covers the platform proxy and the Scrape.ly browser for pages that need JavaScript.
- [Storages](https://scrape.ly/docs/sdk-python-storages.md) covers the default key-value store where `HTTP_STATS` lands.
- The [API reference](https://scrape.ly/docs/sdk-python-reference.md) lists every signature.

---

- Previous: [Environment & proxy](https://scrape.ly/docs/sdk-python-environment.md)
- Next: [Errors & deployment](https://scrape.ly/docs/sdk-python-deploy.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)