--- title: "Rotating Proxies for Web Scraping | Scrape.ly" description: "Rotating proxies for web scraping: per-request vs sticky rotation, residential vs datacenter, tested Python and Crawlee code, and real per-GB prices." url: https://scrape.ly/blog/rotating-proxies-for-web-scraping markdown: https://scrape.ly/blog/rotating-proxies-for-web-scraping.md --- [Guides](https://scrape.ly/blog/category/guides.md) # Rotating Proxies for Web Scraping: How Rotation Works, with Tested Code Rotating proxies for web scraping spread your requests over many IP addresses. Per-request rotation vs sticky sessions, residential vs datacenter pools, tested Python and Crawlee code, and what rotation costs per GB on Scrape.ly and Evomi. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 26, 2026 - Reading time 8 min read ![Pixel-art robot with an orange >\_ on its terminal head carries a cable from the one orange-lit globe in a ring of four blue-and-green globes turning round a small server, amber arrowheads marking the rotation, towards a web page with an image block](https://scrape.ly/media/rotating-proxies-for-web-scraping.png?w=1080) Rotating proxies for web scraping spread your requests across many IP addresses, so no single address sends enough traffic to get rate-limited or blocked. Instead of every request leaving from your server's one IP, each request, or each session, goes out through a different proxy from a pool. Rotation is one of the most effective anti-blocking techniques, and one of the easiest to get wrong: rotate too little and an IP gets flagged, rotate too much and you break the sessions a site expects. Below: how rotation works, per-request rotation vs sticky sessions, which pool to rotate through, Python and Crawlee code we tested on 26 October 2026, and what rotating proxies cost on Scrape.ly and on Evomi, its sister proxy network. ## Rotating proxies for web scraping at a glance | Approach | How it works | Good for | Watch out for | | ------------------------------ | ---------------------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------- | | Per-request rotation | Every request uses the next IP | Independent pages: listings, search results, product pages | Breaks logins, carts and multi-step flows | | Sticky sessions | One IP for a sequence of requests, then a new one | Pagination, sites that tie cookies to an IP | One flagged IP affects the whole session | | Your own proxy list | Your code picks from a list of proxy URLs | Full control, mixing providers | You handle failures and bans yourself | | Rotating gateway (backconnect) | One endpoint; the provider assigns an IP per request or per session ID | Large pools without managing addresses | Session syntax differs by provider | ## How rotating proxies work A proxy is a server that forwards your request, so the website sees the proxy's IP instead of yours. Rotation means you don't stick to one. There are two ways to do it. **You rotate.** You hold a list of proxy URLs and your code picks the next one for each request. Every address is visible to you, and you decide when to drop a bad one. **The provider rotates.** You send all traffic to a single gateway address, and the provider picks an exit IP from its pool for each connection. To keep the same IP for a while, you pass a session ID, usually inside the proxy username. Scrape.ly's managed proxy works this way: its SDK encodes options in the username, for example `groups-RESIDENTIAL,session-abc,country-US`, and a session ID keeps the same exit IP for that session ([proxy docs](https://scrape.ly/docs/proxy.md)). Either way, the website sees requests arriving from many addresses, each carrying only a small share of the traffic. ## Per-request rotation vs sticky sessions The choice comes down to whether a site expects your requests to belong together. Use **per-request rotation** when each page stands on its own. A product page doesn't care which IP fetched the previous one. Spreading 1,000 product URLs over a pool keeps every IP under the site's rate limits. Use **sticky sessions** when requests form a sequence. Many sites set a cookie on the first visit and expect it back from the same IP; a cookie that suddenly arrives from another country looks like a stolen session. Pagination, filtered searches and anything with a consent or region cookie works better on one IP per session. Rotate to a new session when the old one gets blocked or after a fixed number of pages. Most production scrapers mix both: a pool of sessions, each sticky for a handful of requests, rotating out as soon as one gets a `403`, a `429` or a CAPTCHA. That's exactly how Crawlee's session pool works, as the code below shows. ## Residential or datacenter: which pool to rotate Rotation decides how traffic spreads; the pool decides what each IP looks like. - **Datacenter IPs** come from servers. They're fast and cheap, and many public sites serve them without complaint. But IP ownership is public, so a site that distrusts servers can challenge a whole range at once. - **Residential IPs** come from home internet connections. They look like ordinary visitors and are rarely blocked as a range, at a higher price per GB. The rule of thumb: rotate datacenter IPs where they get clean pages, and switch to residential for sites that fight bots or show different content by location. [Residential vs datacenter proxies](https://scrape.ly/blog/residential-vs-datacenter-proxies.md) compares the two in detail, with real traffic numbers. ## Rotating proxies in Python with requests This script rotates through a list of proxy URLs, one per request, retries a blocked request on the next proxy with a backoff, and shows a sticky session for related pages: ```python import itertools import os import time import requests # Your proxy URLs, e.g. "http://user:pass@proxy1.example.com:8000" PROXY_URLS = os.environ.get("PROXY_URLS", "http://127.0.0.1:8901,http://127.0.0.1:8902").split(",") BLOCKED = {403, 407, 429, 503} pool = itertools.cycle(PROXY_URLS) def fetch(url, attempts=3): """Per-request rotation: every request, and every retry, takes the next proxy.""" for attempt in range(attempts): proxy = next(pool) try: resp = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20) except requests.RequestException as err: print(f"{proxy} failed: {err.__class__.__name__}") continue if resp.status_code not in BLOCKED: print(f"{resp.status_code} via {proxy}: {url}") return resp time.sleep(2 ** attempt) # blocked: back off, then try another IP return None for n in range(1, 5): fetch(f"https://books.toscrape.com/catalogue/page-{n}.html") def sticky_session(proxy, urls): """Sticky session: one proxy and one cookie jar for a sequence of related pages.""" with requests.Session() as session: session.proxies = {"http": proxy, "https": proxy} for url in urls: resp = session.get(url, timeout=20) print(f"{resp.status_code} via {proxy} (sticky): {url}") sticky_session( PROXY_URLS[0], [ "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html", "https://books.toscrape.com/catalogue/category/books/mystery_3/page-2.html", ], ) ``` Set `PROXY_URLS` to your provider's URLs, separated by commas. We tested it with two local proxy servers standing in for a provider, with `requests` 2.34.2 on Python 3.12, against the books.toscrape.com sandbox: ```plaintext 200 via http://127.0.0.1:8901: https://books.toscrape.com/catalogue/page-1.html 200 via http://127.0.0.1:8902: https://books.toscrape.com/catalogue/page-2.html 200 via http://127.0.0.1:8901: https://books.toscrape.com/catalogue/page-3.html 200 via http://127.0.0.1:8902: https://books.toscrape.com/catalogue/page-4.html 200 via http://127.0.0.1:8901 (sticky): https://books.toscrape.com/catalogue/category/books/mystery_3/index.html 200 via http://127.0.0.1:8901 (sticky): https://books.toscrape.com/catalogue/category/books/mystery_3/page-2.html ``` The catalog pages alternate between the two proxies; the two category pages stay on one. With a rotating gateway, the list collapses to one URL, and you get stickiness by putting a session ID in the username, in the provider's format. Two details make this production-ready. Treat `407` as a proxy problem, not a site block: it means the proxy rejected your credentials. And drop a proxy from the list after repeated failures instead of cycling back to it forever. ## Proxy rotation with Crawlee [Crawlee](https://crawlee.dev/), the open-source crawling library the Scrape.ly SDKs build on, handles rotation, sessions and retries for you. Give it a `ProxyConfiguration` and turn on the session pool: ```javascript import { CheerioCrawler, ProxyConfiguration } from 'crawlee'; const proxyConfiguration = new ProxyConfiguration({ proxyUrls: (process.env.PROXY_URLS ?? 'http://127.0.0.1:8901,http://127.0.0.1:8902').split(','), }); const crawler = new CheerioCrawler({ proxyConfiguration, useSessionPool: true, persistCookiesPerSession: true, maxRequestsPerCrawl: 6, async requestHandler({ request, $, proxyInfo, session }) { console.log(`${session.id} ${proxyInfo.url} ${request.url} -> ${$('h1').text()}`); }, }); await crawler.run([1, 2, 3, 4, 5, 6].map((n) => `https://books.toscrape.com/catalogue/page-${n}.html`)); ``` With Crawlee 3.18.2 and the same two local proxies, all six pages succeeded, spread across both proxies (four of the six result lines shown): ```plaintext session_uyca62wsWO http://127.0.0.1:8902 https://books.toscrape.com/catalogue/page-5.html -> All products session_ScH458gevG http://127.0.0.1:8901 https://books.toscrape.com/catalogue/page-4.html -> All products session_LSWahNvbQE http://127.0.0.1:8901 https://books.toscrape.com/catalogue/page-2.html -> All products session_Q6tlIFnb9Y http://127.0.0.1:8902 https://books.toscrape.com/catalogue/page-6.html -> All products INFO CheerioCrawler: Finished! Total 6 requests: 6 succeeded, 0 failed. ``` Each session is tied to one proxy and keeps its cookies, so related requests look consistent. When a session hits a blocking status code, Crawlee retires it and the request is retried with a fresh session and proxy. You get sticky sessions and rotation from the same few lines. The [Crawlee + Cheerio template](https://scrape.ly/templates/crawlee-cheerio-js.md) gives you a project like this, ready to push to Scrape.ly. ## Rotating proxies on Scrape.ly When your scraper runs as an Actor on Scrape.ly, every run can use the managed proxy, and you don't configure credentials at all. The platform injects them into the run, and they only work from inside that run. In code it's one line: ```javascript import { Actor, CheerioCrawler } from 'scrapely'; await Actor.main(async () => { const proxyConfiguration = await Actor.createProxyConfiguration({ groups: ['DATACENTER'], countryCode: 'US', }); const crawler = new CheerioCrawler({ proxyConfiguration, requestHandler: async ({ request, $ }) => { await Actor.pushData({ url: request.url, title: $('title').text() }); }, }); await crawler.run(['https://example.com']); }); ``` Leave out `groups` for the residential pool. In Python it's `await Actor.create_proxy_configuration(groups=["RESIDENTIAL"], country_code="US")`, and `new_url()` returns the next URL, rotating round-robin; pass a session ID to stick to one IP. You can also mix in your own proxies with `proxyUrls`, or escalate in tiers, for example from no proxy to the managed proxy to a premium list, with `tieredProxyUrls`. Traffic is billed per GB as run usage, at your plan's rate, and it counts toward the run's maximum charge, so a run stops before it spends more than you allow: | Plan | Monthly price | Residential per GB | Datacenter per GB | | -------- | ----------------------- | ------------------ | ----------------- | | Free | $0 ($5 credit included) | $4.00 | $1.00 | | Starter | $19 | $3.50 | $1.00 | | Scale | $199 | $2.75 | $0.90 | | Business | $999 | $2.00 | $0.75 | Each paid plan's monthly price comes back as usage credit, and there's no separate proxy subscription. The [proxies page](https://scrape.ly/proxies.md) lists every feature. If a page needs a real browser, the [Scrape.ly browser](https://scrape.ly/docs/browser.md) brings its own residential proxy and picks the exit by country, state or city; the [anti-blocking page](https://scrape.ly/anti-blocking.md) shows how the two fit together. Store Actors expose rotation as input. The [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) has a `proxyRotation` option with `RECOMMENDED`, `PER_REQUEST` and `UNTIL_FAILURE`, which map to the strategies above. The [eBay Scraper](https://scrape.ly/store/shelfbright/ebay-scraper.md) defaults to the datacenter pool, which is enough for its public listing pages, and includes its proxy traffic in its price of $4.47 per 1,000 listings. Scrape.ly Store Rotation as a setting Your page function on any public site, with per-request or until-failure proxy rotation built in: $0.71 per 1,000 results, platform usage included. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) ## Rotating proxies for your own servers: Evomi If your scraper runs on your own infrastructure and you buy the IPs yourself, [Evomi](https://evomi.com/pricing) is the proxy network from the team behind Scrape.ly (both are brands of TeraShift GmbH), and its network is the one Scrape.ly's pools run on. Its prices, checked on 26 October 2026: | Evomi pool | Pay as you go | Monthly plan | | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------------- | | [Residential](https://evomi.com/product/residential-proxies) | $0.99 per GB from 15 GB, $0.79 from 1,000 GB, no expiry | $0.49 per GB: $49 for 100 GB, extra GB at the same rate | | [Datacenter](https://evomi.com/product/datacenter-proxies) | $0.45 per GB | $0.40 per GB (200 GB) down to $0.30 (1,000 GB, $300) | The residential pool has 54M+ ethically sourced IPs with rotating and sticky sessions, and Evomi says geo-targeting, sticky sessions and every protocol are included at the entry rate. Every Evomi proxy product can be started on a free trial. Plug its gateway URL into `PROXY_URLS` in the Python script above, or into Crawlee's `proxyUrls`. ## What to watch out for - **Rotation isn't a fingerprint.** A new IP with the same suspicious headers or a headless browser's tells still gets flagged. Pair rotation with real browser headers, or a real browser; see [web scraping without getting blocked](https://scrape.ly/blog/web-scraping-without-getting-blocked.md). - **Keep location consistent.** A session that changes country mid-flow, or a German IP with a US time zone, looks odd. Pin the country per session. - **Rotating doesn't excuse speed.** Spreading 100 requests a second across 100 IPs is still 100 requests a second to the site. Respect its rate limits and robots.txt. - **Challenges still happen.** When a CAPTCHA appears, switching IPs and retrying at once usually makes it worse. [CAPTCHA bypass for web scraping](https://scrape.ly/blog/captcha-solving-for-scraping.md) covers how to detect and handle them. - **Free proxy lists are a risk.** Open proxies can log or alter your traffic and are usually already blocked. - **Cost is bytes.** Per-GB pricing means heavy pages cost more. Block images where you can and use datacenter IPs where they work. ## FAQ ### What is a rotating proxy? A proxy setup that changes the IP address your requests come from, either on every request or for each new session. It's usually a gateway that assigns a new exit IP from a pool, or your own code cycling through a list. ### What is the difference between sticky and rotating proxies? A rotating proxy gives you a new IP per request; a sticky proxy keeps the same IP for a session, from a few requests to several minutes. Use rotation for independent pages and sticky sessions for sequences that depend on cookies. ### How often should I rotate proxies when scraping? Per request for independent pages. For multi-page flows, keep one IP per session and rotate when you see a `403`, a `429` or a CAPTCHA, or after a fixed number of pages. Crawlee's session pool does this for you. ### Do I need residential proxies for web scraping? Not always. Datacenter IPs work on many public sites and cost a quarter as much on Scrape.ly's Free plan ($1 vs $4 per GB). Switch to residential where datacenter traffic is blocked or where you need local results. ### Are free rotating proxies safe? Generally not for real work. Public proxy lists are often slow, already banned by big sites, and run by operators who can see or modify your traffic. A paid pool with per-GB billing costs little at scraping volumes. ### How much do rotating proxies cost? On Scrape.ly, residential traffic costs $2 to $4 per GB and datacenter $0.75 to $1, depending on the plan. Evomi's residential pool costs $0.49 per GB on its $49 plan and $0.99 per GB pay as you go, checked on 26 October 2026. About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head, disguised in a dark fedora, a raised collar and a cloak, pulls a data card out of a browser window showing the green Puppeteer logo and drops it into a crate](https://scrape.ly/media/puppeteer-stealth.png?w=750)](https://scrape.ly/blog/puppeteer-stealth.md) ### [Puppeteer Stealth: Setup, What It Patches, and Where It Stops](https://scrape.ly/blog/puppeteer-stealth.md) [Puppeteer stealth is puppeteer-extra-plugin-stealth, a plugin that hides the signs of a headless Chrome. Its maintenance status in October 2026, tested setup with Puppeteer 25, before-and-after results, the header mismatches it creates, a Crawlee setup and the hosted alternative.](https://scrape.ly/blog/puppeteer-stealth.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a row of data from a web page on the left into a table on the right that is filling up row by row](https://scrape.ly/media/what-is-web-scraping.png?w=750)](https://scrape.ly/blog/what-is-web-scraping.md) ### [What Is Web Scraping? How It Works and What It Is Used For](https://scrape.ly/blog/what-is-web-scraping.md) [What is web scraping? The automated collection of data from websites into structured records. See how it works step by step with tested Python code, what teams use it for, how it differs from crawling, APIs and screen scraping, and how to do it without code.](https://scrape.ly/blog/what-is-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a monitor showing the blue and yellow Python logo beside lines of code, and drops it into a crate](https://scrape.ly/media/web-scraping-python.png?w=750)](https://scrape.ly/blog/web-scraping-python.md) ### [Web Scraping With Python: A Complete 2026 Guide](https://scrape.ly/blog/web-scraping-python.md) [A complete guide to web scraping with Python: requests and BeautifulSoup, CSS selectors, pagination, CSV and JSON export, polite delays and retries, async httpx, Playwright for JavaScript pages, Scrapy, and how to run it all in the cloud. Every script tested on 23 October 2026.](https://scrape.ly/blog/web-scraping-python.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.