--- title: "Web Scraping: Python Tutorial and Guide (2026) | Scrape.ly" description: "Learn web scraping: Python code for requests, BeautifulSoup, pagination, CSV export, retries, async, Playwright and Scrapy, all tested, plus cloud scaling." url: https://scrape.ly/blog/web-scraping-python markdown: https://scrape.ly/blog/web-scraping-python.md --- [Guides](https://scrape.ly/blog/category/guides.md) # Web Scraping With Python: A Complete 2026 Guide A complete guide to web scraping with Python: requests and BeautifulSoup, CSS selectors, pagination, CSV and JSON export, polite delays and retries, async httpx, Playwright for JavaScript pages, Scrapy, and how to run it all in the cloud. Every script tested on 23 October 2026. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 10 min read ![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a monitor showing the blue and yellow Python logo beside lines of code, and drops it into a crate](https://scrape.ly/media/web-scraping-python.png?w=1080) For web scraping, Python is the language most people reach for: `requests` downloads a page, BeautifulSoup turns the HTML into something you can query with CSS selectors, and a 40-line script can collect a whole catalogue. This guide takes you from your first request to a scraper that paginates, saves CSV and JSON, backs off politely, handles JavaScript pages and runs on a schedule in the cloud. Every script below was run on 23 October 2026 with Python 3.14 against two sandbox sites built for practice: [books.toscrape.com](https://books.toscrape.com/) and [quotes.toscrape.com](https://quotes.toscrape.com/). New to the topic? Start with [what web scraping is](https://scrape.ly/blog/what-is-web-scraping.md) and come back. ## Web scraping: Python libraries compared | Tool | What it does | Use it when | Version tested | | ------------- | ----------------------------------------------------------- | ------------------------------------------------ | -------------- | | requests | Sends HTTP requests, returns the HTML | The data is in the page source | 2.34.2 | | BeautifulSoup | Parses HTML, finds elements with CSS selectors | Always, next to requests or httpx | 4.15.0 | | httpx | HTTP client with an async API | You fetch hundreds of pages and want concurrency | 0.28.1 | | Playwright | Drives a real Chromium, Firefox or WebKit | The data only appears after JavaScript runs | 1.64.0 | | Selenium | Drives a real browser over WebDriver | You already know it or run Selenium Grid | 4.51.0 | | Scrapy | Full crawling framework: queue, retries, throttling, export | Large crawls across many pages | 2.19.0 | Start with requests and BeautifulSoup. Add the others only when a site forces you to. ## Set up your environment You need Python 3.10 or newer (the current requests, Playwright and Scrapy releases all require it). Create a virtual environment and install the two core libraries: ```shell python3 -m venv .venv source .venv/bin/activate pip install requests beautifulsoup4 ``` If you use [uv](https://docs.astral.sh/uv/), `uv run --with requests --with beautifulsoup4 python scraper.py` does the same without a separate install step. That's how we ran every example here. ## Your first scraper: requests and BeautifulSoup Open books.toscrape.com in your browser, right-click a book and choose **Inspect**. Each book sits in an `
`, the title is in the `title` attribute of the link inside `

`, and the price is in `

`. That's all you need: ```python import requests from bs4 import BeautifulSoup response = requests.get("https://books.toscrape.com/", timeout=15) response.raise_for_status() response.encoding = "utf-8" soup = BeautifulSoup(response.text, "html.parser") for card in soup.select("article.product_pod")[:3]: title = card.select_one("h3 a")["title"] price = card.select_one("p.price_color").text print(title, price) print(len(soup.select("article.product_pod")), "books on this page") ``` Output: ```plaintext A Light in the Attic £51.77 Tipping the Velvet £53.74 Soumission £50.10 20 books on this page ``` Two details save you debugging time. `raise_for_status()` turns a 404 or 500 into an exception instead of letting you parse an error page. And `response.encoding = "utf-8"` matters here: the site sends `Content-Type: text/html` with no charset, so requests falls back to ISO-8859-1 and the pound sign comes out as `£`. Whenever you see garbled characters, check the encoding first. ## CSS selectors you'll actually use `soup.select()` returns every match, `soup.select_one()` returns the first match or `None`. The selector syntax is the same one you use in CSS and in your browser's DevTools: | Selector | Matches | Example on books.toscrape.com | | ------------------------ | ----------------------- | ----------------------------- | | `tag.class` | Elements with a class | `p.price_color` | | `parent child` | Descendants | `article.product_pod h3 a` | | `parent > child` | Direct children only | `li.next > a` | | `[attr]`, `[attr=value]` | Attributes | `p[class*=star-rating]` | | `:nth-of-type(n)` | Position among siblings | `ol.row li:nth-of-type(1)` | Read attributes with `element["href"]` (raises if missing) or `element.get("href")` (returns `None`), and text with `element.text` or `element.get_text(strip=True)`. In DevTools you can right-click an element and choose **Copy → Copy selector**, but those selectors are long and brittle. Prefer short selectors built on meaningful class names. ## Scrape every page: pagination, delays and retries The catalogue has 50 pages. Each one has a **next** link (`li.next a`) until the last. The scraper below follows it, cleans the fields, pauses one second between requests, and retries temporary failures with exponential backoff: ```python import csv import json import time from urllib.parse import urljoin import requests from bs4 import BeautifulSoup from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry START_URL = "https://books.toscrape.com/catalogue/page-1.html" RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5} def make_session() -> requests.Session: session = requests.Session() session.headers["User-Agent"] = "books-tutorial/1.0 (you@example.com)" retry = Retry( total=3, backoff_factor=1, # waits 1 s, 2 s, 4 s between attempts status_forcelist=[429, 500, 502, 503, 504], respect_retry_after_header=True, ) session.mount("https://", HTTPAdapter(max_retries=retry)) return session def parse_books(html: str, page_url: str) -> list[dict]: soup = BeautifulSoup(html, "html.parser") books = [] for card in soup.select("article.product_pod"): link = card.select_one("h3 a") books.append({ "title": link["title"], "price": float(card.select_one("p.price_color").text.lstrip("£")), "rating": RATINGS[card.select_one("p.star-rating")["class"][1]], "in_stock": "In stock" in card.select_one("p.availability").text, "url": urljoin(page_url, link["href"]), }) return books def scrape(max_pages: int = 50, delay: float = 1.0) -> list[dict]: session = make_session() url, books = START_URL, [] for _ in range(max_pages): response = session.get(url, timeout=15) response.raise_for_status() response.encoding = "utf-8" books.extend(parse_books(response.text, url)) next_link = BeautifulSoup(response.text, "html.parser").select_one("li.next a") if next_link is None: break url = urljoin(url, next_link["href"]) time.sleep(delay) # be polite: one request per second return books ``` What each piece does: - **`requests.Session`** reuses the TCP connection across all 50 pages and sends the same headers every time. An honest `User-Agent` with a contact address is good manners on small sites. - **`Retry`** **on the adapter** repeats a request that got a 429 or a 5xx, waiting longer each time, and honours the server's `Retry-After` header. A single network hiccup no longer kills a 50-page run. - **`urljoin`** turns relative links like `page-2.html` into absolute URLs. Links on books.toscrape.com are relative to the current page, so never glue strings together by hand. - **`time.sleep(delay)`** keeps you at about one request per second. On a sandbox it's courtesy; on a real site it's the difference between a finished run and a blocked IP. - **The rating** lives in a class name (`star-rating Three`), so the code reads the second class and maps it to a number. Real sites hide data in odd places like this all the time. ## Save the results to CSV and JSON ```python def save(books: list[dict]) -> None: with open("books.csv", "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=books[0].keys()) writer.writeheader() writer.writerows(books) with open("books.json", "w", encoding="utf-8") as f: json.dump(books, f, ensure_ascii=False, indent=2) if __name__ == "__main__": books = scrape() save(books) print(len(books), "books") print(books[0]) ``` Our run fetched all 50 pages in 70 seconds (almost all of it the polite one-second pauses) and printed: ```plaintext 1000 books {'title': 'A Light in the Attic', 'price': 51.77, 'rating': 3, 'in_stock': True, 'url': 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'} ``` `newline=""` stops the csv module from writing blank lines on Windows, and `ensure_ascii=False` keeps characters like `£` readable in the JSON. If you already use pandas, `pandas.DataFrame(books).to_csv("books.csv", index=False)` does the same in one line. ## Go faster with httpx and asyncio The loop above waits for each page before asking for the next. When you know the URLs up front, an async client fetches several at once. httpx has a requests-like API plus an async one: ```python import asyncio import httpx from bs4 import BeautifulSoup PAGE_URL = "https://books.toscrape.com/catalogue/page-{}.html" async def fetch(client: httpx.AsyncClient, limit: asyncio.Semaphore, page: int) -> list[str]: async with limit: # at most 5 requests in flight response = await client.get(PAGE_URL.format(page)) response.raise_for_status() soup = BeautifulSoup(response.text, "html.parser") return [a["title"] for a in soup.select("article.product_pod h3 a")] async def main() -> None: limit = asyncio.Semaphore(5) transport = httpx.AsyncHTTPTransport(retries=2) # retries failed connections async with httpx.AsyncClient(timeout=15, transport=transport) as client: pages = await asyncio.gather(*(fetch(client, limit, n) for n in range(1, 11))) titles = [title for page in pages for title in page] print(len(titles), titles[:2]) asyncio.run(main()) ``` It printed `200 ['A Light in the Attic', 'Tipping the Velvet']` for the first 10 pages. The semaphore is the important line: it caps concurrency so you don't hammer the server with every request at once. Note that httpx's transport `retries` only covers connection failures; to retry on 429 or 503 you still check the status code yourself. ## Scrape JavaScript pages with Playwright Open [quotes.toscrape.com/js](https://quotes.toscrape.com/js/) and view the page source: there are no quotes in it, only a script that writes them into the page. requests sees exactly that empty source. For pages like this you need a real browser. Install Playwright and its Chromium with `pip install playwright` and `playwright install chromium`: ```python from playwright.sync_api import sync_playwright with sync_playwright() as p: browser = p.chromium.launch(headless=True) page = browser.new_page() page.goto("https://quotes.toscrape.com/js/") page.wait_for_selector("div.quote") # rendered by JavaScript quotes = [ { "text": card.locator("span.text").inner_text(), "author": card.locator("small.author").inner_text(), } for card in page.locator("div.quote").all() ] browser.close() print(len(quotes), quotes[0]) ``` It printed 10 quotes, starting with Albert Einstein's "The world as we have created it is a process of our thinking." Playwright waits for elements automatically before it clicks or types, and can block images and fonts to save bandwidth. If you're weighing browser libraries, our [Playwright vs Selenium comparison](https://scrape.ly/blog/playwright-vs-selenium.md) runs the same scraper in both, and the [Selenium web scraping guide](https://scrape.ly/blog/selenium-web-scraping.md) covers waits, infinite scroll and screenshots in Selenium. ### Check for a JSON API first Before you reach for a browser, open DevTools, switch to the **Network** tab, filter by **Fetch/XHR** and reload. Many JavaScript pages load their data from a JSON endpoint you can call directly. The infinite-scroll version at [quotes.toscrape.com/scroll](https://quotes.toscrape.com/scroll) does exactly that: ```python import requests quotes, page = [], 1 while True: data = requests.get( "https://quotes.toscrape.com/api/quotes", params={"page": page}, timeout=15 ).json() quotes += [{"text": q["text"], "author": q["author"]["name"]} for q in data["quotes"]] if not data["has_next"]: break page += 1 print(len(quotes), "quotes from", page, "API pages") ``` `100 quotes from 10 API pages`, in about a second and with no browser. Structured JSON also doesn't break when the site changes its CSS classes. ## Crawl a whole site with Scrapy When a project grows past a few page types, Scrapy gives you the request queue, retries, throttling, duplicate filtering and export that you'd otherwise write yourself. A complete spider fits in one file: ```python import scrapy class BooksSpider(scrapy.Spider): name = "books" start_urls = ["https://books.toscrape.com/"] custom_settings = { "DOWNLOAD_DELAY": 1, "AUTOTHROTTLE_ENABLED": True, "ROBOTSTXT_OBEY": True, "CLOSESPIDER_PAGECOUNT": 5, # stop early while testing; remove for all 50 pages } def parse(self, response): for card in response.css("article.product_pod"): yield { "title": card.css("h3 a::attr(title)").get(), "price": card.css("p.price_color::text").get(), } yield from response.follow_all(css="li.next a", callback=self.parse) ``` Run it with `scrapy runspider books_spider.py -O books.json`. Our test stopped at the page limit with 80 books in `books.json`. Scrapy decodes the page correctly on its own, follows relative links with `response.follow_all`, and writes CSV, JSON or JSON Lines depending on the file extension. ## Run your Python scraper at scale with Scrape.ly A script on your laptop is the easy part. Running it every morning, rotating proxies when a site starts answering 429, storing the results and getting an alert when a redesign returns empty fields is where most of the time goes. Scrape.ly handles that layer, and you keep writing Python. ### Call a ready-made scraper from Python The [Scrape.ly Store](https://scrape.ly/store.md) has 568 ready-made scrapers, called Actors, for sites like Google Maps, Amazon and LinkedIn, so for big sites you often don't write a parser at all. For any other public site, the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) Actor crawls over HTTP and runs your own small page function on every page. Call it with the [Python client](https://scrape.ly/docs/client-python.md) (`pip install scrapely-client-python`): ```python import os from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") page_function = """async function pageFunction(context) { const $ = context.jQuery; if (!$('div.product_main').length) return null; // listing page: only follow links return { url: context.request.url, title: $('div.product_main h1').text().trim(), price: $('div.product_main p.price_color').text().trim(), inStock: $('div.product_main p.availability').text().includes('In stock'), upc: $('table.table-striped td').first().text().trim(), }; }""" run = client.actor("querystone/web-scraper").call( run_input={ "startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"}], "linkSelector": "article.product_pod h3 a", "globs": [{"glob": "https://books.toscrape.com/catalogue/*/index.html"}], "pageFunction": page_function, "maxPagesPerCrawl": 6, "maxResultsPerCrawl": 5, "maxConcurrency": 2, "respectRobotsTxtFile": True, "proxyConfiguration": {"useScrapelyProxy": True}, } ) print(run.status) # SUCCEEDED, FAILED, ABORTED or TIMED-OUT for item in client.dataset(run.default_dataset_id).iterate_items(): if item.get("title"): print(item["title"], item["price"], item["upc"]) ``` We ran this input on 23 October 2026. The run took 12.4 seconds, fetched 5 pages through Scrape.ly's residential proxy, saved 4 book records (plus 1 metadata record for the listing page, which the page function skips) and cost $0.0029 in total: | title | price | inStock | upc | | -------------------- | ------ | ------- | ---------------- | | Sharp Objects | £47.82 | true | e00eb4fd7b871a48 | | In a Dark, Dark Wood | £19.63 | true | 19ed25f4641d5efd | | The Past Never Ends | £56.50 | true | 5ee94540d0749ea0 | | A Murder in Time | £16.64 | true | f733e8c19d40ec2e | The client is a drop-in replacement for `apify-client`: same method names, change the import and the API URL. Always pass `api_url`, because the client doesn't read it from the environment. ### Deploy your own Python scraper as an Actor To run your own code in the cloud, turn it into an Actor. The [Python SDK](https://scrape.ly/docs/sdk-python.md) is built on Crawlee for Python and mirrors the Apify Python SDK. Install the CLI (it runs on Node.js 20.17 or newer) and start from the [Crawlee + BeautifulSoup template](https://scrape.ly/templates/crawlee-bs4-py.md): ```shell npm install -g scrapely-cli export SCRAPELY_API_URL=https://api.scrape.ly SCRAPELY_WEB_URL=https://my.scrape.ly scrapely login scrapely create my-actor --template crawlee-bs4-py && cd my-actor ``` Put your parsing in the request handler. Here is the books scraper as an Actor: ```python import asyncio from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext from scrapely import Actor async def main() -> None: async with Actor: actor_input = await Actor.get_input() or {} start_urls = [item["url"] for item in actor_input.get("startUrls", [])] or [ "https://books.toscrape.com/" ] crawler = BeautifulSoupCrawler( max_requests_per_crawl=actor_input.get("maxRequestsPerCrawl", 10), ) @crawler.router.default_handler async def handler(context: BeautifulSoupCrawlingContext) -> None: for card in context.soup.select("article.product_pod"): await context.push_data({ "title": card.select_one("h3 a")["title"], "price": card.select_one("p.price_color").text, "url": context.request.url, }) await context.enqueue_links(selector="li.next a") await crawler.run(start_urls) if __name__ == "__main__": asyncio.run(main()) ``` Then push it and run it on the platform: ```shell scrapely push scrapely call -i '{"startUrls":[{"url":"https://books.toscrape.com/"}],"maxRequestsPerCrawl":10}' -o ``` We ran this code locally (Python 3.13, scrapely-python 0.1.31, Crawlee 1.11.0): 10 pages, 200 books, 0 failed requests, 2.1 seconds. On the platform, `context.push_data` writes to the run's dataset, which you can download as CSV or JSON or read over the API. You add schedules, webhooks and integrations such as Google Sheets or S3 without touching the code. Three rules from the docs keep builds clean: - **Never **__`pip install scrapely`__ **.** That PyPI name belongs to an unrelated library. The SDK's PyPI name is `scrapely-python`, it's preinstalled in the Scrape.ly base images, and it stays out of `requirements.txt`. - **Pick a browser image for Playwright.** The `scrapely/actor-python-playwright` images add Playwright; there's a Crawlee + Playwright (Python) template (`crawlee-playwright-py`) for JavaScript-heavy sites. - **Already have a Scrapy project?** Run `scrapely init` in its folder. The CLI writes a `main.py` wrapper that runs your spiders inside the Actor and pushes every item to the dataset. For sites that block plain browsers, turn on the [Scrape.ly browser](https://scrape.ly/docs/browser.md) for your Actor and rebuild: Playwright launches connect to a hosted Chromium with anti-bot fingerprints, Cloudflare Turnstile solving and its own residential proxy, with no code changes. Every run can carry a maximum charge, so a runaway crawl stops at the cap. See all starters on the [templates page](https://scrape.ly/templates.md). ## What to watch out for - **Blocking.** Fast, repetitive requests from one IP get rate limited. Keep delays, honour `Retry-After`, and move to rotating residential proxies when a site insists. On Scrape.ly, Web Scraper switches the platform's residential proxy on with one input field (`proxyConfiguration`). - **Selectors break.** Sites redesign. Validate every record (no empty titles, prices parse as numbers) and fail loudly instead of saving blanks. - **Personal data.** Scraping public pages is broadly accepted, but names, emails and profiles fall under privacy laws such as the GDPR. Collect only what you need, and keep personal fields out of datasets you share. - **Terms and robots.txt.** Read the site's terms and `robots.txt` before a large crawl. Scrapy's `ROBOTSTXT_OBEY` and the Web Scraper's `respectRobotsTxtFile` respect it for you. ## FAQ ### Is Python good for web scraping? Yes. Python has one of the most complete scraping toolkits of any language: requests and httpx for HTTP, BeautifulSoup and lxml for parsing, Playwright and Selenium for browsers, and Scrapy and Crawlee for full crawlers. The code stays short and readable, and pandas takes the data straight into analysis. ### Which Python library is best for web scraping? Start with requests and BeautifulSoup: they cover most static pages. Use Playwright when the data is rendered by JavaScript, and Scrapy or Crawlee when you crawl thousands of pages and want queues, retries and throttling built in. ### How do I scrape a website that needs JavaScript with Python? First check the Network tab for a JSON endpoint you can call with requests, as with quotes.toscrape.com's `/api/quotes`. If there isn't one, render the page with Playwright or Selenium and wait for the element you need before you read it. ### Is Python web scraping legal? Scraping publicly available data is generally lawful in many jurisdictions, but it depends on what you collect and how. Respect the site's terms, avoid overloading servers, and treat personal data under privacy law. This isn't legal advice. Scrape.ly Store Actors only collect publicly available data shown to any visitor. ### How do I avoid getting blocked while scraping with Python? Slow down, reuse a session, send a realistic `User-Agent`, retry with backoff on 429 and 5xx, and rotate residential proxies for larger jobs. For sites with bot protection, a hosted browser with real fingerprints, such as the Scrape.ly browser, does more than any header tweak. Scrape.ly Store Web Scraper Run your own page function on any public site from Python. Link following, proxies and retries included. $0.50 per 1,000 results. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head stands before a pair of balanced golden scales of justice, holding out a data card](https://scrape.ly/media/is-web-scraping-legal.png?w=750)](https://scrape.ly/blog/is-web-scraping-legal.md) ### [Is Web Scraping Legal? US, EU and UK Law, Court Cases and a Checklist (2026)](https://scrape.ly/blog/is-web-scraping-legal.md) [Is web scraping legal? What US, EU and UK law actually says in 2026, from Van Buren and hiQ v. LinkedIn to Meta and X v. Bright Data, GDPR, database rights, robots.txt and the newest SerpApi rulings, plus a practical checklist.](https://scrape.ly/blog/is-web-scraping-legal.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser window showing a web page and drops it into a crate of collected data](https://scrape.ly/media/how-to-scrape-a-website.png?w=750)](https://scrape.ly/blog/how-to-scrape-a-website.md) ### [How to Scrape Data From a Website: 4 Methods, No-Code to Python](https://scrape.ly/blog/how-to-scrape-a-website.md) [How to scrape data from a website four ways: a ready-made scraper with no code, a generic cloud scraper with one extraction function, a Python script you run yourself, and an AI agent over MCP. Each with real runs, exact costs and tested code.](https://scrape.ly/blog/how-to-scrape-a-website.md) - [![Pixel-art robot with an orange >\_ on its terminal head walks past a raised red-and-white boom barrier, carrying a data card from the web page behind it towards a crate](https://scrape.ly/media/web-scraping-without-getting-blocked.png?w=750)](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) ### [Web Scraping Without Getting Blocked: 10 Techniques That Work](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) [Web scraping without getting blocked comes down to ten habits: real headers, pacing, sessions, good IPs, real browsers, clean fingerprints, challenge handling, honeypot checks, robots.txt and caching. Tested Python for each, and how Scrape.ly handles them for you.](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.