Web Scraping With Python: A Complete 2026 Guide

For web scraping, Python is the language most people reach for: requests downloads a page, BeautifulSoup turns the HTML into something you can query with CSS selectors, and a 40-line script can collect a whole catalogue. This guide takes you from your first request to a scraper that paginates, saves CSV and JSON, backs off politely, handles JavaScript pages and runs on a schedule in the cloud.

Every script below was run on 23 October 2026 with Python 3.14 against two sandbox sites built for practice: books.toscrape.com and quotes.toscrape.com. New to the topic? Start with what web scraping is and come back.

Web scraping: Python libraries compared

Start with requests and BeautifulSoup. Add the others only when a site forces you to.

Set up your environment

You need Python 3.10 or newer (the current requests, Playwright and Scrapy releases all require it). Create a virtual environment and install the two core libraries:

python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4

If you use uv, uv run --with requests --with beautifulsoup4 python scraper.py does the same without a separate install step. That's how we ran every example here.

Your first scraper: requests and BeautifulSoup

Open books.toscrape.com in your browser, right-click a book and choose Inspect. Each book sits in an <article class="product_pod">, the title is in the title attribute of the link inside <h3>, and the price is in <p class="price_color">. That's all you need:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://books.toscrape.com/", timeout=15)
response.raise_for_status()
response.encoding = "utf-8"
soup = BeautifulSoup(response.text, "html.parser")

for card in soup.select("article.product_pod")[:3]:
    title = card.select_one("h3 a")["title"]
    price = card.select_one("p.price_color").text
    print(title, price)

print(len(soup.select("article.product_pod")), "books on this page")

Output:

A Light in the Attic £51.77
Tipping the Velvet £53.74
Soumission £50.10
20 books on this page

Two details save you debugging time. raise_for_status() turns a 404 or 500 into an exception instead of letting you parse an error page. And response.encoding = "utf-8" matters here: the site sends Content-Type: text/html with no charset, so requests falls back to ISO-8859-1 and the pound sign comes out as £. Whenever you see garbled characters, check the encoding first.

CSS selectors you'll actually use

soup.select() returns every match, soup.select_one() returns the first match or None. The selector syntax is the same one you use in CSS and in your browser's DevTools:

Read attributes with element["href"] (raises if missing) or element.get("href") (returns None), and text with element.text or element.get_text(strip=True). In DevTools you can right-click an element and choose Copy → Copy selector, but those selectors are long and brittle. Prefer short selectors built on meaningful class names.

Scrape every page: pagination, delays and retries

The catalogue has 50 pages. Each one has a next link (li.next a) until the last. The scraper below follows it, cleans the fields, pauses one second between requests, and retries temporary failures with exponential backoff:

import csv
import json
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

START_URL = "https://books.toscrape.com/catalogue/page-1.html"
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}


def make_session() -> requests.Session:
    session = requests.Session()
    session.headers["User-Agent"] = "books-tutorial/1.0 (you@example.com)"
    retry = Retry(
        total=3,
        backoff_factor=1,  # waits 1 s, 2 s, 4 s between attempts
        status_forcelist=[429, 500, 502, 503, 504],
        respect_retry_after_header=True,
    )
    session.mount("https://", HTTPAdapter(max_retries=retry))
    return session


def parse_books(html: str, page_url: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    books = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        books.append({
            "title": link["title"],
            "price": float(card.select_one("p.price_color").text.lstrip("£")),
            "rating": RATINGS[card.select_one("p.star-rating")["class"][1]],
            "in_stock": "In stock" in card.select_one("p.availability").text,
            "url": urljoin(page_url, link["href"]),
        })
    return books


def scrape(max_pages: int = 50, delay: float = 1.0) -> list[dict]:
    session = make_session()
    url, books = START_URL, []
    for _ in range(max_pages):
        response = session.get(url, timeout=15)
        response.raise_for_status()
        response.encoding = "utf-8"
        books.extend(parse_books(response.text, url))
        next_link = BeautifulSoup(response.text, "html.parser").select_one("li.next a")
        if next_link is None:
            break
        url = urljoin(url, next_link["href"])
        time.sleep(delay)  # be polite: one request per second
    return books

What each piece does:

• requests.Session reuses the TCP connection across all 50 pages and sends the same headers every time. An honest User-Agent with a contact address is good manners on small sites.
• Retry on the adapter repeats a request that got a 429 or a 5xx, waiting longer each time, and honours the server's Retry-After header. A single network hiccup no longer kills a 50-page run.
• urljoin turns relative links like page-2.html into absolute URLs. Links on books.toscrape.com are relative to the current page, so never glue strings together by hand.
• time.sleep(delay) keeps you at about one request per second. On a sandbox it's courtesy; on a real site it's the difference between a finished run and a blocked IP.
• The rating lives in a class name (star-rating Three), so the code reads the second class and maps it to a number. Real sites hide data in odd places like this all the time.

Save the results to CSV and JSON

def save(books: list[dict]) -> None:
    with open("books.csv", "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=books[0].keys())
        writer.writeheader()
        writer.writerows(books)
    with open("books.json", "w", encoding="utf-8") as f:
        json.dump(books, f, ensure_ascii=False, indent=2)


if __name__ == "__main__":
    books = scrape()
    save(books)
    print(len(books), "books")
    print(books[0])

Our run fetched all 50 pages in 70 seconds (almost all of it the polite one-second pauses) and printed:

1000 books
{'title': 'A Light in the Attic', 'price': 51.77, 'rating': 3, 'in_stock': True, 'url': 'https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html'}

newline="" stops the csv module from writing blank lines on Windows, and ensure_ascii=False keeps characters like £ readable in the JSON. If you already use pandas, pandas.DataFrame(books).to_csv("books.csv", index=False) does the same in one line.

Go faster with httpx and asyncio

The loop above waits for each page before asking for the next. When you know the URLs up front, an async client fetches several at once. httpx has a requests-like API plus an async one:

import asyncio

import httpx
from bs4 import BeautifulSoup

PAGE_URL = "https://books.toscrape.com/catalogue/page-{}.html"


async def fetch(client: httpx.AsyncClient, limit: asyncio.Semaphore, page: int) -> list[str]:
    async with limit:  # at most 5 requests in flight
        response = await client.get(PAGE_URL.format(page))
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")
        return [a["title"] for a in soup.select("article.product_pod h3 a")]


async def main() -> None:
    limit = asyncio.Semaphore(5)
    transport = httpx.AsyncHTTPTransport(retries=2)  # retries failed connections
    async with httpx.AsyncClient(timeout=15, transport=transport) as client:
        pages = await asyncio.gather(*(fetch(client, limit, n) for n in range(1, 11)))
    titles = [title for page in pages for title in page]
    print(len(titles), titles[:2])


asyncio.run(main())

It printed 200 ['A Light in the Attic', 'Tipping the Velvet'] for the first 10 pages. The semaphore is the important line: it caps concurrency so you don't hammer the server with every request at once. Note that httpx's transport retries only covers connection failures; to retry on 429 or 503 you still check the status code yourself.

Scrape JavaScript pages with Playwright

Open quotes.toscrape.com/js and view the page source: there are no quotes in it, only a script that writes them into the page. requests sees exactly that empty source. For pages like this you need a real browser. Install Playwright and its Chromium with pip install playwright and playwright install chromium:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/")
    page.wait_for_selector("div.quote")  # rendered by JavaScript
    quotes = [
        {
            "text": card.locator("span.text").inner_text(),
            "author": card.locator("small.author").inner_text(),
        }
        for card in page.locator("div.quote").all()
    ]
    browser.close()

print(len(quotes), quotes[0])

It printed 10 quotes, starting with Albert Einstein's "The world as we have created it is a process of our thinking." Playwright waits for elements automatically before it clicks or types, and can block images and fonts to save bandwidth. If you're weighing browser libraries, our Playwright vs Selenium comparison runs the same scraper in both, and the Selenium web scraping guide covers waits, infinite scroll and screenshots in Selenium.

Check for a JSON API first

Before you reach for a browser, open DevTools, switch to the Network tab, filter by Fetch/XHR and reload. Many JavaScript pages load their data from a JSON endpoint you can call directly. The infinite-scroll version at quotes.toscrape.com/scroll does exactly that:

import requests

quotes, page = [], 1
while True:
    data = requests.get(
        "https://quotes.toscrape.com/api/quotes", params={"page": page}, timeout=15
    ).json()
    quotes += [{"text": q["text"], "author": q["author"]["name"]} for q in data["quotes"]]
    if not data["has_next"]:
        break
    page += 1

print(len(quotes), "quotes from", page, "API pages")

100 quotes from 10 API pages, in about a second and with no browser. Structured JSON also doesn't break when the site changes its CSS classes.

Crawl a whole site with Scrapy

When a project grows past a few page types, Scrapy gives you the request queue, retries, throttling, duplicate filtering and export that you'd otherwise write yourself. A complete spider fits in one file:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]
    custom_settings = {
        "DOWNLOAD_DELAY": 1,
        "AUTOTHROTTLE_ENABLED": True,
        "ROBOTSTXT_OBEY": True,
        "CLOSESPIDER_PAGECOUNT": 5,  # stop early while testing; remove for all 50 pages
    }

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css("p.price_color::text").get(),
            }
        yield from response.follow_all(css="li.next a", callback=self.parse)

Run it with scrapy runspider books_spider.py -O books.json. Our test stopped at the page limit with 80 books in books.json. Scrapy decodes the page correctly on its own, follows relative links with response.follow_all, and writes CSV, JSON or JSON Lines depending on the file extension.

Run your Python scraper at scale with Scrape.ly

A script on your laptop is the easy part. Running it every morning, rotating proxies when a site starts answering 429, storing the results and getting an alert when a redesign returns empty fields is where most of the time goes. Scrape.ly handles that layer, and you keep writing Python.

Call a ready-made scraper from Python

The Scrape.ly Store has 568 ready-made scrapers, called Actors, for sites like Google Maps, Amazon and LinkedIn, so for big sites you often don't write a parser at all. For any other public site, the Web Scraper Actor crawls over HTTP and runs your own small page function on every page. Call it with the Python client (pip install scrapely-client-python):

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

page_function = """async function pageFunction(context) {
    const $ = context.jQuery;
    if (!$('div.product_main').length) return null; // listing page: only follow links
    return {
        url: context.request.url,
        title: $('div.product_main h1').text().trim(),
        price: $('div.product_main p.price_color').text().trim(),
        inStock: $('div.product_main p.availability').text().includes('In stock'),
        upc: $('table.table-striped td').first().text().trim(),
    };
}"""

run = client.actor("querystone/web-scraper").call(
    run_input={
        "startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"}],
        "linkSelector": "article.product_pod h3 a",
        "globs": [{"glob": "https://books.toscrape.com/catalogue/*/index.html"}],
        "pageFunction": page_function,
        "maxPagesPerCrawl": 6,
        "maxResultsPerCrawl": 5,
        "maxConcurrency": 2,
        "respectRobotsTxtFile": True,
        "proxyConfiguration": {"useScrapelyProxy": True},
    }
)
print(run.status)  # SUCCEEDED, FAILED, ABORTED or TIMED-OUT
for item in client.dataset(run.default_dataset_id).iterate_items():
    if item.get("title"):
        print(item["title"], item["price"], item["upc"])

We ran this input on 23 October 2026. The run took 12.4 seconds, fetched 5 pages through Scrape.ly's residential proxy, saved 4 book records (plus 1 metadata record for the listing page, which the page function skips) and cost $0.0029 in total:

The client is a drop-in replacement for apify-client: same method names, change the import and the API URL. Always pass api_url, because the client doesn't read it from the environment.

Deploy your own Python scraper as an Actor

To run your own code in the cloud, turn it into an Actor. The Python SDK is built on Crawlee for Python and mirrors the Apify Python SDK. Install the CLI (it runs on Node.js 20.17 or newer) and start from the Crawlee + BeautifulSoup template:

npm install -g scrapely-cli
export SCRAPELY_API_URL=https://api.scrape.ly SCRAPELY_WEB_URL=https://my.scrape.ly
scrapely login
scrapely create my-actor --template crawlee-bs4-py && cd my-actor

Put your parsing in the request handler. Here is the books scraper as an Actor:

import asyncio

from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from scrapely import Actor


async def main() -> None:
    async with Actor:
        actor_input = await Actor.get_input() or {}
        start_urls = [item["url"] for item in actor_input.get("startUrls", [])] or [
            "https://books.toscrape.com/"
        ]
        crawler = BeautifulSoupCrawler(
            max_requests_per_crawl=actor_input.get("maxRequestsPerCrawl", 10),
        )

        @crawler.router.default_handler
        async def handler(context: BeautifulSoupCrawlingContext) -> None:
            for card in context.soup.select("article.product_pod"):
                await context.push_data({
                    "title": card.select_one("h3 a")["title"],
                    "price": card.select_one("p.price_color").text,
                    "url": context.request.url,
                })
            await context.enqueue_links(selector="li.next a")

        await crawler.run(start_urls)


if __name__ == "__main__":
    asyncio.run(main())

Then push it and run it on the platform:

scrapely push
scrapely call -i '{"startUrls":[{"url":"https://books.toscrape.com/"}],"maxRequestsPerCrawl":10}' -o

We ran this code locally (Python 3.13, scrapely-python 0.1.31, Crawlee 1.11.0): 10 pages, 200 books, 0 failed requests, 2.1 seconds. On the platform, context.push_data writes to the run's dataset, which you can download as CSV or JSON or read over the API. You add schedules, webhooks and integrations such as Google Sheets or S3 without touching the code. Three rules from the docs keep builds clean:

• Never pip install scrapely. That PyPI name belongs to an unrelated library. The SDK's PyPI name is scrapely-python, it's preinstalled in the Scrape.ly base images, and it stays out of requirements.txt.
• Pick a browser image for Playwright. The scrapely/actor-python-playwright images add Playwright; there's a Crawlee + Playwright (Python) template (crawlee-playwright-py) for JavaScript-heavy sites.
• Already have a Scrapy project? Run scrapely init in its folder. The CLI writes a main.py wrapper that runs your spiders inside the Actor and pushes every item to the dataset.

For sites that block plain browsers, turn on the Scrape.ly browser for your Actor and rebuild: Playwright launches connect to a hosted Chromium with anti-bot fingerprints, Cloudflare Turnstile solving and its own residential proxy, with no code changes. Every run can carry a maximum charge, so a runaway crawl stops at the cap. See all starters on the templates page.

What to watch out for

• Blocking. Fast, repetitive requests from one IP get rate limited. Keep delays, honour Retry-After, and move to rotating residential proxies when a site insists. On Scrape.ly, Web Scraper switches the platform's residential proxy on with one input field (proxyConfiguration).
• Selectors break. Sites redesign. Validate every record (no empty titles, prices parse as numbers) and fail loudly instead of saving blanks.
• Personal data. Scraping public pages is broadly accepted, but names, emails and profiles fall under privacy laws such as the GDPR. Collect only what you need, and keep personal fields out of datasets you share.
• Terms and robots.txt. Read the site's terms and robots.txt before a large crawl. Scrapy's ROBOTSTXT_OBEY and the Web Scraper's respectRobotsTxtFile respect it for you.

FAQ

Is Python good for web scraping?

Yes. Python has one of the most complete scraping toolkits of any language: requests and httpx for HTTP, BeautifulSoup and lxml for parsing, Playwright and Selenium for browsers, and Scrapy and Crawlee for full crawlers. The code stays short and readable, and pandas takes the data straight into analysis.

Which Python library is best for web scraping?

Start with requests and BeautifulSoup: they cover most static pages. Use Playwright when the data is rendered by JavaScript, and Scrapy or Crawlee when you crawl thousands of pages and want queues, retries and throttling built in.

How do I scrape a website that needs JavaScript with Python?

First check the Network tab for a JSON endpoint you can call with requests, as with quotes.toscrape.com's /api/quotes. If there isn't one, render the page with Playwright or Selenium and wait for the element you need before you read it.

Is Python web scraping legal?

Scraping publicly available data is generally lawful in many jurisdictions, but it depends on what you collect and how. Respect the site's terms, avoid overloading servers, and treat personal data under privacy law. This isn't legal advice. Scrape.ly Store Actors only collect publicly available data shown to any visitor.

How do I avoid getting blocked while scraping with Python?

Slow down, reuse a session, send a realistic User-Agent, retry with backoff on 429 and 5xx, and rotate residential proxies for larger jobs. For sites with bot protection, a hosted browser with real fingerprints, such as the Scrape.ly browser, does more than any header tweak.