Scrapy vs BeautifulSoup: Which Python Scraping Tool to Use (Tested)

Scrapy vs BeautifulSoup is not a contest between two equal tools: BeautifulSoup is an HTML parser, and Scrapy is a complete crawling framework. BeautifulSoup turns HTML you've already downloaded into something you can query; Scrapy downloads pages concurrently, retries failures, follows links, throttles itself and exports the results, and has its own selectors for parsing.

So the real question is whether your job needs a framework. To answer it with numbers, we ran the same job with both on 26 October 2026: collect all 1,000 books from the 50 catalogue pages of books.toscrape.com, a sandbox site made for practice. Below: the code, the timings, what each tool gives you for free, when to use which, and how to run a Scrapy project in the cloud.

Scrapy vs BeautifulSoup at a glance

The same job in BeautifulSoup

With BeautifulSoup you write the loop yourself: request a page, parse it, find the next link, repeat.

import json
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

start = time.perf_counter()
session = requests.Session()
url, books = "https://books.toscrape.com/catalogue/page-1.html", []

while url:
    response = session.get(url, timeout=15)
    response.raise_for_status()
    soup = BeautifulSoup(response.content, "lxml")  # bytes: lets the parser detect the encoding
    for card in soup.select("article.product_pod"):
        books.append({
            "title": card.select_one("h3 a")["title"],
            "price": card.select_one("p.price_color").text,
            "rating": card.select_one("p.star-rating")["class"][1],
        })
    next_link = soup.select_one("li.next a")
    url = urljoin(url, next_link["href"]) if next_link else None

with open("books_bs4.json", "w", encoding="utf-8") as f:
    json.dump(books, f, ensure_ascii=False, indent=2)
print(f"{len(books)} books in {time.perf_counter() - start:.1f} s")

Install with pip install requests beautifulsoup4 lxml. It printed 1000 books in 8.8 s. Passing response.content (bytes) instead of response.text matters on this site: the server sends no charset, so requests would guess ISO-8859-1 and mangle every £, while lxml reads the charset from the page's meta tag.

The same job in Scrapy

A Scrapy spider describes what to do with each response; the framework handles the requests:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css("p.price_color::text").get(),
                "rating": card.css("p.star-rating::attr(class)").get().split()[-1],
            }
        yield from response.follow_all(css="li.next a", callback=self.parse)

Run it with pip install scrapy and scrapy runspider books_spider.py -O books.json. It scraped item_scraped_count: 1000 from 50 requests. response.follow_all resolves the relative next link, Scrapy decodes the page correctly on its own, and -O writes the file in the format the extension names.

One trap: the JSON file from runspider stores the pound sign as \u00a3, because Scrapy's JSON exporter escapes non-ASCII characters unless FEED_EXPORT_ENCODING is set to "utf-8". Projects created with scrapy startproject set it for you.

Timings: which is faster?

Following a next link is sequential by nature: you can't request page 2 before you've read its URL on page 1. So we also ran a Scrapy spider that queues all 50 page URLs up front, which is where its concurrency kicks in:

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books_fast"

    async def start(self):
        for page in range(1, 51):
            yield scrapy.Request(f"https://books.toscrape.com/catalogue/page-{page}.html")

    def parse(self, response):
        for card in response.css("article.product_pod"):
            yield {
                "title": card.css("h3 a::attr(title)").get(),
                "price": card.css("p.price_color::text").get(),
                "rating": card.css("p.star-rating::attr(class)").get().split()[-1],
            }

Three runs of each on 26 October 2026, same machine and connection, no download delay, 1,000 books every time:

Three things to read from this. When the crawl is a chain, the parser doesn't decide the speed; the network does, and both tools land in the same range. When Scrapy can run requests in parallel, it finished the job in about a quarter of the time in two of three runs. And the crash: one BeautifulSoup run died on a connection timeout to the server, because that script has no retries. Scrapy retries failed requests twice by default, so a flaky connection costs it seconds instead of the whole run. You can add retries to requests (an HTTPAdapter with Retry does it), but you have to know to do it.

Treat these as a snapshot of one machine and one site, not a benchmark. And be careful with concurrency on real sites: eight parallel requests is more than many small sites expect. Since Scrapy 2.13.3, projects created with scrapy startproject default to one request at a time per domain with a one-second delay; runspider keeps the faster built-in defaults.

What Scrapy gives you that BeautifulSoup doesn't

• A scheduler and duplicate filter. Every request goes through a queue that drops URLs it has already seen, so link loops don't trap you.
• Retries, timeouts and throttling. RETRY_TIMES, DOWNLOAD_TIMEOUT, DOWNLOAD_DELAY and AutoThrottle, which adjusts the delay to the server's response times.
• Item pipelines. Clean, validate, deduplicate and store each item in small classes instead of one long script.
• Middlewares. Change every request or response in one place: headers, proxies, cookies.
• Feed exports. JSON, JSON Lines, CSV and XML to a file, with an S3 or FTP destination if you need it.
• Stats and logs. Every run ends with request counts, response codes, item counts and the finish reason.

BeautifulSoup's advantage is the opposite: there's nothing to learn beyond selectors. It parses any HTML from any source (a file, an email, an API response, a page you rendered with Playwright), and it's often the fastest way to get one page into a list.

You can use both

Scrapy doesn't care how you parse the response. If you know BeautifulSoup's API, use it inside a callback:

import scrapy
from bs4 import BeautifulSoup


class MixedSpider(scrapy.Spider):
    name = "mixed"
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        soup = BeautifulSoup(response.text, "lxml")
        for card in soup.select("article.product_pod"):
            yield {"title": card.select_one("h3 a")["title"], "price": card.select_one("p.price_color").text}

It scraped 20 items from the homepage. Scrapy's own selectors (built on parsel and lxml) are faster and support XPath, but for moderate crawls the difference is rarely what slows you down.

When to use Scrapy and when to use BeautifulSoup

Use BeautifulSoup when:

• you scrape one page or a handful, or the URLs come from somewhere else (a spreadsheet, an API);
• you're parsing HTML you already have, such as a saved page or the output of a headless browser;
• the scraper is a step inside a bigger Python program, a notebook or a Lambda function;
• you're learning: it shows you every step that Scrapy hides.

Use Scrapy when:

• you crawl hundreds or thousands of pages, follow links across a site, or run the same crawl regularly;
• you need retries, throttling, robots.txt handling and duplicate filtering, and don't want to write them;
• several people maintain the scraper and benefit from its project structure, pipelines and settings.

Neither one runs JavaScript. For pages that render with scripts, look for the JSON API in your browser's Network tab first, then use Playwright, either on its own with BeautifulSoup parsing the result, or inside Scrapy through the scrapy-playwright plugin. Our Playwright vs Selenium comparison helps you pick the browser library, and the complete web scraping with Python guide covers requests, httpx and Playwright in more depth.

Run Scrapy (or BeautifulSoup) in the cloud on Scrape.ly

The tool choice matters less once the scraper runs every day: then you need a machine that's always on, IPs that don't get blocked, storage, schedules and alerts. Scrape.ly runs your Scrapy project as an Actor without rewriting it.

In your Scrapy project folder (the one with scrapy.cfg), install the CLI and run scrapely init:

npm install -g scrapely-cli
export SCRAPELY_API_URL=https://api.scrape.ly
scrapely login
cd bookstore
scrapely init

init detects the Scrapy project and writes three things: scrapely.json (the Actor config), requirements.txt with scrapely-python and scrapy, and a main.py wrapper that runs every spider in the project inside the Actor and pushes each scraped item to the run's dataset, after your own pipelines. Your spiders don't change. Test it locally, then deploy:

scrapely run
scrapely push
scrapely call -o

We tried this on a fresh scrapy startproject bookstore with the books spider and a CLOSESPIDER_PAGECOUNT of 3. With CLI 0.1.16, scrapely init printed Scrapy project detected (scrapy.cfg) - created main.py Actor wrapper, and scrapely run crawled 2 catalogue pages (the robots.txt request counts toward the page limit) and stored 40 books in the local dataset. On the platform, push uploads and builds the project, and every run's items land in a hosted dataset you can download as CSV, JSON or Excel, or read over the API. Note that init doesn't create a Dockerfile; the Python deploy guide shows the build options and the base images.

Once it's deployed, the rest is configuration, not code:

• Schedules. Run the spider daily or on any cron expression, in your time zone, and get an email when a run fails. See how to schedule and automate web scraping.
• Deliveries. Send each run's items to Google Sheets, Slack, email, S3 or a webhook, keep only new items, or add an AI summary.
• Cost control. Each run takes a maximum charge and stops when it reaches it.

BeautifulSoup scripts deploy the same way with a few lines of SDK code around your function; how to build a web scraper shows the wrapper and a tested scrapely run.

Or skip the scraper

For popular sites, someone has already written and maintains the scraper. The Scrape.ly Store has 570+ ready-made Actors, from Amazon to Google Maps. For any other public site, the Web Scraper Actor follows links with a CSS selector, runs a small page function you write, and returns rows at $0.000715 per result, with the residential proxy and compute included. A Mystery-category run of books.toscrape.com that we started on 23 October fetched 5 pages and saved 4 book records with title, price, stock and UPC in 12.4 seconds; at today's list price that's 4 × $0.000715 = $0.0029.

What to watch out for

• Getting blocked. Scrapy's speed is a liability on real sites without throttling. Set DOWNLOAD_DELAY or AutoThrottle and respect Retry-After; rotate residential proxies for larger crawls. Our guide to web scraping without getting blocked covers the layers.
• Encoding. Pass bytes to BeautifulSoup or set the encoding when a server sends no charset, and set FEED_EXPORT_ENCODING = "utf-8" in Scrapy for readable JSON.
• Silent selector breakage. Both tools return None or an empty list when a selector stops matching. Validate items, in a pipeline in Scrapy, and fail loudly.
• Personal data and terms. Collect public data only, respect robots.txt and site terms, and handle personal data under privacy law. Scrape.ly Store Actors only collect publicly available data shown to any visitor.

FAQ

Is Scrapy faster than BeautifulSoup?

When there are many URLs to fetch, yes: Scrapy downloads up to 16 pages at once by default, and in our test it finished 50 known pages in 1.8 to 2.5 seconds on good runs, against 8.2 to 8.8 seconds for a sequential requests and BeautifulSoup script. For a single page or a strictly sequential crawl, the network dominates and they're about the same.

Can I use BeautifulSoup with Scrapy?

Yes. Parse response.text with BeautifulSoup inside any spider callback, as in the example above. You keep Scrapy's scheduling, retries and exports and use the parsing API you know.

Is BeautifulSoup or Scrapy better for beginners?

BeautifulSoup. You see every step (request, parse, select, save) and can run it in a notebook. Move to Scrapy when your scripts start reimplementing queues, retries and exports.

Can Scrapy or BeautifulSoup scrape JavaScript websites?

Neither executes JavaScript. Look for the JSON API behind the page first. If there isn't one, render the page with Playwright and parse the HTML with BeautifulSoup, or use the scrapy-playwright plugin inside Scrapy.

What about Selenium vs Scrapy vs BeautifulSoup?

Selenium drives a real browser, so it handles JavaScript but is much slower and heavier per page. Use it (or Playwright) only for pages that need a browser, BeautifulSoup to parse, and Scrapy to crawl at scale. Our Selenium web scraping guide covers the browser side.