--- title: "Scrapy vs BeautifulSoup: Which to Use | Scrape.ly" description: "Scrapy vs BeautifulSoup tested on the same 1,000-book job: timings, retries, concurrency, exports, when to use which, and running Scrapy in the cloud." url: https://scrape.ly/blog/scrapy-vs-beautifulsoup markdown: https://scrape.ly/blog/scrapy-vs-beautifulsoup.md --- [Guides](https://scrape.ly/blog/category/guides.md) # Scrapy vs BeautifulSoup: Which Python Scraping Tool to Use (Tested) Scrapy vs BeautifulSoup, tested on the same 1,000-book job: what each tool is, measured run times, retries, concurrency, exports and JavaScript pages, when to pick which, and how to run a Scrapy project in the cloud with one command. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 26, 2026 - Reading time 9 min read ![Pixel-art robot with an orange >\_ on its terminal head stands between two matching arcade cabinets, one with the teal Scrapy logo and one with a steaming soup bowl, thinking of the Scrape.ly mark next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/scrapy-vs-beautifulsoup.png?w=1080) Scrapy vs BeautifulSoup is not a contest between two equal tools: BeautifulSoup is an HTML parser, and Scrapy is a complete crawling framework. BeautifulSoup turns HTML you've already downloaded into something you can query; Scrapy downloads pages concurrently, retries failures, follows links, throttles itself and exports the results, and has its own selectors for parsing. So the real question is whether your job needs a framework. To answer it with numbers, we ran the same job with both on 26 October 2026: collect all 1,000 books from the 50 catalogue pages of [books.toscrape.com](https://books.toscrape.com/), a sandbox site made for practice. Below: the code, the timings, what each tool gives you for free, when to use which, and how to run a Scrapy project in the cloud. ## Scrapy vs BeautifulSoup at a glance | | BeautifulSoup | Scrapy | | ---------------------- | -------------------------------------------------------- | ----------------------------------------------------- | | What it is | HTML/XML parser library | Crawling and scraping framework | | Version tested | beautifulsoup4 4.15.0 (with requests 2.34.2, lxml 6.1.3) | Scrapy 2.19.0 | | Downloads pages | No, pair it with requests or httpx | Yes, built in | | Concurrency | None, you add it | 16 requests at once, 8 per domain by default | | Retries | None, you add it | 2 retries on 408, 429 and common 5xx codes by default | | Selectors | CSS (`select`), `find`/`find_all` | CSS and XPath (`response.css`, `response.xpath`) | | Export | You write CSV/JSON yourself | `-O books.json` or `.csv`, `.jsonl`, `.xml` | | robots.txt, throttling | You implement them | `ROBOTSTXT_OBEY`, `DOWNLOAD_DELAY`, AutoThrottle | | JavaScript pages | No (use Playwright, then parse) | No (add a plugin such as scrapy-playwright) | | Learning curve | Minutes | An afternoon: projects, spiders, settings, pipelines | | Best for | Single pages, small scripts, parsing any HTML | Crawls of hundreds to millions of pages | ## The same job in BeautifulSoup With BeautifulSoup you write the loop yourself: request a page, parse it, find the next link, repeat. ```python import json import time from urllib.parse import urljoin import requests from bs4 import BeautifulSoup start = time.perf_counter() session = requests.Session() url, books = "https://books.toscrape.com/catalogue/page-1.html", [] while url: response = session.get(url, timeout=15) response.raise_for_status() soup = BeautifulSoup(response.content, "lxml") # bytes: lets the parser detect the encoding for card in soup.select("article.product_pod"): books.append({ "title": card.select_one("h3 a")["title"], "price": card.select_one("p.price_color").text, "rating": card.select_one("p.star-rating")["class"][1], }) next_link = soup.select_one("li.next a") url = urljoin(url, next_link["href"]) if next_link else None with open("books_bs4.json", "w", encoding="utf-8") as f: json.dump(books, f, ensure_ascii=False, indent=2) print(f"{len(books)} books in {time.perf_counter() - start:.1f} s") ``` Install with `pip install requests beautifulsoup4 lxml`. It printed `1000 books in 8.8 s`. Passing `response.content` (bytes) instead of `response.text` matters on this site: the server sends no charset, so `requests` would guess ISO-8859-1 and mangle every `£`, while lxml reads the charset from the page's meta tag. ## The same job in Scrapy A Scrapy spider describes what to do with each response; the framework handles the requests: ```python import scrapy class BooksSpider(scrapy.Spider): name = "books" start_urls = ["https://books.toscrape.com/catalogue/page-1.html"] def parse(self, response): for card in response.css("article.product_pod"): yield { "title": card.css("h3 a::attr(title)").get(), "price": card.css("p.price_color::text").get(), "rating": card.css("p.star-rating::attr(class)").get().split()[1], } yield from response.follow_all(css="li.next a", callback=self.parse) ``` Run it with `pip install scrapy` and `scrapy runspider books_spider.py -O books.json`. It scraped `item_scraped_count: 1000` from 50 requests. `response.follow_all` resolves the relative next link, Scrapy decodes the page correctly on its own, and `-O` writes the file in the format the extension names. One trap: the JSON file from `runspider` stores the pound sign as `\u00a3`, because Scrapy's JSON exporter escapes non-ASCII characters unless `FEED_EXPORT_ENCODING` is set to `"utf-8"`. Projects created with `scrapy startproject` set it for you. ## Timings: which is faster? Following a **next** link is sequential by nature: you can't request page 2 before you've read its URL on page 1. So we also ran a Scrapy spider that queues all 50 page URLs up front, which is where its concurrency kicks in: ```python import scrapy class BooksSpider(scrapy.Spider): name = "books_fast" async def start(self): for page in range(1, 51): yield scrapy.Request(f"https://books.toscrape.com/catalogue/page-{page}.html") def parse(self, response): for card in response.css("article.product_pod"): yield { "title": card.css("h3 a::attr(title)").get(), "price": card.css("p.price_color::text").get(), "rating": card.css("p.star-rating::attr(class)").get().split()[1], } ``` Three runs of each on 26 October 2026, same machine and connection, no download delay, 1,000 books every time: | Approach | Requests | Times measured | | ------------------------------------------ | --------------------- | --------------------- | | requests + BeautifulSoup, follow next link | 50, one at a time | 8.8 s, 8.2 s, 1 crash | | Scrapy, follow next link | 50, one at a time | 27.2 s, 11.4 s, 8.6 s | | Scrapy, 50 URLs queued up front | 50, up to 8 at a time | 10.6 s, 1.8 s, 2.5 s | Three things to read from this. When the crawl is a chain, the parser doesn't decide the speed; the network does, and both tools land in the same range. When Scrapy can run requests in parallel, it finished the job in about a quarter of the time in two of three runs. And the crash: one BeautifulSoup run died on a connection timeout to the server, because that script has no retries. Scrapy retries failed requests twice by default, so a flaky connection costs it seconds instead of the whole run. You can add retries to requests (an `HTTPAdapter` with `Retry` does it), but you have to know to do it. Treat these as a snapshot of one machine and one site, not a benchmark. And be careful with concurrency on real sites: eight parallel requests is more than many small sites expect. Since Scrapy 2.13.3, projects created with `scrapy startproject` default to one request at a time per domain with a one-second delay; `runspider` keeps the faster built-in defaults. ## What Scrapy gives you that BeautifulSoup doesn't - **A scheduler and duplicate filter.** Every request goes through a queue that drops URLs it has already seen, so link loops don't trap you. - **Retries, timeouts and throttling.** `RETRY_TIMES`, `DOWNLOAD_TIMEOUT`, `DOWNLOAD_DELAY` and AutoThrottle, which adjusts the delay to the server's response times. - **Item pipelines.** Clean, validate, deduplicate and store each item in small classes instead of one long script. - **Middlewares.** Change every request or response in one place: headers, proxies, cookies. - **Feed exports.** JSON, JSON Lines, CSV and XML to a file, with an S3 or FTP destination if you need it. - **Stats and logs.** Every run ends with request counts, response codes, item counts and the finish reason. BeautifulSoup's advantage is the opposite: there's nothing to learn beyond selectors. It parses any HTML from any source (a file, an email, an API response, a page you rendered with Playwright), and it's often the fastest way to get one page into a list. ### You can use both Scrapy doesn't care how you parse the response. If you know BeautifulSoup's API, use it inside a callback: ```python import scrapy from bs4 import BeautifulSoup class MixedSpider(scrapy.Spider): name = "mixed" start_urls = ["https://books.toscrape.com/"] def parse(self, response): soup = BeautifulSoup(response.text, "lxml") for card in soup.select("article.product_pod"): yield {"title": card.select_one("h3 a")["title"], "price": card.select_one("p.price_color").text} ``` It scraped 20 items from the homepage. Scrapy's own selectors (built on parsel and lxml) are faster and support XPath, but for moderate crawls the difference is rarely what slows you down. ## When to use Scrapy and when to use BeautifulSoup Use **BeautifulSoup** when: - you scrape one page or a handful, or the URLs come from somewhere else (a spreadsheet, an API); - you're parsing HTML you already have, such as a saved page or the output of a headless browser; - the scraper is a step inside a bigger Python program, a notebook or a Lambda function; - you're learning: it shows you every step that Scrapy hides. Use **Scrapy** when: - you crawl hundreds or thousands of pages, follow links across a site, or run the same crawl regularly; - you need retries, throttling, robots.txt handling and duplicate filtering, and don't want to write them; - several people maintain the scraper and benefit from its project structure, pipelines and settings. Neither one runs JavaScript. For pages that render with scripts, look for the JSON API in your browser's Network tab first, then use Playwright, either on its own with BeautifulSoup parsing the result, or inside Scrapy through the scrapy-playwright plugin. Our [Playwright vs Selenium comparison](https://scrape.ly/blog/playwright-vs-selenium.md) helps you pick the browser library, and the complete [web scraping with Python guide](https://scrape.ly/blog/web-scraping-python.md) covers requests, httpx and Playwright in more depth. ## Run Scrapy (or BeautifulSoup) in the cloud on Scrape.ly The tool choice matters less once the scraper runs every day: then you need a machine that's always on, IPs that don't get blocked, storage, schedules and alerts. Scrape.ly runs your Scrapy project as an Actor without rewriting it. In your Scrapy project folder (the one with `scrapy.cfg`), install the CLI and run `scrapely init`: ```shell npm install -g scrapely-cli export SCRAPELY_API_URL=https://api.scrape.ly scrapely login cd bookstore scrapely init ``` `init` detects the Scrapy project and writes three things: `scrapely.json` (the Actor config), `requirements.txt` with `scrapely-python` and `scrapy`, and a `main.py` wrapper that runs every spider in the project inside the Actor and pushes each scraped item to the run's dataset, after your own pipelines. Your spiders don't change. Test it locally, then deploy: ```shell scrapely run scrapely push scrapely call -o ``` We tried this on a fresh `scrapy startproject bookstore` with the books spider and a `CLOSESPIDER_PAGECOUNT` of 3. With CLI 0.1.16, `scrapely init` printed `Scrapy project detected (scrapy.cfg) - created main.py Actor wrapper`, and `scrapely run` crawled 2 catalogue pages (the robots.txt request counts toward the page limit) and stored 40 books in the local dataset. On the platform, `push` uploads and builds the project, and every run's items land in a hosted dataset you can download as CSV, JSON or Excel, or read over the API. Note that `init` doesn't create a Dockerfile; the [Python deploy guide](https://scrape.ly/docs/sdk-python-deploy.md) shows the build options and the base images. Once it's deployed, the rest is configuration, not code: - **Schedules.** Run the spider daily or on any cron expression, in your time zone, and get an email when a run fails. See [how to schedule and automate web scraping](https://scrape.ly/blog/schedule-web-scraping.md). - **Deliveries.** Send each run's items to Google Sheets, Slack, email, S3 or a webhook, keep only new items, or add an AI summary. - **Cost control.** Each run takes a maximum charge and stops when it reaches it. BeautifulSoup scripts deploy the same way with a few lines of SDK code around your function; [how to build a web scraper](https://scrape.ly/blog/how-to-build-a-web-scraper.md) shows the wrapper and a tested `scrapely run`. ### Or skip the scraper For popular sites, someone has already written and maintains the scraper. The [Scrape.ly Store](https://scrape.ly/store.md) has 570+ ready-made Actors, from Amazon to Google Maps. For any other public site, the [Web Scraper](https://scrape.ly/store/querystone/web-scraper.md) Actor follows links with a CSS selector, runs a small page function you write, and returns rows at $0.000715 per result, with the residential proxy and compute included. A Mystery-category run of books.toscrape.com that we started on 23 October fetched 5 pages and saved 4 book records with title, price, stock and UPC in 12.4 seconds; at today's list price that's 4 × $0.000715 = $0.0029. Scrape.ly Store Web Scraper Follow links with a CSS selector and extract fields with a small page function, with no Scrapy project to maintain. $0.71 per 1,000 results. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) ## What to watch out for - **Getting blocked.** Scrapy's speed is a liability on real sites without throttling. Set `DOWNLOAD_DELAY` or AutoThrottle and respect `Retry-After`; rotate residential proxies for larger crawls. Our guide to [web scraping without getting blocked](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) covers the layers. - **Encoding.** Pass bytes to BeautifulSoup or set the encoding when a server sends no charset, and set `FEED_EXPORT_ENCODING = "utf-8"` in Scrapy for readable JSON. - **Silent selector breakage.** Both tools return `None` or an empty list when a selector stops matching. Validate items, in a pipeline in Scrapy, and fail loudly. - **Personal data and terms.** Collect public data only, respect `robots.txt` and site terms, and handle personal data under privacy law. Scrape.ly Store Actors only collect publicly available data shown to any visitor. ## FAQ ### Is Scrapy faster than BeautifulSoup? When there are many URLs to fetch, yes: Scrapy downloads up to 16 pages at once by default, and in our test it finished 50 known pages in 1.8 to 2.5 seconds on good runs, against 8.2 to 8.8 seconds for a sequential requests and BeautifulSoup script. For a single page or a strictly sequential crawl, the network dominates and they're about the same. ### Can I use BeautifulSoup with Scrapy? Yes. Parse `response.text` with BeautifulSoup inside any spider callback, as in the example above. You keep Scrapy's scheduling, retries and exports and use the parsing API you know. ### Is BeautifulSoup or Scrapy better for beginners? BeautifulSoup. You see every step (request, parse, select, save) and can run it in a notebook. Move to Scrapy when your scripts start reimplementing queues, retries and exports. ### Can Scrapy or BeautifulSoup scrape JavaScript websites? Neither executes JavaScript. Look for the JSON API behind the page first. If there isn't one, render the page with Playwright and parse the HTML with BeautifulSoup, or use the scrapy-playwright plugin inside Scrapy. ### What about Selenium vs Scrapy vs BeautifulSoup? Selenium drives a real browser, so it handles JavaScript but is much slower and heavier per page. Use it (or Playwright) only for pages that need a browser, BeautifulSoup to parse, and Scrapy to crawl at scale. Our [Selenium web scraping guide](https://scrape.ly/blog/selenium-web-scraping.md) covers the browser side. About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from the one orange-lit globe in a ring of four blue-and-green globes turning round a small server, amber arrowheads marking the rotation, towards a web page with an image block](https://scrape.ly/media/rotating-proxies-for-web-scraping.png?w=750)](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) ### [Rotating Proxies for Web Scraping: How Rotation Works, with Tested Code](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) [Rotating proxies for web scraping spread your requests over many IP addresses. Per-request rotation vs sticky sessions, residential vs datacenter pools, tested Python and Crawlee code, and what rotation costs per GB on Scrape.ly and Evomi.](https://scrape.ly/blog/rotating-proxies-for-web-scraping.md) - [![Pixel-art robot with an orange >\_ on its terminal head, disguised in a dark fedora, a raised collar and a cloak, pulls a data card out of a browser window showing the green Puppeteer logo and drops it into a crate](https://scrape.ly/media/puppeteer-stealth.png?w=750)](https://scrape.ly/blog/puppeteer-stealth.md) ### [Puppeteer Stealth: Setup, What It Patches, and Where It Stops](https://scrape.ly/blog/puppeteer-stealth.md) [Puppeteer stealth is puppeteer-extra-plugin-stealth, a plugin that hides the signs of a headless Chrome. Its maintenance status in October 2026, tested setup with Puppeteer 25, before-and-after results, the header mismatches it creates, a Crawlee setup and the hosted alternative.](https://scrape.ly/blog/puppeteer-stealth.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a row of data from a web page on the left into a table on the right that is filling up row by row](https://scrape.ly/media/what-is-web-scraping.png?w=750)](https://scrape.ly/blog/what-is-web-scraping.md) ### [What Is Web Scraping? How It Works and What It Is Used For](https://scrape.ly/blog/what-is-web-scraping.md) [What is web scraping? The automated collection of data from websites into structured records. See how it works step by step with tested Python code, what teams use it for, how it differs from crawling, APIs and screen scraping, and how to do it without code.](https://scrape.ly/blog/what-is-web-scraping.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.