--- title: "Web Crawling vs Web Scraping: Key Differences | Scrape.ly" description: "Web crawling vs web scraping explained: what each does, a Python script that does both, and a real cloud crawl with its exact cost and logs." url: https://scrape.ly/blog/web-crawling-vs-web-scraping markdown: https://scrape.ly/blog/web-crawling-vs-web-scraping.md --- [Guides](https://scrape.ly/blog/category/guides.md) # Web Crawling vs Web Scraping: The Difference, With Code and Real Runs Web crawling finds pages; web scraping pulls data out of them. See the difference in one Python script, in a real crawl of crawlee.dev that cost $0.0031, and learn which one your project needs. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 8 min read ![Pixel-art scene in two halves: on the left a small robot walks along a cable linking web pages, on the right a robot with an orange >\_ on its terminal head pulls a data card out of a single web page](https://scrape.ly/media/web-crawling-vs-web-scraping.png?w=1080) Web crawling vs web scraping comes down to one question: are you finding pages, or reading them? A web crawler discovers URLs by following links or reading sitemaps; a web scraper extracts specific fields, such as a price or a title, from pages whose addresses you already have. Most real projects use both: crawl to find the product pages, then scrape each one. This guide shows the difference in a short Python script that does both jobs, then runs the same two steps in the cloud on a real site, with the exact output, logs and cost. ## Web crawling vs web scraping at a glance | | Web crawling | Web scraping | | ------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | | Question it answers | Which pages exist? | What is on this page? | | Input | One or more seed URLs or a sitemap | A list of known URLs (or a search) | | Output | URLs, or whole pages as text or Markdown | Structured records: fields like `title`, `price`, `rating` | | Core logic | Queue, link extraction, de-duplication, scope rules | Selectors or parsers for one page type | | Stops when | Page cap, depth cap, or the queue is empty | The URL list is done | | Main risk | Wandering off-site, crawl traps, too much load | Layout changes break selectors | | Typical users | Search engines, SEO audits, RAG and LLM pipelines | Price monitoring, lead lists, market research | | On Scrape.ly | [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md), $0.50 per 1,000 pages | Site-specific Actors, such as the Google Search Scraper at $3.00 per 1,000 result pages | ## What a web crawler does A crawler starts from seed URLs and repeats one loop: fetch a page, pull out its links, add the new ones to a queue, take the next URL. The design decisions are all about that loop: - **Scope.** Which links to follow: same domain, one path such as `/blog/`, or a URL pattern. Without a scope, a crawler wanders off to every site the first page links to. - **De-duplication.** The same page often has several URLs: tracking parameters, `#fragments`, trailing slashes, canonical tags. A crawler that doesn't normalise them fetches the same page many times. - **Order.** Breadth-first visits everything one click from the start before going deeper, which suits site maps. Depth-first follows one path to the bottom. - **Stopping rules.** A page cap, a depth cap, or both. - **Politeness.** Respect robots.txt, keep concurrency low, pause between requests and back off on errors. Crawlers don't have to follow links at all. Most sites publish an XML sitemap that lists their pages, so a crawler can read the sitemap and skip discovery entirely. That's how search engines and many SEO tools find pages that nothing links to. ## What a web scraper does A scraper knows what kind of page it is looking at and pulls out the fields you care about. For a product page that might be the name, price, availability and SKU; for a Google results page, the title, URL and snippet of each result. The work is in the parsing: CSS selectors or XPath for HTML, the site's own JSON where it exists, and a browser when the content only appears after JavaScript runs. Because a scraper is tied to a page layout, it is more fragile than a crawler. A redesign that renames a CSS class breaks it. That is why ready-made scrapers that someone else maintains are worth paying for on busy sites. Our guide to [scraping Google search results](https://scrape.ly/blog/scrape-google-search-results.md) is a good example of the scraping side: one known page type, structured fields out. ## Scraping vs crawling in code: one script, both jobs This script crawls one category of [books.toscrape.com](https://books.toscrape.com/), a sandbox site built for scraping practice, then scrapes the book pages it found. The `crawl` function only finds URLs; the `scrape_book` function only reads fields. Install the two libraries with `pip install requests beautifulsoup4`. ```python import re import time from collections import deque from urllib.parse import urldefrag, urljoin from urllib.robotparser import RobotFileParser import requests from bs4 import BeautifulSoup USER_AGENT = "example-crawler/1.0 (+mailto:you@example.com)" session = requests.Session() session.headers["User-Agent"] = USER_AGENT def crawl(start_url: str, follow: str, max_pages: int = 25, delay: float = 0.5) -> list[str]: """Discover URLs by following links that match the `follow` pattern, breadth first.""" robots = RobotFileParser(urljoin(start_url, "/robots.txt")) robots.read() queue, seen, visited = deque([start_url]), {start_url}, [] while queue and len(visited) < max_pages: url = queue.popleft() if not robots.can_fetch(USER_AGENT, url): continue response = session.get(url, timeout=15) if response.status_code != 200: continue visited.append(url) soup = BeautifulSoup(response.content, "html.parser") for a in soup.select("a[href]"): link = urldefrag(urljoin(url, a["href"]))[0] if re.search(follow, link) and link not in seen: seen.add(link) queue.append(link) time.sleep(delay) # one request at a time, with a pause return visited def scrape_book(url: str) -> dict: """Extract fields from one known page type: a book detail page.""" soup = BeautifulSoup(session.get(url, timeout=15).content, "html.parser") main = soup.select_one("div.product_main") return { "url": url, "title": main.select_one("h1").get_text(strip=True), "price": main.select_one("p.price_color").get_text(strip=True), "in_stock": "In stock" in main.select_one("p.availability").get_text(), "upc": soup.select_one("table.table-striped td").get_text(strip=True), } if __name__ == "__main__": start = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" book_page = r"/catalogue/[^/]+_\d+/index\.html$" urls = crawl(start, follow=rf"/category/books/mystery_3/|{book_page}") books = [u for u in urls if re.search(book_page, u)] print(f"Crawled {len(urls)} pages, {len(books)} of them book pages") for row in map(scrape_book, books[:3]): print(row) ``` We ran it on 22 October 2026 with Python 3.14, requests 2.34 and Beautiful Soup 4.15. It printed: ```plaintext Crawled 25 pages, 23 of them book pages {'url': 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html', 'title': 'Sharp Objects', 'price': '£47.82', 'in_stock': True, 'upc': 'e00eb4fd7b871a48'} {'url': 'https://books.toscrape.com/catalogue/in-a-dark-dark-wood_963/index.html', 'title': 'In a Dark, Dark Wood', 'price': '£19.63', 'in_stock': True, 'upc': '19ed25f4641d5efd'} {'url': 'https://books.toscrape.com/catalogue/the-past-never-ends_942/index.html', 'title': 'The Past Never Ends', 'price': '£56.50', 'in_stock': True, 'upc': '5ee94540d0749ea0'} ``` The `follow` pattern is the crawler's scope. In our first attempt we started from the home page and followed every same-site link: all 30 pages the crawler visited were category listings from the site menu, and not one was a book. Notice also what the crawler doesn't know: it has no idea what a price is. And the scraper has no idea where the URLs came from. Keeping the two apart is what lets you reuse each one. ## Web crawler vs web scraper: which one do you need? - **You have the URLs already** (a product list, a spreadsheet of profiles): you need a scraper only. - **You need every page of a site as text**, for search, an LLM knowledge base or a content audit: you need a crawler that saves page content. No field-level scraping required. - **You need structured fields from pages you haven't found yet**: crawl first, filter the URLs to the page type you want, then scrape. - **You need data from one big platform** such as Google, Amazon or Google Maps: use a site-specific scraper that searches the platform directly. Crawling a platform's link graph is slow and gets you blocked. - **You only need the list of URLs**: read the sitemap. It's the cheapest crawl there is. ## A website crawler on Scrape.ly: two real runs We ran both steps on [crawlee.dev](https://crawlee.dev/) on 22 October 2026, using the ready-made Store presets as they are. **Step 1: discover URLs from sitemaps.** The [Sitemap Extractor](https://scrape.ly/store/querystone/sitemap-extractor.md) with the preset [All sitemap URLs of crawlee.dev, 15 URLs](https://scrape.ly/store/querystone/sitemap-extractor/presets/thorough-discovery.md) checked robots.txt and the usual sitemap locations, found two sitemaps and saved 15 URLs in 4.3 seconds. The first five: | URL | Change freq | Priority | Found via | | ---------------------------------------------------- | ----------- | -------- | ---------- | | crawlee.dev/python/blog | weekly | 0.5 | robots.txt | | crawlee.dev/python/search | weekly | 0.5 | robots.txt | | crawlee.dev/python/api/0.6 | weekly | 0.5 | robots.txt | | crawlee.dev/python/api/0.6/class/\_AsyncSession | weekly | 0.5 | robots.txt | | crawlee.dev/python/api/0.6/class/\_AutoscaledPoolRun | weekly | 0.5 | robots.txt | ```plaintext 20261022T18:25:10.104Z INFO [Status message]: Finding sitemaps for 1 site(s)... 20261022T18:25:11.702Z INFO Done in 1.7s: 5 requests, 0.03 MB, codes {'404': 2}. Saved 15 unique URL(s) from 2 sitemap(s)feed(s) on 1 site(s). Stopped at the maximum of 15 URLs. ``` The two 404s are the extractor probing common sitemap paths that don't exist on this site. That's expected in thorough mode. **Step 2: crawl and save page content.** The [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) with the preset [Crawlee site via sitemap, 5 pages](https://scrape.ly/store/querystone/website-content-crawler/presets/sitemap-defuddle.md) used this input: ```json { "startUrls": [{ "url": "https://crawlee.dev/" }], "useSitemaps": true, "htmlTransformer": "defuddle", "maxCrawlPages": 5, "maxResults": 5 } ``` It saved 5 pages in 6.2 seconds. Each record has the page's URL, title, description, headers, clean text and Markdown: | Page | Title | Text characters | | ---------------------------------------------------------- | -------------------------------------------------------------------------------- | --------------- | | crawlee.dev/ | Crawlee · Build reliable crawlers. Fast. | 1,813 | | crawlee.dev/blog/building-gradcracker-scraper-with-crawlee | Building a Gradcracker scraper with Crawlee: anti-bot failures and redirect bugs | 10,365 | | crawlee.dev/blog/archive | Archive | 1,716 | | crawlee.dev/blog/authors | Authors | 585 | | crawlee.dev/blog | Crawlee Blog - learn how to build better scrapers | 1,117 | ```plaintext 20261022T18:25:00.268Z INFO Fetching with HTTP. Chrome is used when a page needs JavaScript or a browser input is set. 20261022T18:25:01.857Z INFO:src.crawl:Sitemap https://crawlee.dev/sitemap.xml listed 3877 pages 20261022T18:25:02.201Z INFO:src.crawl:Sitemap https://crawlee.dev/python/sitemap.xml listed 983 pages 20261022T18:25:03.770Z INFO Saved 5 content items. Timing: python 1.0s, imports 0.2s, start 0.2s, crawl 3.5s, 8 requests, median 0.7s, max 1.1s, hedged 0 (won 0) ``` The log shows the crawling side clearly: the sitemaps listed 4,860 pages between them, and the `maxCrawlPages` cap stopped the crawl at five. Remove the cap and the same input would work through the whole site. ### What the two runs cost | Run | Event fees | Platform usage | Total charged | | -------------------------------- | ---------------------- | -------------- | ------------- | | Sitemap Extractor, 15 URLs | 15 × $0.0003 = $0.0045 | $0.00035 | $0.0049 | | Website Content Crawler, 5 pages | 5 × $0.0005 = $0.0025 | $0.00064 | $0.0031 | At list price, crawling 1,000 pages costs $0.50 in page fees and listing 1,000 sitemap URLs costs $0.30, plus usage of the same kind. Both runs carried a maximum charge of $0.10; neither came close. Scrape.ly Store Website Content Crawler Crawl any public site from a start URL or its sitemap and get clean text and Markdown for every page. $0.50 per 1,000 pages. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) ### Run it from code The crawl above took about 6 seconds, so the synchronous endpoint returns the pages directly. Create a token under **Settings → API & Integrations** and export it as `SCRAPELY_TOKEN`: ```shell curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" -H "Authorization: Bearer $SCRAPELY_TOKEN" -H "Content-Type: application/json" -d '{"startUrls": [{"url": "https://crawlee.dev/"}], "useSitemaps": true, "maxCrawlPages": 5, "maxResults": 5}' ``` For bigger crawls, use the Python client (`pip install scrapely-client-python`), which waits as long as the run takes: ```python import os from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") run = client.actor("querystone/sitemap-extractor").call( run_input={"startUrls": [{"url": "https://crawlee.dev"}], "maxUrls": 15, "discoveryMode": "thorough"} ) print(run.status) # SUCCEEDED, FAILED, ABORTED or TIMED-OUT for item in client.dataset(run.default_dataset_id).iterate_items(): print(item["url"], item.get("lastmod")) ``` ## What to watch out for - **Crawl traps.** Calendars, faceted filters and session IDs in URLs can generate endless "new" pages. Use include and exclude patterns (`includeUrlGlobs`, `excludeUrlGlobs` on the Website Content Crawler) and a page cap. - **Duplicates.** Normalise URLs and trust canonical tags. The Website Content Crawler skips pages whose canonical URL it has already saved. - **JavaScript pages.** A plain HTTP fetch returns an empty shell on some sites. The Website Content Crawler fetches with HTTP first and opens Chrome only for pages that need it, which keeps simple sites cheap. - **Load and politeness.** Keep concurrency modest and honour robots.txt. The crawler has a `respectRobotsTxtFile` switch and a `maxConcurrency` limit. See our guide on [whether web scraping is legal](https://scrape.ly/blog/is-web-scraping-legal.md) for why load matters. - **One bad URL.** Long crawls hit timeouts and 404s. The Website Content Crawler writes failed URLs to a separate errors dataset and keeps going. ## How Scrape.ly does it - **Crawling without a server.** The Website Content Crawler and the Sitemap Extractor run in the cloud with proxies included, write to datasets you can export as JSON, CSV or Excel, and run on a schedule. - **Scraping without selectors.** The [Scrape.ly Store](https://scrape.ly/store.md) has 568 ready-made Actors for specific sites. For example, the [Google Maps Scraper](https://scrape.ly/store/cornerledger/google-maps-scraper.md) returns places as structured records; our [Google Maps tutorial](https://scrape.ly/blog/how-to-scrape-google-maps.md) shows a real run. - **Your own code, hosted.** Start from a Crawlee template on the [templates page](https://scrape.ly/templates.md), push it with the CLI, and it runs with the same proxies, storage and scheduling. - **For AI agents.** The Scrape.ly MCP server lets Claude, Cursor or VS Code run any of these Actors and read the results. ## FAQ ### Is Google a web crawler or a web scraper? Google Search runs a crawler, Googlebot, which discovers and fetches pages so they can be indexed. Its indexing then extracts information from those pages, so in practice search engines do both: crawl to find, then process what they found. ### Can a web crawler also scrape data? Yes. Many crawlers save page content as they go, and the Website Content Crawler saves clean text and Markdown for every page. What a general crawler doesn't do is pull site-specific fields such as a price or a rating; that's the scraper's job. ### What is a website crawler used for? Building search indexes, SEO and content audits, feeding documentation or help centers into LLM and RAG pipelines, monitoring a site for changes, and finding the URLs a scraper should visit next. ### Is web crawling legal? Crawling public pages is generally lawful, but the same rules as scraping apply: respect terms you've accepted, personal data law, copyright and the site's capacity. Robots.txt is not law in itself, but honouring it is the norm and sometimes expected by regulators. ### Which is faster, crawling or scraping? Scraping a known list of URLs is faster per useful record, because a crawler also spends requests on pages it only visits to find links. Reading a sitemap is the fastest way to discover URLs: our Sitemap Extractor run needed 5 requests to list 15 URLs. About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head carries a data card out between two routes: a cable from a browser window on the left and a cable from an API socket with curly braces and a Scrape.ly-branded plug on the right](https://scrape.ly/media/web-scraping-vs-api.png?w=750)](https://scrape.ly/blog/web-scraping-vs-api.md) ### [Web Scraping vs API: Which to Use, What Each Costs, and How to Get Both](https://scrape.ly/blog/web-scraping-vs-api.md) [Web scraping vs API: an official API gives you what the platform chooses to expose, on its terms and prices; scraping gets you what any visitor sees. Real 2026 prices for X, Google Places and Reddit, and how to call a scraper like an API.](https://scrape.ly/blog/web-scraping-vs-api.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands thinking between two matching arcade cabinets, one with the Playwright logo and one with the green Selenium logo, next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/playwright-vs-selenium.png?w=750)](https://scrape.ly/blog/playwright-vs-selenium.md) ### [Playwright vs Selenium for Web Scraping: The Same Scraper in Both (Python)](https://scrape.ly/blog/playwright-vs-selenium.md) [Playwright vs Selenium for web scraping: languages, browsers, waiting, network interception, parallel sessions and detection, from the official docs, plus the same Python scraper written in both and tested on 22 October 2026.](https://scrape.ly/blog/playwright-vs-selenium.md) - [![Pixel-art robot with an orange >\_ on its terminal head stands thinking between two matching browser windows on stands, one with the Playwright logo and one with the green Puppeteer logo, next to a crate bearing the Scrape.ly logo](https://scrape.ly/media/playwright-vs-puppeteer.png?w=750)](https://scrape.ly/blog/playwright-vs-puppeteer.md) ### [Playwright vs Puppeteer for Web Scraping: The Same Scraper in Both (Node.js)](https://scrape.ly/blog/playwright-vs-puppeteer.md) [Playwright vs Puppeteer for web scraping: languages, browsers, auto-waiting, locators, network interception, tracing and stealth plugins, checked on the official docs on 23 October 2026, plus the same Node.js scraper written in both and tested.](https://scrape.ly/blog/playwright-vs-puppeteer.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.