Web Crawling vs Web Scraping: The Difference, With Code and Real Runs

Web crawling vs web scraping comes down to one question: are you finding pages, or reading them? A web crawler discovers URLs by following links or reading sitemaps; a web scraper extracts specific fields, such as a price or a title, from pages whose addresses you already have.

Most real projects use both: crawl to find the product pages, then scrape each one. This guide shows the difference in a short Python script that does both jobs, then runs the same two steps in the cloud on a real site, with the exact output, logs and cost.

Web crawling vs web scraping at a glance

What a web crawler does

A crawler starts from seed URLs and repeats one loop: fetch a page, pull out its links, add the new ones to a queue, take the next URL. The design decisions are all about that loop:

• Scope. Which links to follow: same domain, one path such as /blog/, or a URL pattern. Without a scope, a crawler wanders off to every site the first page links to.
• De-duplication. The same page often has several URLs: tracking parameters, #fragments, trailing slashes, canonical tags. A crawler that doesn't normalise them fetches the same page many times.
• Order. Breadth-first visits everything one click from the start before going deeper, which suits site maps. Depth-first follows one path to the bottom.
• Stopping rules. A page cap, a depth cap, or both.
• Politeness. Respect robots.txt, keep concurrency low, pause between requests and back off on errors.

Crawlers don't have to follow links at all. Most sites publish an XML sitemap that lists their pages, so a crawler can read the sitemap and skip discovery entirely. That's how search engines and many SEO tools find pages that nothing links to.

What a web scraper does

A scraper knows what kind of page it is looking at and pulls out the fields you care about. For a product page that might be the name, price, availability and SKU; for a Google results page, the title, URL and snippet of each result. The work is in the parsing: CSS selectors or XPath for HTML, the site's own JSON where it exists, and a browser when the content only appears after JavaScript runs.

Because a scraper is tied to a page layout, it is more fragile than a crawler. A redesign that renames a CSS class breaks it. That is why ready-made scrapers that someone else maintains are worth paying for on busy sites. Our guide to scraping Google search results is a good example of the scraping side: one known page type, structured fields out.

Scraping vs crawling in code: one script, both jobs

This script crawls one category of books.toscrape.com, a sandbox site built for scraping practice, then scrapes the book pages it found. The crawl function only finds URLs; the scrape_book function only reads fields. Install the two libraries with pip install requests beautifulsoup4.

import re
import time
from collections import deque
from urllib.parse import urldefrag, urljoin
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

USER_AGENT = "example-crawler/1.0 (+mailto:you@example.com)"
session = requests.Session()
session.headers["User-Agent"] = USER_AGENT

def crawl(start_url: str, follow: str, max_pages: int = 25, delay: float = 0.5) -> list[str]:
    """Discover URLs by following links that match the `follow` pattern, breadth first."""
    robots = RobotFileParser(urljoin(start_url, "/robots.txt"))
    robots.read()

    queue, seen, visited = deque([start_url]), {start_url}, []
    while queue and len(visited) < max_pages:
        url = queue.popleft()
        if not robots.can_fetch(USER_AGENT, url):
            continue
        response = session.get(url, timeout=15)
        if response.status_code != 200:
            continue
        visited.append(url)
        soup = BeautifulSoup(response.content, "html.parser")
        for a in soup.select("a[href]"):
            link = urldefrag(urljoin(url, a["href"]))[0]
            if re.search(follow, link) and link not in seen:
                seen.add(link)
                queue.append(link)
        time.sleep(delay)  # one request at a time, with a pause
    return visited

def scrape_book(url: str) -> dict:
    """Extract fields from one known page type: a book detail page."""
    soup = BeautifulSoup(session.get(url, timeout=15).content, "html.parser")
    main = soup.select_one("div.product_main")
    return {
        "url": url,
        "title": main.select_one("h1").get_text(strip=True),
        "price": main.select_one("p.price_color").get_text(strip=True),
        "in_stock": "In stock" in main.select_one("p.availability").get_text(),
        "upc": soup.select_one("table.table-striped td").get_text(strip=True),
    }

if __name__ == "__main__":
    start = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
    book_page = r"/catalogue/[^/]+_\d+/index\.html$"
    urls = crawl(start, follow=rf"/category/books/mystery_3/|{book_page}")
    books = [u for u in urls if re.search(book_page, u)]
    print(f"Crawled {len(urls)} pages, {len(books)} of them book pages")
    for row in map(scrape_book, books[:3]):
        print(row)

We ran it on 22 October 2026 with Python 3.14, requests 2.34 and Beautiful Soup 4.15. It printed:

Crawled 25 pages, 23 of them book pages
{'url': 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html', 'title': 'Sharp Objects', 'price': '£47.82', 'in_stock': True, 'upc': 'e00eb4fd7b871a48'}
{'url': 'https://books.toscrape.com/catalogue/in-a-dark-dark-wood_963/index.html', 'title': 'In a Dark, Dark Wood', 'price': '£19.63', 'in_stock': True, 'upc': '19ed25f4641d5efd'}
{'url': 'https://books.toscrape.com/catalogue/the-past-never-ends_942/index.html', 'title': 'The Past Never Ends', 'price': '£56.50', 'in_stock': True, 'upc': '5ee94540d0749ea0'}

The follow pattern is the crawler's scope. In our first attempt we started from the home page and followed every same-site link: all 30 pages the crawler visited were category listings from the site menu, and not one was a book. Notice also what the crawler doesn't know: it has no idea what a price is. And the scraper has no idea where the URLs came from. Keeping the two apart is what lets you reuse each one.

Web crawler vs web scraper: which one do you need?

• You have the URLs already (a product list, a spreadsheet of profiles): you need a scraper only.
• You need every page of a site as text, for search, an LLM knowledge base or a content audit: you need a crawler that saves page content. No field-level scraping required.
• You need structured fields from pages you haven't found yet: crawl first, filter the URLs to the page type you want, then scrape.
• You need data from one big platform such as Google, Amazon or Google Maps: use a site-specific scraper that searches the platform directly. Crawling a platform's link graph is slow and gets you blocked.
• You only need the list of URLs: read the sitemap. It's the cheapest crawl there is.

A website crawler on Scrape.ly: two real runs

We ran both steps on crawlee.dev on 22 October 2026, using the ready-made Store presets as they are.

Step 1: discover URLs from sitemaps. The Sitemap Extractor with the preset All sitemap URLs of crawlee.dev, 15 URLs checked robots.txt and the usual sitemap locations, found two sitemaps and saved 15 URLs in 4.3 seconds. The first five:

2026-10-22T18:25:10.104Z INFO  [Status message]: Finding sitemaps for 1 site(s)...
2026-10-22T18:25:11.702Z INFO  Done in 1.7s: 5 requests, 0.03 MB, codes {'404': 2}. Saved 15 unique URL(s) from 2 sitemap(s)/feed(s) on 1 site(s). Stopped at the maximum of 15 URLs.

The two 404s are the extractor probing common sitemap paths that don't exist on this site. That's expected in thorough mode.

Step 2: crawl and save page content. The Website Content Crawler with the preset Crawlee site via sitemap, 5 pages used this input:

{
  "startUrls": [{ "url": "https://crawlee.dev/" }],
  "useSitemaps": true,
  "htmlTransformer": "defuddle",
  "maxCrawlPages": 5,
  "maxResults": 5
}

It saved 5 pages in 6.2 seconds. Each record has the page's URL, title, description, headers, clean text and Markdown:

2026-10-22T18:25:00.268Z INFO  Fetching with HTTP. Chrome is used when a page needs JavaScript or a browser input is set.
2026-10-22T18:25:01.857Z INFO:src.crawl:Sitemap https://crawlee.dev/sitemap.xml listed 3877 pages
2026-10-22T18:25:02.201Z INFO:src.crawl:Sitemap https://crawlee.dev/python/sitemap.xml listed 983 pages
2026-10-22T18:25:03.770Z INFO  Saved 5 content items. Timing: python 1.0s, imports 0.2s, start 0.2s, crawl 3.5s, 8 requests, median 0.7s, max 1.1s, hedged 0 (won 0)

The log shows the crawling side clearly: the sitemaps listed 4,860 pages between them, and the maxCrawlPages cap stopped the crawl at five. Remove the cap and the same input would work through the whole site.

What the two runs cost

At list price, crawling 1,000 pages costs $0.50 in page fees and listing 1,000 sitemap URLs costs $0.30, plus usage of the same kind. Both runs carried a maximum charge of $0.10; neither came close.

Run it from code

The crawl above took about 6 seconds, so the synchronous endpoint returns the pages directly. Create a token under Settings → API & Integrations and export it as SCRAPELY_TOKEN:

curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://crawlee.dev/"}], "useSitemaps": true, "maxCrawlPages": 5, "maxResults": 5}'

For bigger crawls, use the Python client (pip install scrapely-client-python), which waits as long as the run takes:

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

run = client.actor("querystone/sitemap-extractor").call(
    run_input={"startUrls": [{"url": "https://crawlee.dev"}], "maxUrls": 15, "discoveryMode": "thorough"}
)
print(run.status)  # SUCCEEDED, FAILED, ABORTED or TIMED-OUT
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["url"], item.get("lastmod"))

What to watch out for

• Crawl traps. Calendars, faceted filters and session IDs in URLs can generate endless "new" pages. Use include and exclude patterns (includeUrlGlobs, excludeUrlGlobs on the Website Content Crawler) and a page cap.
• Duplicates. Normalise URLs and trust canonical tags. The Website Content Crawler skips pages whose canonical URL it has already saved.
• JavaScript pages. A plain HTTP fetch returns an empty shell on some sites. The Website Content Crawler fetches with HTTP first and opens Chrome only for pages that need it, which keeps simple sites cheap.
• Load and politeness. Keep concurrency modest and honour robots.txt. The crawler has a respectRobotsTxtFile switch and a maxConcurrency limit. See our guide on whether web scraping is legal for why load matters.
• One bad URL. Long crawls hit timeouts and 404s. The Website Content Crawler writes failed URLs to a separate errors dataset and keeps going.

How Scrape.ly does it

• Crawling without a server. The Website Content Crawler and the Sitemap Extractor run in the cloud with proxies included, write to datasets you can export as JSON, CSV or Excel, and run on a schedule.
• Scraping without selectors. The Scrape.ly Store has 568 ready-made Actors for specific sites. For example, the Google Maps Scraper returns places as structured records; our Google Maps tutorial shows a real run.
• Your own code, hosted. Start from a Crawlee template on the templates page, push it with the CLI, and it runs with the same proxies, storage and scheduling.
• For AI agents. The Scrape.ly MCP server lets Claude, Cursor or VS Code run any of these Actors and read the results.

FAQ

Is Google a web crawler or a web scraper?

Google Search runs a crawler, Googlebot, which discovers and fetches pages so they can be indexed. Its indexing then extracts information from those pages, so in practice search engines do both: crawl to find, then process what they found.

Can a web crawler also scrape data?

Yes. Many crawlers save page content as they go, and the Website Content Crawler saves clean text and Markdown for every page. What a general crawler doesn't do is pull site-specific fields such as a price or a rating; that's the scraper's job.

What is a website crawler used for?

Building search indexes, SEO and content audits, feeding documentation or help centers into LLM and RAG pipelines, monitoring a site for changes, and finding the URLs a scraper should visit next.

Is web crawling legal?

Crawling public pages is generally lawful, but the same rules as scraping apply: respect terms you've accepted, personal data law, copyright and the site's capacity. Robots.txt is not law in itself, but honouring it is the norm and sometimes expected by regulators.

Which is faster, crawling or scraping?

Scraping a known list of URLs is faster per useful record, because a crawler also spends requests on pages it only visits to find links. Reading a sitemap is the fastest way to discover URLs: our Sitemap Extractor run needed 5 requests to list 15 URLs.