What Is Web Scraping? How It Works and What It Is Used For

What is web scraping? It's the automated collection of data from websites: a program loads web pages, picks out the fields you care about, such as prices, names, ratings or dates, and saves them as structured data in a spreadsheet, a database or JSON. It does what you would do by copying and pasting from a browser, only for thousands of pages and on a schedule.

This guide explains how web scraping works step by step, with a short Python scraper we tested, how it compares with web crawling, APIs and screen scraping, what teams use it for, where the legal lines are, and how to get the same data without writing or hosting any code.

What is web scraping, in plain words?

A web scraper answers one question again and again: "what does this page say?" It knows what kind of page it is reading (a product page, a search result, a job listing) and where the useful fields sit on it. Feed it a list of URLs and you get back one clean record per page or per item.

The output is the point. A product page is made for people to read; a scraper turns it into something like {"title": "Sharp Objects", "price": 47.82, "in_stock": true}, which you can sort, chart, compare with yesterday's prices or feed into an AI model.

Web scraping vs web crawling vs APIs vs screen scraping

These four terms get mixed up constantly. Here is how they differ:

In practice they combine. A typical project crawls a site to find product URLs, scrapes each product page, and uses an official API where one exists and covers the data at a fair price.

How web scraping works, step by step

Every scraper, from a ten-line script to a cloud platform, runs the same five steps.

1. Choose the pages. Start from a list of URLs, a search, a category page, or a sitemap. If you don't know the URLs yet, a crawler finds them first.
2. Fetch each page. Send an HTTP request, the same one your browser sends, and get back the page's HTML. Some sites build their content with JavaScript after the page loads; for those you need a real browser (headless Chrome via Playwright, Puppeteer or Selenium) or the JSON the page loads in the background.
3. Parse the HTML. Turn the raw text into a tree you can query, then locate the fields with CSS selectors or XPath, such as p.price_color for a price.
4. Clean and structure. Strip currency symbols, convert "Four stars" into 4, normalise dates, drop duplicates, and check that required fields are present.
5. Store and repeat. Save the records as CSV, JSON, Excel or rows in a database, and run the job again on a schedule so the data stays fresh.

A web scraper in nine lines of Python

This script scrapes the Travel category of books.toscrape.com, a sandbox shop built for scraping practice. Install the two libraries with pip install requests beautifulsoup4:

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/catalogue/category/books/travel_2/index.html"
response = requests.get(url, headers={"User-Agent": "my-first-scraper/1.0"}, timeout=15)
soup = BeautifulSoup(response.content, "html.parser")

for book in soup.select("article.product_pod"):
    title = book.select_one("h3 a")["title"]
    price = book.select_one("p.price_color").get_text(strip=True)
    print(f"{price}  {title}")

We ran it on 23 October 2026 with Python 3.12, requests 2.34 and Beautiful Soup 4.15. It printed all 11 books in the category; the first four lines:

£45.17  It's Only the Himalayas
£49.43  Full Moon over Noah’s Ark: An Odyssey to Mount Ararat and Beyond
£48.87  See America: A Celebration of Our National Parks & Treasured Sites
£36.94  Vagabonding: An Uncommon Guide to the Art of Long-Term World Travel

Steps 2 to 4 are all here: requests.get fetches, BeautifulSoup parses, and the two select_one calls extract. Two details matter. The title comes from the link's title attribute because the visible text is cut off for long titles, and we pass response.content (bytes) rather than response.text so Beautiful Soup detects the encoding itself and the pound sign comes out right.

A real project adds the rest: following the "next" link to page through results, pauses between requests, retries, and saving to a file. Our guide on how to scrape data from a website builds that version and compares it with three no-code and AI routes, and the Python web scraping guide goes deeper into the libraries.

Four kinds of web scrapers

Scrapers differ mainly in how they fetch and read a page. Picking the lightest one that works keeps a project fast and cheap.

• HTTP scrapers. Download the HTML and parse it, like the script above. Fast and cheap, and enough for most catalogue, news and directory sites, because the data is already in the HTML the server sends.
• Browser scrapers. Drive headless Chrome or Firefox with Playwright, Puppeteer or Selenium, wait for scripts to run, then read the finished page. Needed for single-page apps, infinite scroll and content behind clicks, but heavier: the Scrape.ly Web Scraper's documentation puts the same 19-page crawl at about $0.0015 of platform usage over HTTP and about $0.045 in Chrome.
• Data-call scrapers. Many pages load their content from a JSON endpoint in the background. Find that request in your browser's developer tools (Network tab) and call it directly: you skip both rendering and parsing.
• AI-assisted scrapers. A language model reads the page text and returns the fields you describe in plain words. Handy for one-off jobs and messy layouts; slower and pricier per page than selectors, so teams usually let AI write the selectors and run them conventionally.

Ready-made scrapers mix these approaches per site. That's why the Web Scraper fetches over HTTP by default and opens Chromium only when you set useChrome.

What is data scraping?

Data scraping is the wider term: extracting data from any output meant for people rather than programs. Web scraping is the most common kind. Others include parsing PDFs and reports, reading the output of terminal programs, and screen scraping from desktop applications or old mainframe screens. When people search for "data scraping" they almost always mean web scraping, and the techniques above apply.

What web scraping is used for

Web scraping is how most public web data ends up in spreadsheets, dashboards and models. The common jobs:

• Price monitoring. Retailers and brands track competitor prices, stock levels and promotions daily. See our guide to price scraping for a worked setup.
• Lead generation. Sales teams build prospect lists from business directories and maps: company name, category, website, phone and rating. Our lead generation scraping guide covers the sources and the rules around contact data.
• Job market data. Recruiters, job boards and labour-market analysts collect postings, salaries and skills across sites. See job scraping.
• Social media and brand monitoring. Marketing teams follow public posts, comments and creator metrics to measure reach and sentiment. See social media scraping.
• Market research and reviews. Product teams read thousands of public reviews to find complaints and feature requests faster than any survey.
• SEO and search. Agencies track rankings, featured snippets and competitors' pages in Google results.
• AI and LLM data. Retrieval pipelines and AI agents need fresh, clean page text, and a scraper is how they read the live web.
• Research and journalism. Researchers and reporters collect public records, news archives and government data sets that have no download button.

Is web scraping legal?

Collecting publicly available data is a common, lawful activity in many contexts, but the answer depends on what you collect and how. Four questions decide most cases: Did you accept the site's terms, for example by logging in? Does the data include personal information covered by privacy law such as the GDPR or CCPA? Is the content protected by copyright or database rights you would be copying? And does your scraper put real load on the site? Stay on public pages, collect only what you need, and keep request rates modest. Our guide is web scraping legal? covers the court cases and the rules in detail. This section is general information, not legal advice.

How Scrape.ly does web scraping

Writing a scraper is the easy part. Keeping it running is the work: proxies when a site rate-limits you, a browser when content needs JavaScript, fixes when the layout changes, and a server and scheduler to run it every day. Scrape.ly takes that work off your plate in three ways.

A Store of ready-made scrapers. The Scrape.ly Store has 568 Actors, which are maintained scrapers for specific sites and jobs: Google Maps, Amazon, LinkedIn Jobs, Reddit, Zillow and hundreds more. You fill in a form (a search term, a location, a limit), click Start, and download the results as JSON, CSV or Excel. Each Actor shows its price per result up front; the Google Maps Scraper, for example, costs $3.60 per 1,000 places.

One API for every scraper, plus generic ones. Every Actor runs behind the same REST API and clients for Python and JavaScript. For a site without a dedicated Actor, the Web Scraper runs your own extraction function on each page in the cloud. We pointed it at the same sandbox shop, told it to follow book links from the Mystery category, and ran it on 23 October 2026 with a $0.05 maximum charge. It saved four books in 12 seconds:

The run cost $0.0029 in total: 4 results × $0.0005 = $0.002 in result fees, plus $0.0009 of compute, proxy and storage. The residential proxy took one input flag, with no proxy account to set up. A shorter version of the same job from Python (pip install scrapely-client-python):

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

page_function = """async function pageFunction(context) {
    const $ = context.jQuery;
    if (!$('div.product_main').length) return null;
    return {
        url: context.request.url,
        title: $('div.product_main h1').text().trim(),
        price: $('div.product_main p.price_color').text().trim(),
    };
}"""

run = client.actor("querystone/web-scraper").call(
    run_input={
        "startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"}],
        "linkSelector": "article.product_pod h3 a",
        "pageFunction": page_function,
        "maxResultsPerCrawl": 5,
        "proxyConfiguration": {"useScrapelyProxy": True},
    }
)
print(run.status)  # SUCCEEDED, FAILED, ABORTED or TIMED-OUT
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item.get("title"), item.get("price"))

Scraping for AI agents. The Scrape.ly MCP server at https://api.scrape.ly/mcp lets Claude, Cursor, VS Code, ChatGPT and other agents search the Store, run any Actor and read the results. In Claude Code, add it with claude mcp add --transport http scrapely https://api.scrape.ly/mcp, sign in with /mcp, and ask: "Use scrapely to get the 20 best-rated cafés in Lisbon from Google Maps as a CSV with name, rating and website."

The free plan includes $5 of usage every month with no credit card, and every run can carry a maximum charge, so a test never costs more than you set.

What to watch out for

• JavaScript-heavy pages. If requests returns a page without your data, the content is drawn by scripts. Use the page's own JSON or a browser; on Scrape.ly, switch on useChrome in the Web Scraper.
• Blocking. Busy sites rate-limit repeated requests from one IP. Slow down first, then use proxies; Store Actors include them.
• Layout changes. Selectors break when a site redesigns. Validate your output on every run, and prefer maintained Actors for sites that change often.
• Personal data. Reviews, profiles and comments are written by people. Collect only the fields you need and delete what you no longer use.

Our web scraping best practices guide turns these into 12 concrete habits, with code.

FAQ

Is web scraping the same as data scraping?

Almost. Data scraping is the broader term for extracting data from anything built for human eyes, including PDFs, terminal output and desktop screens. Web scraping is data scraping applied to websites, and it's what most people mean by either term.

What is an example of web scraping?

A retailer that collects competitors' product prices every morning is scraping the web. So is a recruiter who pulls new job postings into a spreadsheet, or a researcher who downloads thousands of public reviews to study complaints. The Python script above, which lists the books and prices in one shop category, is the smallest working example.

Do I need to know how to code to scrape a website?

No. Ready-made scrapers such as the Actors in the Scrape.ly Store take a form input and return a spreadsheet. Code helps when no ready-made scraper covers your site, and even then a generic scraper with a short extraction function is often enough.

Can websites block web scraping?

Yes. Sites rate-limit, block IP addresses, show CAPTCHAs or require a login. Polite scrapers keep request rates low, respect robots.txt and back off on errors, and managed platforms add proxy rotation and real browsers for sites that need them.

What programming language is best for web scraping?

Python is the most popular choice thanks to requests, Beautiful Soup, Scrapy and Playwright. JavaScript with Node.js is a close second and strong for browser automation with Puppeteer and Playwright. Crawlee, the framework Scrape.ly's SDKs build on, works in both.