How to Scrape Data From a Website: 4 Methods, No-Code to Python

How to scrape data from a website comes down to one decision: build the scraper yourself, or use one that already exists. If a ready-made scraper covers the site, you can have a spreadsheet in under a minute without code; if not, a generic cloud scraper, a short Python script or an AI agent will get the data.

This guide walks through all four methods, from no-code to Python, on real targets. Every run below happened on 22 or 23 October 2026, and every cost is what we were actually charged.

How to scrape data from a website: 4 methods at a glance

Before you start: three quick checks

1. Is the data public? Stick to what a logged-out visitor can see. Data behind a login comes with terms you accepted, and personal data comes with privacy law. Our web scraping guide covers the basics.
2. Does a scraper already exist? Search the Scrape.ly Store for the site's name. If there's a match, method 1 is the fastest route by far.
3. Is the data in the HTML? Open the page, right-click, choose View page source and search for a value you want, such as a price. If it's there, plain HTTP scraping works. If it isn't, the page builds its content with JavaScript and you need a browser or the page's own data request.

Method 1: Use a ready-made scraper (no code)

The Scrape.ly Store has 568 Actors, each a maintained scraper for one site or job. You fill in a form, start the run and download the results. Nothing to install, and when the site changes its layout, the Actor's maintainer updates it.

Here's the whole process with the Google Maps Scraper:

1. Open the Actor page and pick the preset Restaurants in Konstanz, 10 places. It fills the form with this input:

{
  "searchStringsArray": ["restaurant"],
  "locationQuery": "Konstanz",
  "maxCrawledPlacesPerSearch": 10,
  "language": "en"
}

2. Click Start. Our run on 22 October 2026 took 5.7 seconds and saved 10 places:

2026-10-22T15:22:55.025Z [scrapely.actor] INFO  Saved 10 places in 3.2s (11 requests, 0 hedged, 0.93 MB); start-up: python 1.13s, SDK import 0.00s, SDK init 0.27s

3. Open the results and export them as CSV, Excel or JSON. The first three rows:

Each record also carries the phone number, website, opening hours, coordinates and more. The run cost $0.036888: 10 places × $0.0036 = $0.036 in place fees, plus $0.0009 of compute and proxy. The free plan's $5 monthly credit covers about 1,350 places at that rate.

To change the job, edit the search term, location and limit, and click Start again. To repeat it every morning, add a schedule under Automations.

Method 2: Use a generic cloud scraper (one function)

When no Actor exists for your site, a generic scraper does the fetching, link following, proxies and retries, and you supply only the extraction logic. Scrape.ly has two:

• Web Scraper runs your JavaScript page function on each page and saves whatever object it returns. Best for structured fields: prices, titles, specs. $0.50 per 1,000 results.
• Website Content Crawler crawls a site from a start URL or its sitemap and saves each page's clean text and Markdown. Best for documentation, knowledge bases and LLM pipelines. $0.50 per 1,000 pages.

Web Scraper: four book pages in 12 seconds

We pointed the Web Scraper at the Mystery category of books.toscrape.com, a sandbox shop built for scraping practice, and told it to follow each book link and pull four fields from the book page:

{
  "startUrls": [{ "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" }],
  "linkSelector": "article.product_pod h3 a",
  "globs": [{ "glob": "https://books.toscrape.com/catalogue/*/index.html" }],
  "pageFunction": "async function pageFunction(context) {\n  const $ = context.jQuery;\n  if (!$('div.product_main').length) return null;\n  return {\n    url: context.request.url,\n    title: $('div.product_main h1').text().trim(),\n    price: $('div.product_main p.price_color').text().trim(),\n    inStock: $('div.product_main p.availability').text().includes('In stock'),\n    upc: $('table.table-striped td').first().text().trim()\n  };\n}",
  "maxPagesPerCrawl": 6,
  "maxResultsPerCrawl": 5,
  "maxConcurrency": 2,
  "respectRobotsTxtFile": true,
  "proxyConfiguration": { "useScrapelyProxy": true }
}

The fields that matter:

• linkSelector and globs decide which links get followed: here only the book links on the listing page.
• pageFunction gets jQuery on each page. On the listing page it returns null, so the page is only used to find links; on a book page it returns the record.
• maxResultsPerCrawl stops the run after five records; maxConcurrency keeps the load on the site low.
• respectRobotsTxtFile skips links the site's robots.txt disallows.

The run on 23 October 2026 fetched 5 pages and saved 4 book records in 12 seconds:

2026-10-23T03:02:50.015Z [scrapely.proxy_configuration] INFO  Platform proxy access check succeeded.
2026-10-23T03:02:58.070Z [scrapely.actor] INFO  Done: 5 pages fetched, 4 records saved in 8 s

It was charged $0.002919: 4 results × $0.0005 = $0.002, plus $0.0009 of compute, proxy and storage. The listing page's null record wasn't charged. For pages that only show data after JavaScript runs, set "useChrome": true and the same page function runs on the rendered page.

Website Content Crawler: a whole site as text

If you want every page's text rather than specific fields, the crawler is simpler: give it a start URL and a page cap. On 22 October 2026 it read crawlee.dev's sitemaps (4,860 pages listed) and saved the first 5 pages as clean text and Markdown in 6.2 seconds, for $0.0031. Runs that short can use the synchronous endpoint, which returns the items in the same response:

curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://crawlee.dev/"}], "useSitemaps": true, "maxCrawlPages": 5, "maxResults": 5}'

Create the token under Settings → API & Integrations and export it as SCRAPELY_TOKEN. Add "includeUrlGlobs" to keep the crawl inside one section, such as /docs/.

Method 3: Write a Python scraper

Code gives you full control and costs nothing but time. This script scrapes every book in the same Mystery category, pages through the results by following the "next" link, and writes a CSV. Install the libraries with pip install requests beautifulsoup4:

import csv
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

START = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

session = requests.Session()
session.headers["User-Agent"] = "books-scraper/1.0 (+mailto:you@example.com)"

def scrape_listing(url):
    """Return the books on one listing page and the URL of the next page (or None)."""
    soup = BeautifulSoup(session.get(url, timeout=15).content, "html.parser")
    rows = []
    for book in soup.select("article.product_pod"):
        link = book.select_one("h3 a")
        rows.append({
            "title": link["title"],
            "price_gbp": float(book.select_one("p.price_color").get_text(strip=True).lstrip("£")),
            "rating": RATINGS[book.select_one("p.star-rating")["class"][1]],
            "in_stock": "In stock" in book.select_one("p.availability").get_text(),
            "url": urljoin(url, link["href"]),
        })
    next_link = soup.select_one("li.next a")
    return rows, urljoin(url, next_link["href"]) if next_link else None

books, url = [], START
while url:
    rows, url = scrape_listing(url)
    books.extend(rows)
    time.sleep(1)  # be polite: one page per second

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=books[0].keys())
    writer.writeheader()
    writer.writerows(books)

print(f"Saved {len(books)} books to books.csv")
print(books[0])

We ran it on 23 October 2026 with Python 3.12, requests 2.34 and Beautiful Soup 4.15:

Saved 32 books to books.csv
{'title': 'Sharp Objects', 'price_gbp': 47.82, 'rating': 4, 'in_stock': True, 'url': 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html'}

How it works:

• Selectors. Each book sits in an article.product_pod. The full title is in the link's title attribute; the star rating is a CSS class (star-rating Four), so a small dictionary turns it into a number.
• Pagination. li.next a holds the next page's relative URL; urljoin makes it absolute. When there's no next link, the loop ends. The category has 32 books on two pages.
• Politeness. A shared session reuses the connection, the User-Agent says who you are, and time.sleep(1) keeps it to one request per second.
• Clean output. Prices become floats and ratings become integers before they reach the CSV, so the file opens correctly in Excel or pandas.

When the data only appears after JavaScript runs, swap requests for Playwright, which drives a real Chromium. Our Python web scraping guide covers Playwright, Scrapy and the trade-offs.

The hard part of this method comes later: running it every day, rotating proxies when the site starts answering 429, and noticing when a redesign quietly returns empty fields. You can host the script on Scrape.ly to get the scheduling, proxies and storage: start from the Python + Beautiful Soup template and push it with the CLI.

Method 4: Ask an AI agent over MCP

AI assistants can scrape too, if you give them a tool. The Scrape.ly MCP server lets Claude, Cursor, VS Code, ChatGPT and other MCP clients search the Store, read an Actor's input schema, run it and read the results. In Cursor, add it to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "scrapely": {
      "url": "https://api.scrape.ly/mcp",
      "headers": { "Authorization": "Bearer YOUR_TOKEN" }
    }
  }
}

In Claude Code, run claude mcp add --transport http scrapely https://api.scrape.ly/mcp and sign in with /mcp. Then ask in plain words:

Use scrapely to run cornerledger/google-maps-scraper for restaurants in Konstanz, 10 places, and give me a CSV of name, rating and phone.

The agent calls search-actors or goes straight to the named Actor, reads its schema with fetch-actor-details, starts it with call-actor and fetches the rows with get-actor-output. Runs started through MCP are normal runs on your account, charged at the Actor's usual price. For agents that should never overspend, give them a scoped API key with a daily charge limit, and they can pass maxTotalChargeUsd on each call. See Scrape.ly MCP for the client list.

Which method should you use?

• The site is a big platform (Google, Amazon, LinkedIn, Instagram, Zillow): method 1. Platforms change often and block aggressively, and a maintained Actor absorbs both.
• The site is a niche shop, directory or catalogue: method 2 with the Web Scraper, or method 3 if you want the code in your own repo.
• You need a whole site's text for search or an LLM: the Website Content Crawler.
• You're answering a question, not building a pipeline: method 4.
• You want the data in a spreadsheet: any of the four works. Store runs export CSV and Excel directly, and the Python script writes a CSV.

What to watch out for

• Empty results. The usual cause is JavaScript rendering. Check the page source first; switch to useChrome or Playwright if the data isn't there.
• Blocking and 429 errors. Slow down, reduce concurrency, and use proxies. Store Actors and the generic scrapers include them.
• Silent breakage. A redesign can leave your scraper "working" while every price comes back empty. Check field counts on every run.
• Runaway costs. Set the Actor's own limit (such as maxResultsPerCrawl) and a maximum charge for the run. The run stops when it reaches the cap.
• Personal data and terms. Scrape public pages, collect only the fields you need, and don't scrape behind a login you agreed terms for.

FAQ

Can I scrape data from any website?

Technically you can fetch any public page, but some sites block bots heavily, and some data is off-limits because it's behind a login or is personal data you have no basis to collect. Public product, business, job and news data is the common, low-risk case.

How do I scrape a website without coding?

Use a ready-made scraper: find the site in the Scrape.ly Store, fill in the form or pick a preset, click Start and export the results as CSV, Excel or JSON. For sites without a dedicated scraper, the Web Scraper only needs a short extraction function.

Is it legal to scrape data from a website?

Scraping publicly available data is common and lawful in many contexts, but terms you accepted, privacy law and copyright still apply. Our guide is web scraping legal? explains the rules and court cases. This is general information, not legal advice.

How do I scrape a website that uses JavaScript?

Either find the JSON request the page makes in your browser's developer tools and call it directly, or use a real browser: Playwright or Selenium in your own code, or useChrome in the Scrape.ly Web Scraper.

How much does it cost to scrape a website?

Your own script costs only time and hosting. On Scrape.ly you pay per result at the price shown on each Actor page: $0.50 per 1,000 results for the Web Scraper and $3.60 per 1,000 places for the Google Maps Scraper, plus small platform usage. The free plan includes $5 of usage every month.