--- title: "How to Scrape Data From a Website (4 Ways) | Scrape.ly" description: "How to scrape data from a website: 4 methods from no-code to Python, with real runs, exact costs, tested code and an AI-agent option." url: https://scrape.ly/blog/how-to-scrape-a-website markdown: https://scrape.ly/blog/how-to-scrape-a-website.md --- [Guides](https://scrape.ly/blog/category/guides.md) # How to Scrape Data From a Website: 4 Methods, No-Code to Python How to scrape data from a website four ways: a ready-made scraper with no code, a generic cloud scraper with one extraction function, a Python script you run yourself, and an AI agent over MCP. Each with real runs, exact costs and tested code. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 9 min read ![Pixel-art robot with an orange >\_ on its terminal head pulls a data card out of a browser window showing a web page and drops it into a crate of collected data](https://scrape.ly/media/how-to-scrape-a-website.png?w=1080) How to scrape data from a website comes down to one decision: build the scraper yourself, or use one that already exists. If a ready-made scraper covers the site, you can have a spreadsheet in under a minute without code; if not, a generic cloud scraper, a short Python script or an AI agent will get the data. This guide walks through all four methods, from no-code to Python, on real targets. Every run below happened on 22 or 23 October 2026, and every cost is what we were actually charged. ## How to scrape data from a website: 4 methods at a glance | | 1. Ready-made scraper | 2) Generic cloud scraper | 3. Python script | 4) AI agent over MCP | | ---------------------------------- | ---------------------------------------------------- | ------------------------------------------- | ------------------------------------- | ------------------------------------------ | | You need | A browser | A few CSS selectors | Python and some HTML | An AI assistant such as Claude or Cursor | | Your work | Fill in a form | Write one extraction function | Write, run and host the whole scraper | Describe the data in a sentence | | Proxies, retries, hosting | Included | Included | Yours to build | Included | | Who fixes it when the site changes | The scraper's maintainer | You (one function) | You | The maintainer of the Actor the agent uses | | Best for | Big platforms: Google Maps, Amazon, LinkedIn, Reddit | Any public site without a dedicated scraper | Learning, one-off jobs, full control | Ad hoc questions and agent workflows | | Example cost | 10 places for $0.037 | 4 pages for $0.0029 | Free, plus your time and servers | Same as the Actor it runs | ## Before you start: three quick checks 1. **Is the data public?** Stick to what a logged-out visitor can see. Data behind a login comes with terms you accepted, and personal data comes with privacy law. Our [web scraping guide](https://scrape.ly/blog/what-is-web-scraping.md) covers the basics. 2. **Does a scraper already exist?** Search the [Scrape.ly Store](https://scrape.ly/store.md) for the site's name. If there's a match, method 1 is the fastest route by far. 3. **Is the data in the HTML?** Open the page, right-click, choose **View page source** and search for a value you want, such as a price. If it's there, plain HTTP scraping works. If it isn't, the page builds its content with JavaScript and you need a browser or the page's own data request. ## Method 1: Use a ready-made scraper (no code) The [Scrape.ly Store](https://scrape.ly/store.md) has 568 Actors, each a maintained scraper for one site or job. You fill in a form, start the run and download the results. Nothing to install, and when the site changes its layout, the Actor's maintainer updates it. Here's the whole process with the [Google Maps Scraper](https://scrape.ly/store/cornerledger/google-maps-scraper.md): 1. Open the Actor page and pick the preset [Restaurants in Konstanz, 10 places](https://scrape.ly/store/cornerledger/google-maps-scraper/presets/search-2.md). It fills the form with this input: ```json { "searchStringsArray": ["restaurant"], "locationQuery": "Konstanz", "maxCrawledPlacesPerSearch": 10, "language": "en" } ``` 2. Click **Start**. Our run on 22 October 2026 took 5.7 seconds and saved 10 places: ```plaintext 20261022T15:22:55.025Z [scrapely.actor] INFO Saved 10 places in 3.2s (11 requests, 0 hedged, 0.93 MB); startup: python 1.13s, SDK import 0.00s, SDK init 0.27s ``` 3. Open the results and export them as CSV, Excel or JSON. The first three rows: | Name | Category | Rating | Reviews | Street | | -------------------- | ------------------ | ------ | ------- | -------------------- | | The Dom | Italian restaurant | 4.7 | 1,136 | Brückengasse 1 | | Constanzer Wirtshaus | German restaurant | 4.4 | 7,521 | Spanierstraße 3 | | Chu Minh Restaurant | Asian restaurant | 4.7 | 249 | Frieda-Sigrist-Weg 1 | Each record also carries the phone number, website, opening hours, coordinates and more. The run cost $0.036888: 10 places × $0.0036 = $0.036 in place fees, plus $0.0009 of compute and proxy. The free plan's $5 monthly credit covers about 1,350 places at that rate. To change the job, edit the search term, location and limit, and click **Start** again. To repeat it every morning, add a schedule under **Automations**. ## Method 2: Use a generic cloud scraper (one function) When no Actor exists for your site, a generic scraper does the fetching, link following, proxies and retries, and you supply only the extraction logic. Scrape.ly has two: - [**Web Scraper**](https://scrape.ly/store/querystone/web-scraper.md) runs your JavaScript page function on each page and saves whatever object it returns. Best for structured fields: prices, titles, specs. $0.50 per 1,000 results. - [**Website Content Crawler**](https://scrape.ly/store/querystone/website-content-crawler.md) crawls a site from a start URL or its sitemap and saves each page's clean text and Markdown. Best for documentation, knowledge bases and LLM pipelines. $0.50 per 1,000 pages. ### Web Scraper: four book pages in 12 seconds We pointed the Web Scraper at the Mystery category of [books.toscrape.com](https://books.toscrape.com/), a sandbox shop built for scraping practice, and told it to follow each book link and pull four fields from the book page: ```json { "startUrls": [{ "url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" }], "linkSelector": "article.product_pod h3 a", "globs": [{ "glob": "https://books.toscrape.com/catalogue/*/index.html" }], "pageFunction": "async function pageFunction(context) {\n const $ = context.jQuery;\n if (!$('div.product_main').length) return null;\n return {\n url: context.request.url,\n title: $('div.product_main h1').text().trim(),\n price: $('div.product_main p.price_color').text().trim(),\n inStock: $('div.product_main p.availability').text().includes('In stock'),\n upc: $('table.table-striped td').first().text().trim()\n };\n}", "maxPagesPerCrawl": 6, "maxResultsPerCrawl": 5, "maxConcurrency": 2, "respectRobotsTxtFile": true, "proxyConfiguration": { "useScrapelyProxy": true } } ``` The fields that matter: - `linkSelector` and `globs` decide which links get followed: here only the book links on the listing page. - `pageFunction` gets jQuery on each page. On the listing page it returns `null`, so the page is only used to find links; on a book page it returns the record. - `maxResultsPerCrawl` stops the run after five records; `maxConcurrency` keeps the load on the site low. - `respectRobotsTxtFile` skips links the site's robots.txt disallows. The run on 23 October 2026 fetched 5 pages and saved 4 book records in 12 seconds: ```plaintext 20261023T03:02:50.015Z [scrapely.proxy_configuration] INFO Platform proxy access check succeeded. 20261023T03:02:58.070Z [scrapely.actor] INFO Done: 5 pages fetched, 4 records saved in 8 s ``` | Title | Price | In stock | UPC | | -------------------- | ------ | -------- | ---------------- | | Sharp Objects | £47.82 | Yes | e00eb4fd7b871a48 | | In a Dark, Dark Wood | £19.63 | Yes | 19ed25f4641d5efd | | The Past Never Ends | £56.50 | Yes | 5ee94540d0749ea0 | | A Murder in Time | £16.64 | Yes | f733e8c19d40ec2e | It was charged $0.002919: 4 results × $0.0005 = $0.002, plus $0.0009 of compute, proxy and storage. The listing page's `null` record wasn't charged. For pages that only show data after JavaScript runs, set `"useChrome": true` and the same page function runs on the rendered page. ### Website Content Crawler: a whole site as text If you want every page's text rather than specific fields, the crawler is simpler: give it a start URL and a page cap. On 22 October 2026 it read crawlee.dev's sitemaps (4,860 pages listed) and saved the first 5 pages as clean text and Markdown in 6.2 seconds, for $0.0031. Runs that short can use the synchronous endpoint, which returns the items in the same response: ```shell curl -X POST "https://api.scrape.ly/v2/actors/querystone~website-content-crawler/run-sync-get-dataset-items" -H "Authorization: Bearer $SCRAPELY_TOKEN" -H "Content-Type: application/json" -d '{"startUrls": [{"url": "https://crawlee.dev/"}], "useSitemaps": true, "maxCrawlPages": 5, "maxResults": 5}' ``` Create the token under **Settings → API & Integrations** and export it as `SCRAPELY_TOKEN`. Add `"includeUrlGlobs"` to keep the crawl inside one section, such as `/docs/`. ## Method 3: Write a Python scraper Code gives you full control and costs nothing but time. This script scrapes every book in the same Mystery category, pages through the results by following the "next" link, and writes a CSV. Install the libraries with `pip install requests beautifulsoup4`: ```python import csv import time from urllib.parse import urljoin import requests from bs4 import BeautifulSoup START = "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html" RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5} session = requests.Session() session.headers["User-Agent"] = "books-scraper/1.0 (+mailto:you@example.com)" def scrape_listing(url): """Return the books on one listing page and the URL of the next page (or None).""" soup = BeautifulSoup(session.get(url, timeout=15).content, "html.parser") rows = [] for book in soup.select("article.product_pod"): link = book.select_one("h3 a") rows.append({ "title": link["title"], "price_gbp": float(book.select_one("p.price_color").get_text(strip=True).lstrip("£")), "rating": RATINGS[book.select_one("p.star-rating")["class"][1]], "in_stock": "In stock" in book.select_one("p.availability").get_text(), "url": urljoin(url, link["href"]), }) next_link = soup.select_one("li.next a") return rows, urljoin(url, next_link["href"]) if next_link else None books, url = [], START while url: rows, url = scrape_listing(url) books.extend(rows) time.sleep(1) # be polite: one page per second with open("books.csv", "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=books[0].keys()) writer.writeheader() writer.writerows(books) print(f"Saved {len(books)} books to books.csv") print(books[0]) ``` We ran it on 23 October 2026 with Python 3.12, requests 2.34 and Beautiful Soup 4.15: ```plaintext Saved 32 books to books.csv {'title': 'Sharp Objects', 'price_gbp': 47.82, 'rating': 4, 'in_stock': True, 'url': 'https://books.toscrape.com/catalogue/sharp-objects_997/index.html'} ``` How it works: - **Selectors.** Each book sits in an `article.product_pod`. The full title is in the link's `title` attribute; the star rating is a CSS class (`star-rating Four`), so a small dictionary turns it into a number. - **Pagination.** `li.next a` holds the next page's relative URL; `urljoin` makes it absolute. When there's no next link, the loop ends. The category has 32 books on two pages. - **Politeness.** A shared session reuses the connection, the User-Agent says who you are, and `time.sleep(1)` keeps it to one request per second. - **Clean output.** Prices become floats and ratings become integers before they reach the CSV, so the file opens correctly in Excel or pandas. When the data only appears after JavaScript runs, swap `requests` for Playwright, which drives a real Chromium. Our [Python web scraping guide](https://scrape.ly/blog/web-scraping-python.md) covers Playwright, Scrapy and the trade-offs. The hard part of this method comes later: running it every day, rotating proxies when the site starts answering 429, and noticing when a redesign quietly returns empty fields. You can host the script on Scrape.ly to get the scheduling, proxies and storage: start from the [Python + Beautiful Soup template](https://scrape.ly/templates/python-beautifulsoup.md) and push it with the CLI. ## Method 4: Ask an AI agent over MCP AI assistants can scrape too, if you give them a tool. The Scrape.ly MCP server lets Claude, Cursor, VS Code, ChatGPT and other MCP clients search the Store, read an Actor's input schema, run it and read the results. In Cursor, add it to `~/.cursor/mcp.json`: ```json { "mcpServers": { "scrapely": { "url": "https://api.scrape.ly/mcp", "headers": { "Authorization": "Bearer YOUR_TOKEN" } } } } ``` In Claude Code, run `claude mcp add --transport http scrapely https://api.scrape.ly/mcp` and sign in with `/mcp`. Then ask in plain words: > Use scrapely to run cornerledger/google-maps-scraper for restaurants in Konstanz, 10 places, and give me a CSV of name, rating and phone. The agent calls `search-actors` or goes straight to the named Actor, reads its schema with `fetch-actor-details`, starts it with `call-actor` and fetches the rows with `get-actor-output`. Runs started through MCP are normal runs on your account, charged at the Actor's usual price. For agents that should never overspend, give them a scoped API key with a daily charge limit, and they can pass `maxTotalChargeUsd` on each call. See [Scrape.ly MCP](https://scrape.ly/mcp.md) for the client list. ## Which method should you use? - **The site is a big platform** (Google, Amazon, LinkedIn, Instagram, Zillow): method 1. Platforms change often and block aggressively, and a maintained Actor absorbs both. - **The site is a niche shop, directory or catalogue**: method 2 with the Web Scraper, or method 3 if you want the code in your own repo. - **You need a whole site's text** for search or an LLM: the Website Content Crawler. - **You're answering a question, not building a pipeline**: method 4. - **You want the data in a spreadsheet**: any of the four works. Store runs export CSV and Excel directly, and the Python script writes a CSV. ## What to watch out for - **Empty results.** The usual cause is JavaScript rendering. Check the page source first; switch to `useChrome` or Playwright if the data isn't there. - **Blocking and 429 errors.** Slow down, reduce concurrency, and use proxies. Store Actors and the generic scrapers include them. - **Silent breakage.** A redesign can leave your scraper "working" while every price comes back empty. Check field counts on every run. - **Runaway costs.** Set the Actor's own limit (such as `maxResultsPerCrawl`) and a maximum charge for the run. The run stops when it reaches the cap. - **Personal data and terms.** Scrape public pages, collect only the fields you need, and don't scrape behind a login you agreed terms for. ## FAQ ### Can I scrape data from any website? Technically you can fetch any public page, but some sites block bots heavily, and some data is off-limits because it's behind a login or is personal data you have no basis to collect. Public product, business, job and news data is the common, low-risk case. ### How do I scrape a website without coding? Use a ready-made scraper: find the site in the Scrape.ly Store, fill in the form or pick a preset, click Start and export the results as CSV, Excel or JSON. For sites without a dedicated scraper, the Web Scraper only needs a short extraction function. ### Is it legal to scrape data from a website? Scraping publicly available data is common and lawful in many contexts, but terms you accepted, privacy law and copyright still apply. Our guide [is web scraping legal?](https://scrape.ly/blog/is-web-scraping-legal.md) explains the rules and court cases. This is general information, not legal advice. ### How do I scrape a website that uses JavaScript? Either find the JSON request the page makes in your browser's developer tools and call it directly, or use a real browser: Playwright or Selenium in your own code, or `useChrome` in the Scrape.ly Web Scraper. ### How much does it cost to scrape a website? Your own script costs only time and hosting. On Scrape.ly you pay per result at the price shown on each Actor page: $0.50 per 1,000 results for the Web Scraper and $3.60 per 1,000 places for the Google Maps Scraper, plus small platform usage. The free plan includes $5 of usage every month. Scrape.ly Store Web Scraper Scrape any public site with one page function. Link following, proxies and retries included. $0.50 per 1,000 results. [Try it free](https://scrape.ly/store/querystone/web-scraper.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in Guides - [![Pixel-art robot with an orange >\_ on its terminal head walks past a raised red-and-white boom barrier, carrying a data card from the web page behind it towards a crate](https://scrape.ly/media/web-scraping-without-getting-blocked.png?w=750)](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) ### [Web Scraping Without Getting Blocked: 10 Techniques That Work](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) [Web scraping without getting blocked comes down to ten habits: real headers, pacing, sessions, good IPs, real browsers, clean fingerprints, challenge handling, honeypot checks, robots.txt and caching. Tested Python for each, and how Scrape.ly handles them for you.](https://scrape.ly/blog/web-scraping-without-getting-blocked.md) - [![Pixel-art robot with an orange >\_ on its terminal head reaches for the last empty box on a clipboard checklist whose first four items carry orange ticks, with a web page standing behind it](https://scrape.ly/media/web-scraping-best-practices.png?w=750)](https://scrape.ly/blog/web-scraping-best-practices.md) ### [12 Web Scraping Best Practices for Reliable, Responsible Scrapers](https://scrape.ly/blog/web-scraping-best-practices.md) [Twelve web scraping best practices that keep scrapers polite, unblocked and accurate: robots.txt, rate limits, caching, retries, validation, monitoring, personal data and more. Each with tested Python code and how to do it on Scrape.ly.](https://scrape.ly/blog/web-scraping-best-practices.md) - [![Pixel-art scene in two halves: on the left a small robot walks along a cable linking web pages, on the right a robot with an orange >\_ on its terminal head pulls a data card out of a single web page](https://scrape.ly/media/web-crawling-vs-web-scraping.png?w=750)](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) ### [Web Crawling vs Web Scraping: The Difference, With Code and Real Runs](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) [Web crawling finds pages; web scraping pulls data out of them. See the difference in one Python script, in a real crawl of crawlee.dev that cost $0.0031, and learn which one your project needs.](https://scrape.ly/blog/web-crawling-vs-web-scraping.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.