--- title: "AI Web Scraping: LLMs, Agents and MCP | Scrape.ly" description: "AI web scraping explained: LLM extraction, AI agents and MCP compared with rule-based scraping, plus working code and two real runs with exact costs." url: https://scrape.ly/blog/ai-web-scraping markdown: https://scrape.ly/blog/ai-web-scraping.md --- [AI agents](https://scrape.ly/blog/category/ai-agents.md) # AI Web Scraping: How LLMs and Agents Extract Web Data AI web scraping uses language models to read pages and fill structured fields, and AI agents to decide what to fetch. Learn how LLM extraction, agents and MCP work, when plain rules still win, and run both with working code and real runs. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 23, 2026 - Reading time 8 min read ![Pixel-art robot with an orange >\_ on its terminal head, an AI sparkle in a speech bubble above it, pulls a data card out of a web page and drops it into a crate](https://scrape.ly/media/ai-web-scraping.png?w=1080) AI web scraping is the use of AI models, mostly large language models, to collect data from websites: a model reads a page and fills in the fields you asked for, or an AI agent decides which pages to fetch and which tools to run. It replaces hand-written selectors where layouts vary, and it lets you ask for data in plain language instead of writing code. That doesn't make the classic approach obsolete. The best pipelines mix three techniques: rules where the page is predictable, an LLM where it isn't, and an agent on top that picks the right tool. This guide explains each one, shows when to use which, and gives you working code: a Scrape.ly MCP setup for your agent, structured extraction that needs no LLM at all, and search-to-Markdown for retrieval. Two real runs from October 2026 show the exact output and cost. ## Rule-based scraping vs LLM extraction vs AI agents | | Rule-based scraping | LLM extraction | AI agents | | ----------------------- | ------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------- | | How it finds the data | CSS or XPath selectors, or the page's structured markup | A model reads the page text and fills a schema you define | A model plans steps, picks tools and runs them | | Setup per site | Write and test selectors | Write a schema and an instruction | Describe the goal | | When the layout changes | Breaks until you fix the selectors | Usually keeps working | Keeps working if its tools do | | Cost per page | Compute only | Compute plus model tokens on every page | Model tokens for every step, plus tool costs | | Same output every run | Yes | Mostly; validate it | Varies with the plan the model picks | | Best for | High volume on known sites | Many sites with different layouts | Research, one-off questions, multi-source tasks | The pattern that works: let the agent decide, let tested scrapers do the volume, and spend LLM tokens only on the pages that need them. ## How LLM extraction works LLM extraction has four steps, whatever tool you use: 1. **Fetch** the page, with a browser if its content needs JavaScript. 2. **Clean** it to Markdown or text. Raw HTML is mostly markup the model doesn't need: the books.toscrape.com home page is 51,294 bytes of HTML and 13,476 characters as Markdown in our [Crawl4AI tests](https://scrape.ly/blog/crawl4ai-guide.md), and every character you remove is tokens you don't pay for. 3. **Extract** with a schema and an instruction, for example: ```json { "instruction": "Extract every book on the page. Use null when a value is missing. Do not guess.", "schema": { "type": "object", "properties": { "books": { "type": "array", "items": { "type": "object", "properties": { "title": { "type": "string" }, "price_gbp": { "type": "number" }, "in_stock": { "type": "boolean" } }, "required": ["title", "price_gbp", "in_stock"] } } } } } ``` 4. **Validate** the JSON against the schema and spot-check values against the page. Models can drop rows or invent a plausible value when a field is missing, so "use null, do not guess" belongs in every instruction. The major model APIs accept a JSON schema for structured output, which keeps the response parseable. Libraries such as Crawl4AI wrap the whole loop; its `LLMExtractionStrategy` takes a schema and an instruction and works with any provider LiteLLM supports. On Scrape.ly, the AI step in an automation has an **Extract** mode that pulls structured fields out of a run's results as JSON, for example "From each description, get the company size and the tech stack." ## When you don't need an LLM: structured data already on the page Many pages already publish their data in machine-readable form for search engines: schema.org JSON-LD, Microdata, RDFa and Open Graph tags. Products carry their price and availability, recipes their ingredients, articles their author and date. Reading that markup is faster, cheaper and more exact than asking a model to read the visible text. The [Structured Data Extractor](https://scrape.ly/store/querystone/structured-data-extractor.md) does exactly that. It reads every syntax, resolves `@graph` blocks and `@id` references, and returns one row per page with a normalized main entity: the same flat fields (price, currency, rating, author, dates, address) whatever markup the site used. It doesn't guess values from the visible text, and it flags properties Google's rich results expect but the page is missing. ### A real run: a recipe, a product and a film On 23 October 2026 we ran it from the Store preset [Recipe, product and movie pages, 3 URLs](https://scrape.ly/store/querystone/structured-data-extractor/presets/only-structured.md), unchanged. The run extracted 3 pages in 9.1 seconds: | Page | Main type | Name | Price | Rating | Reviews | Entities | | ----------------------- | --------- | ------------------------------- | ------- | ------ | ------- | -------- | | allrecipes.com recipe | Recipe | Spinach and Feta Turkey Burgers | | 4.6 | 862 | 1 | | apple.com shop page | Product | iPhone 16 | 799 USD | | | 3 | | rottentomatoes.com film | Movie | The Shawshank Redemption | | 89 | 147 | 1 | The recipe's normalized entity went well beyond the table: `prepTimeMinutes` 20, `cookTimeMinutes` 15, `totalTimeMinutes` 35, a yield of 8, "233 kcal", the cuisine, the full ingredient list and every instruction step. The Apple page's validation listed `brand`, `sku` and `offers.availability` as missing recommended properties. The recipe author's name is a member's display name, so we leave it out here. From the run's log: ```plaintext 20261023T03:07:07.029Z [Scrapely] Container started (setup 5.2 s) 20261023T03:07:08.602Z [scrapely.actor] INFO Extracting structured data from 3 page(s) via datacenter proxy. 20261023T03:07:10.570Z [scrapely.actor] INFO 3 page(s) extracted. Downloaded 1.6 MB. 20261023T03:07:11.133Z [Scrapely] Container state: status=exited exitCode=0 oomKilled=false ``` It was charged $0.0036: $0.003 for 3 pages at $1 per 1,000, plus $0.0006 of compute, datacenter proxy and storage. No tokens, and the same result next time. Because the run took 9 seconds, the synchronous endpoint returns the rows directly: ```shell curl -X POST "https://api.scrape.ly/v2/actors/querystone~structured-data-extractor/run-sync-get-dataset-items" -H "Authorization: Bearer $SCRAPELY_TOKEN" -H "Content-Type: application/json" -d '{"startUrls": ["https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/", "https://www.apple.com/shop/buy-iphone/iphone-16", "https://www.rottentomatoes.com/m/shawshank_redemption"], "onlyPagesWithStructuredData": true}' ``` Add `sitemapUrls` or `crawlDepth` to cover a whole site, and `entityTypes: ["Product"]` to keep only product pages. Feed those clean entities to your model instead of raw HTML and the LLM part of your pipeline gets smaller and more accurate. ## AI agents: from "fetch this page" to "find me this data" An AI agent is a model in a loop: it reads your goal, picks a tool, looks at the result and decides the next step. For web data, agents use two kinds of tools: - **Browser tools** let the agent click, type and read pages one step at a time, as with Playwright MCP. Good for exploring a site or acting inside a session; slow and token-hungry for volume, because every page passes through the model. - **Data tools** run a complete scraper in the cloud and hand back structured results. The agent makes one call and reads only the rows it needs. Agents get those tools through the Model Context Protocol (MCP), the open standard for connecting models to tools. One MCP server can give an agent search, crawling and dozens of site-specific scrapers at once. Our roundup of the [best MCP servers for web scraping](https://scrape.ly/blog/best-mcp-servers-for-web-scraping.md) compares the main options. ## Connect an agent to Scrape.ly over MCP The [Scrape.ly MCP server](https://scrape.ly/mcp.md) is hosted at `https://api.scrape.ly/mcp` and signs in with OAuth. In Claude Code: ```shell claude mcp add --transport http scrapely https://api.scrape.ly/mcp ``` Then run `/mcp` and choose `scrapely` to sign in. In Cursor, add this to `~/.cursor/mcp.json` (drop `headers` to sign in with OAuth instead of a token): ```json { "mcpServers": { "scrapely": { "url": "https://api.scrape.ly/mcp", "headers": { "Authorization": "Bearer YOUR_TOKEN" } } } } ``` The agent works through four tools: `search-actors` finds a scraper in the [Store](https://scrape.ly/store.md), `fetch-actor-details` reads its input schema and price, `call-actor` runs it (waiting up to 45 seconds), and `get-actor-output` pages through the results with field selection, so a big dataset never lands in the context in one go. Setup for ChatGPT, VS Code, Windsurf and others is on the [agents page](https://scrape.ly/agents.md). ### A real agent call: search results as Markdown On 22 October 2026 we called the [RAG Web Browser](https://scrape.ly/store/querystone/rag-web-browser.md) through the MCP server with this `call-actor` request: ```json { "actor": "querystone/rag-web-browser", "input": { "query": "web scraping with python", "maxResults": 3, "scrapingTool": "raw-http", "outputFormats": ["markdown"] }, "waitSecs": 45, "callOptions": { "maxTotalChargeUsd": 0.1 } } ``` It finished inside the wait: `SUCCEEDED` after 7.8 seconds with 3 items. `get-actor-output` then returned the top three Google results as Markdown: a University of Texas library guide (8,470 characters), the Scrapy home page (9,343) and a Medium tutorial (12,761), each with title, canonical URL and language. It cost $0.0039: $0.0015 for one Google results page, $0.0010 of compute and $0.0013 of residential proxy. The `maxTotalChargeUsd` cap means a confused agent can't spend more than you allow. ## AI web scraping in Python: search, read, hand to your model The same RAG Web Browser call from Python gives a retrieval step for your own LLM app. This uses the Scrape.ly Python client (`pip install scrapely-client-python`): ```python import os from scrapely_client import ScrapelyClient client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly") run = client.actor("querystone/rag-web-browser").call( run_input={"query": "web scraping with python", "maxResults": 3, "outputFormats": ["markdown"]} ) context = [] for page in client.dataset(run.default_dataset_id).iterate_items(): context.append(f"Source: {page['metadata']['url']}\n\n{page['markdown']}") prompt = "Answer using only these sources and cite them.\n\n" + "\n\n---\n\n".join(context) print(len(prompt), "characters of grounded context") ``` Send `prompt` to the model you already use. Swap the query for a URL and the Actor reads that one page instead. For whole sites, the Website Content Crawler does the same at $0.50 per 1,000 pages. ## How Scrape.ly does AI web scraping Scrape.ly is built for all three techniques at once: - **568 ready-made Actors** in the Store do the rule-based volume work for sites like Google Maps, Amazon, LinkedIn Jobs and TikTok, each with fixed output fields and a price per result on its page. - **The MCP server** lets any agent find, run and read those Actors with one sign-in, with scoped API keys and daily charge limits for agents you don't watch. - **The **[**AI chat**](https://scrape.ly/docs/ai-chat.md) in the dashboard does it for you: describe the data, and it picks an Actor, fills its input, runs it and summarizes the results. It treats web pages, dataset rows and READMEs as data, never as instructions, and asks for approval before a run that only third-party content suggested. - **Automations with an AI step** run on a schedule and summarize, filter, compare or extract before the results reach Slack, email or a sheet. - **A free plan** with $5 of usage every month and no card, enough for thousands of pages. ## What to watch out for - **Made-up values.** Ask for null when a value is missing, validate against the schema, and prefer structured markup where the page has it. - **Token costs at volume.** An LLM call per page adds up. Clean pages to Markdown first, and use rules or a ready-made scraper for the high-volume part. - **Prompt injection.** Pages can contain text written to steer a model. Treat page content as data, keep agents on scoped keys, and cap every run. - **Blocking.** AI doesn't get you past rate limits or bot checks. Scrape.ly Actors include proxies, and the Scrapely browser handles sites that block headless Chrome. - **Personal data and terms.** Collect public data you need, respect each site's terms, and treat personal data under the GDPR and CCPA. ## FAQ ### What is AI web scraping? It's collecting web data with the help of AI models: an LLM extracts fields from page text into a schema, or an AI agent decides what to fetch and which scraping tools to run. It's most useful when layouts vary or when you'd rather describe the data than write selectors. ### Is AI scraping better than traditional scraping? For varied layouts and one-off questions, yes. For high volume on known sites, rule-based scrapers are cheaper, faster and return the same output every time. The strongest setup combines both: an agent that calls tested scrapers. ### Can ChatGPT or Claude scrape websites? They can search the web, read pages you link and write scraper code, and with a scraping MCP server connected they can run real scrapers and return the data. Our guide to [web scraping with ChatGPT](https://scrape.ly/blog/web-scraping-with-chatgpt.md) shows the setup and an example session. ### How much does AI web scraping cost? Fractions of a cent per page on Scrape.ly. Our structured-data run cost $0.0036 for 3 pages with no model tokens, and the agent's search-and-read call cost $0.0039 for 3 pages of Markdown. LLM extraction adds your model's token price for every page you send it. ### Is AI web scraping legal? The same rules apply as to any scraping: public data, the site's terms, copyright and privacy law. Using a model to read the page doesn't change what you're allowed to collect. ### Which tool should I use for AI web scraping? Scrape.ly. One MCP sign-in gives your agent 568 ready-made scrapers, structured extraction and search-to-Markdown, with a dollar cap on every run and $5 of free usage every month. Scrape.ly Store Extract structured data without an LLM Read JSON-LD, Microdata, RDFa and Open Graph from any page, sitemap or crawl into clean fields: $1 per 1,000 pages, callable from any MCP client. [Try it free](https://scrape.ly/store/querystone/structured-data-extractor.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in AI agents - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a browser window showing the Playwright logo into a server rack labelled MCP](https://scrape.ly/media/playwright-mcp.png?w=750)](https://scrape.ly/blog/playwright-mcp.md) ### [Playwright MCP: Setup for Cursor, Claude Code and VS Code, and When to Use It for Scraping](https://scrape.ly/blog/playwright-mcp.md) [Playwright MCP lets an AI agent drive a real browser through accessibility snapshots. Set it up in Cursor, Claude Code or VS Code in a minute, learn the options and tools that matter, and see where it shines for scraping and where to hand off.](https://scrape.ly/blog/playwright-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head, an AI sparkle in a speech bubble above it, reaches out to a standing text file with a folded corner whose dark plate reads LLMS, next to a web page](https://scrape.ly/media/what-is-llms-txt.png?w=750)](https://scrape.ly/blog/what-is-llms-txt.md) ### [What Is llms.txt? Format, Real Examples and How to Create One](https://scrape.ly/blog/what-is-llms-txt.md) [llms.txt is a Markdown file at the root of a website that gives AI agents a short, curated map of its content. The format rules from the v2 proposal, real files from Anthropic, OpenAI, Stripe and Cloudflare, llms-full.txt, Markdown twins, and how to create one in 11 seconds.](https://scrape.ly/blog/what-is-llms-txt.md) - [![Pixel-art robot with an orange >\_ on its terminal head pulls a page of text out of a monitor showing the cyan Crawl4AI logo above lines of code, and drops it into a crate](https://scrape.ly/media/crawl4ai-guide.png?w=750)](https://scrape.ly/blog/crawl4ai-guide.md) ### [Crawl4AI Tutorial: Setup, Examples and a Hosted Alternative](https://scrape.ly/blog/crawl4ai-guide.md) [A tested Crawl4AI tutorial for v0.9.5: install it, crawl a page to Markdown, extract JSON with CSS selectors or an LLM, deep crawl a site, run it in Docker or over MCP, and see when a hosted crawler saves you the infrastructure.](https://scrape.ly/blog/crawl4ai-guide.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. The advantage lasts forever. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:support@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.