ScrapeGraphAI Tutorial: How It Works, a Tested Setup and a Hosted Alternative

ScrapeGraphAI is an open-source Python library that extracts data from web pages by sending their content to a large language model with your prompt. You give ScrapeGraphAI a URL and a sentence such as "list every book with its price", and a small graph of steps fetches the page, cleans it, and asks the model for JSON.

This tutorial covers what you need to judge it for real work: installation, the SmartScraperGraph pipeline, schemas, local and hosted models, the hosted ScrapeGraphAI API and its credit prices, and where an LLM per page stops making sense. We installed version 2.3.1 on 26 October 2026 and ran it against the practice site books.toscrape.com; we show exactly what happened, including the step that needs a model. The last part covers the Scrape.ly route, which returns structured data from the same kinds of pages with no LLM tokens at all.

ScrapeGraphAI at a glance

How to install ScrapeGraphAI

The README installs the package and then the browser it uses to fetch pages:

pip install scrapegraphai
playwright install

ScrapeGraphAI is built on LangChain, so the install is large. In a clean environment, uv resolved 98 packages for version 2.3.1, including the LangChain integrations for OpenAI, AWS, Mistral and Ollama, Playwright and undetected-playwright. The first import of SmartScraperGraph took 21.6 seconds on our machine. To try it without touching your global Python, this is the command we used:

SCRAPEGRAPHAI_TELEMETRY_ENABLED=false uv run --python 3.12 --with scrapegraphai==2.3.1 python first_graph.py

Your first graph: SmartScraperGraph

SmartScraperGraph takes a prompt, a source URL and a config whose llm block picks the model. This is the README's shape, pointed at books.toscrape.com and a local Llama model through Ollama:

import json
from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {"model": "ollama/llama3.2", "model_tokens": 8192, "format": "json"},
    "headless": True,
    "verbose": True,
}

graph = SmartScraperGraph(
    prompt="List every book on the page with its title and price",
    source="https://books.toscrape.com/",
    config=graph_config,
)
print(json.dumps(graph.run(), indent=2))

With verbose on, the run prints each node as it executes. Here is what our machine printed, without an Ollama server running:

--- Executing Fetch Node ---
--- (Fetching HTML from: https://books.toscrape.com/) ---
--- Executing ParseNode Node ---
--- Executing GenerateAnswer Node ---
Error during chain execution: [Errno 61] Connection refused

That trace is the whole design in four lines. The Fetch node loaded the page in a browser, the Parse node turned the HTML into text chunks, and the GenerateAnswer node tried to send those chunks to the model, which wasn't there. Every extraction in ScrapeGraphAI ends in that LLM call. There is no selector-based path to fall back on, so you need either a local model (install Ollama and ollama pull llama3.2) or an API key before the library returns anything.

To use a hosted model instead, change only the llm block, as the README shows:

graph_config = {
    "llm": {"api_key": "YOUR_OPENAI_API_KEY", "model": "openai/gpt-4o-mini"},
    "verbose": True,
    "headless": False,
}

We didn't run the OpenAI variant: it needs a paid key, and the output would depend on the model rather than on the library.

Get typed output with a schema

Free-form JSON from an LLM changes shape between runs: one call says price, the next says book_price. In 2.3.1, SmartScraperGraph takes an optional schema argument (a Pydantic model class), which tells the model the exact shape to return:

from typing import List
from pydantic import BaseModel
from scrapegraphai.graphs import SmartScraperGraph

class Book(BaseModel):
    title: str
    price: str

class Books(BaseModel):
    books: List[Book]

graph = SmartScraperGraph(
    prompt="List every book on the page with its title and price",
    source="https://books.toscrape.com/",
    config=graph_config,
    schema=Books,
)

A schema narrows the output; it doesn't make it correct. The model can still skip a row or misread a price, so validate the result before you store it, and spot-check it against the page.

The other ScrapeGraphAI pipelines

Version 2.3.1 exports 24 graph classes. The README highlights six:

The rest cover local files (CSVScraperGraph, JSONScraperGraph, XMLScraperGraph, DocumentScraperGraph), screenshots, depth crawling (DepthSearchGraph) and "lite" variants. ScriptCreatorGraph is the most interesting for production: it uses the LLM once to write a scraper, and the script then runs without a model. That's the same idea behind every approach below that doesn't pay per page.

The hosted ScrapeGraphAI API

ScrapeGraphAI also sells a managed API, used through the scrapegraph-py and scrapegraph-js SDKs. The README describes the split: with the open-source library you bring your own LLM and handle browsers, proxies and scaling; with the API those are managed and you pay in credits. The SDK call is short:

from scrapegraph_py import ScrapeGraphAI

sgai = ScrapeGraphAI()  # reads SGAI_API_KEY
res = sgai.extract("Extract product names and prices", url="https://example.com")
print(res.data.json_data if res.status == "success" else res.error)

The pricing page (checked 26 October 2026) lists a free plan with 500 credits, one time, and paid plans at $20 a month for 10,000 credits, $100 for 100,000 and $500 for 750,000. An Extract call costs 5 credits, and turning on stealth (anti-bot bypass) adds 5 more (Extract docs). On the $20 plan, a credit costs $0.002, so:

• 1,000 pages through Extract = 5,000 credits = $10.
• 1,000 pages through Extract with stealth = 10,000 credits = $20, the whole monthly allowance.

What to watch out for with an LLM on every page

ScrapeGraphAI is quick to prototype with, because you write a sentence instead of selectors. The bill for that convenience arrives in production:

• Tokens on every page. The open-source library's cost model, in its own README, is "LLM tokens + your own infra". Every page you extract is another model call, forever, even when the layout never changes.
• Output that drifts. Two runs on the same page can return different field names, a missing row or a rounded price. A schema and validation reduce this; they don't remove it.
• Slow by design. A model call per page is slower than parsing HTML, and rate limits on your LLM provider become rate limits on your scraper.
• Browsers, proxies and blocking. In the open-source library, the README lists proxies, anti-bot handling, scaling and maintenance as your responsibility.
• Telemetry. The library collects anonymous usage metrics unless you set SCRAPEGRAPHAI_TELEMETRY_ENABLED=false.
• Rules and privacy. Scrape public pages, respect rate limits and site terms, and treat personal data under the GDPR and CCPA.

Most pages you'll want to scrape at scale don't need a model to understand them. Product, recipe, event and article pages usually publish their data as schema.org markup for search engines, and the rest have a stable layout that a page function reads for a fraction of a cent.

The Scrape.ly route: structured data with no LLM tokens

Scrape.ly runs ready-made scrapers, called Actors, in the cloud. You call them from the console, the REST API, Python or Node.js clients, or an AI agent over MCP, and every Actor has a fixed price per result. Three of them replace a ScrapeGraphAI pipeline, depending on the page.

1. Pages that publish structured data. The Structured Data Extractor reads JSON-LD, Microdata, RDFa, Open Graph and meta tags, and normalises the page's main entity into flat columns: price, currency, availability, rating, author, dates, address. No model is involved, so the same page returns the same row every time. On 23 October 2026 we ran its preset "Recipe, product and movie pages, 3 URLs":

It finished in 9.1 seconds. At today's list price, three pages cost 3 × $0.00111 = $0.0033, platform usage included: $1.11 per 1,000 pages, against $10 for 1,000 Extract calls on the hosted ScrapeGraphAI plan above.

2. Pages with a stable layout. The Web Scraper runs your own JavaScript page function on each page and follows the links you choose. This is the input we ran on 23 October 2026 against books.toscrape.com's mystery category, the same site as the ScrapeGraphAI example:

{
  "startUrls": [{"url": "https://books.toscrape.com/catalogue/category/books/mystery_3/index.html"}],
  "linkSelector": "article.product_pod h3 a",
  "globs": [{"glob": "https://books.toscrape.com/catalogue/*/index.html"}],
  "pageFunction": "async function pageFunction(context) {\n    const $ = context.jQuery;\n    if (!$('div.product_main').length) return null; // listing page: only follow links\n    return {\n        url: context.request.url,\n        title: $('div.product_main h1').text().trim(),\n        price: $('div.product_main p.price_color').text().trim(),\n        inStock: $('div.product_main p.availability').text().includes('In stock'),\n        upc: $('table.table-striped td').first().text().trim(),\n    };\n}",
  "maxPagesPerCrawl": 6,
  "maxResultsPerCrawl": 5,
  "maxConcurrency": 2,
  "respectRobotsTxtFile": true,
  "proxyConfiguration": {"useScrapelyProxy": true}
}

It returned four books in 12.4 seconds, among them "Sharp Objects" at £47.82 and "In a Dark, Dark Wood" at £19.63, each with its stock flag and UPC. The Web Scraper costs $0.715 per 1,000 results, platform usage included, and the selectors give the same answer on every run. If you'd rather not write selectors, ask an LLM to write the page function once, the way ScriptCreatorGraph does, and then run it at scale without a model.

3. Text for your own LLM. When you do want a model to read the page (summaries, Q&A, RAG), the Website Content Crawler turns a site into clean Markdown for $0.50 per 1,000 pages plus platform usage, so you pay tokens only for the clean text. Our guide on turning a website into Markdown for RAG has a real 10-page crawl and the chunking code.

Large sites that block generic scrapers (Amazon, Google Maps, LinkedIn jobs, Zillow) have dedicated Actors in the Store, with the fields already mapped.

Call it from Python or an agent

Create an API token under Settings → API & Integrations and export it as SCRAPELY_TOKEN. The Python client (pip install scrapely-client-python) waits for the run and reads the rows:

import os
from scrapely_client import ScrapelyClient

client = ScrapelyClient(os.environ["SCRAPELY_TOKEN"], api_url="https://api.scrape.ly")

run = client.actor("querystone/structured-data-extractor").call(
    run_input={
        "startUrls": ["https://www.apple.com/shop/buy-iphone/iphone-16"],
        "onlyPagesWithStructuredData": True,
    }
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["primaryType"], row["primaryName"], row.get("price"), row.get("priceCurrency"))

The prompt-style experience that makes ScrapeGraphAI appealing lives in your agent instead. Add the hosted Scrape.ly MCP server to Claude Code with claude mcp add --transport http scrapely https://mcp.scrape.ly, sign in with /mcp, and ask: "Use scrapely to get the price and rating from these three product pages with querystone/structured-data-extractor." The agent picks the Actor, runs it and reads the result; the extraction itself costs no model tokens.

ScrapeGraphAI vs Scrape.ly

ScrapeGraphAI is a good way to see what an LLM can pull out of a page. For a pipeline you run every day, Scrape.ly is the better choice: deterministic output, a fixed price per row with no token bill, and the scheduling and delivery already built. Our overview of AI web scraping compares rule-based, LLM and agent extraction, and the Crawl4AI tutorial covers the other popular open-source option.

FAQ

Is ScrapeGraphAI free?

The library is open source under the MIT license, so the code is free. Every extraction calls an LLM, so you pay for model tokens (or run a local model through Ollama) plus your own servers. The hosted ScrapeGraphAI API has a one-time free allowance of 500 credits and paid plans from $20 a month.

Does ScrapeGraphAI work without an OpenAI key?

Yes, with a local model. Point the llm config at an Ollama model such as ollama/llama3.2 and run an Ollama server. Without any model, the graph stops at the GenerateAnswer node, as our test showed.

Which Python version does ScrapeGraphAI need?

Version 2.3.1 requires Python 3.12 or newer. The scrapegraph-py SDK for the hosted API (2.1.0 and later) also needs 3.12.

What is the difference between scrapegraphai and scrapegraph-py?

scrapegraphai is the open-source library that runs graphs on your machine with your LLM. scrapegraph-py is the SDK for the paid hosted API, where ScrapeGraphAI runs the browser, proxies and model for you and bills in credits.

What is a good ScrapeGraphAI alternative for production?

Scrape.ly. Pages with schema.org markup go through the Structured Data Extractor ($1.11 per 1,000 pages), pages with a stable layout through the Web Scraper with your page function, and big sites through dedicated Actors, all without an LLM call per page. Agents run the same Actors over MCP.