Web Scraping for LangChain and LangGraph Agents

LangChain web scraping usually means one of two jobs: loading web pages as Document objects for a RAG index, or giving an agent a tool it can call to fetch live data while it works. The langchain-scrapely package does both. It runs a Scrape.ly Actor in the cloud, waits for it, and hands you the results as LangChain documents or as tool output, so the crawling, rendering, proxies and retries happen on Scrape.ly while your chain stays a few lines of Python.

This guide covers installing the package, a tested document loader that crawls a site and splits it into chunks, building a vector index, agent tools for LangChain and LangGraph, which Actor to use for which job, and a real crawl from 27 October 2026 with its exact cost.

Why LangChain web scraping belongs outside the chain

You can fetch pages from inside a chain with a few lines of requests, and for a handful of static pages that's fine. Real projects hit the same walls quickly: pages that need JavaScript, sites that block datacenter addresses, crawls that take minutes rather than seconds, and agents that call a fetch tool fifty times in one conversation. Every one of those turns into infrastructure you now own inside your application process.

Moving the scraping into an Actor separates the concerns. The Actor handles discovery, rendering, proxies, retries and storage in the cloud, with a maximum charge per run. Your LangChain code only says what to run and how to turn each result into a Document. That also makes the data reusable: the same crawl can feed a vector index, a nightly report and an agent's tool without being fetched three times.

It changes how agents behave, too. A tool backed by a site-specific Actor returns structured fields (price, rating, URL) instead of raw HTML, so the model spends its tokens on the question instead of on parsing markup.

Install

langchain-scrapely is on PyPI (version 0.1.0 on 27 October 2026) and needs Python 3.11 or newer. Its source is on GitHub.

pip install langchain-scrapely langchain-text-splitters
export SCRAPELY_TOKEN=your-api-key

Create the API key in the dashboard under Settings → API & Integrations → API keys. A scoped key needs actors:run, runs:read and datasets:read, per the LangChain docs.

Load a website as LangChain Documents

ScrapelyWrapper().call_actor() starts an Actor, waits for the run to finish and returns a loader for its dataset. You decide how each result becomes a Document. This script crawls four pages of the Python tutorial with the Website Content Crawler and splits them for embedding:

from langchain_core.documents import Document
from langchain_scrapely import ScrapelyWrapper
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = ScrapelyWrapper().call_actor(
    actor_id="querystone/website-content-crawler",
    run_input={"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxCrawlPages": 4},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get("text") or "",
        metadata={"source": item["url"], "title": item["metadata"]["title"]},
    ),
)
docs = loader.load()
chunks = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100).split_documents(docs)

for doc in docs:
    print(f"{len(doc.page_content):>6} chars  {doc.metadata['title']}")
print(len(docs), "documents,", len(chunks), "chunks")

Its output on the real crawl, below:

  2318 chars  The Python Tutorial — Python 3.15.2 documentation
  4431 chars  1. Whetting Your Appetite — Python 3.15.2 documentation
 12619 chars  3. An Informal Introduction to Python — Python 3.15.2 documentation
  4389 chars  2. Using the Python Interpreter — Python 3.15.2 documentation
4 documents, 28 chunks

How we tested it: the crawl ran on Scrape.ly through the MCP server, then the script above ran unchanged with langchain-scrapely 0.1.0 against a local replay of that run's API responses. The package made the same four calls a live run makes: start the run, wait for it, and read the dataset in pages.

Any Actor works the same way. Only actor_id, run_input and the mapping function change: map markdown instead of text for Markdown chunks, or build page_content from a product's title and description for an ecommerce index.

Build a vector index

With documents in hand, the rest is standard LangChain. With OpenAI embeddings (pip install langchain-openai, OPENAI_API_KEY set):

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings

store = InMemoryVectorStore.from_documents(chunks, OpenAIEmbeddings())
retriever = store.as_retriever()
retriever.invoke("How do I start the Python interpreter?")

Swap InMemoryVectorStore for any vector store LangChain supports. Our guide to website to Markdown for LLMs covers chunking choices for RAG in more depth.

Load results that already exist

If a scheduled run already crawled the site, read its dataset without starting a new one:

from langchain_core.documents import Document
from langchain_scrapely import ScrapelyDatasetLoader

loader = ScrapelyDatasetLoader(
    dataset_id="YOUR_DATASET_ID",
    dataset_mapping_function=lambda item: Document(page_content=item.get("text") or ""),
)
docs = loader.load()

lazy_load() yields one document at a time for large datasets. That pairs well with a schedule: crawl nightly on Scrape.ly, rebuild the index from the latest dataset in the morning.

Give a LangChain agent a scraping tool

ScrapelyActorsTool turns one Actor into a tool. It reads the Actor's description and input schema, so the model knows what the tool does and how to fill in its input:

from langchain.agents import create_agent
from langchain_scrapely import ScrapelyActorsTool

agent = create_agent(
    "openai:gpt-4o",
    tools=[ScrapelyActorsTool("querystone/google-search-scraper")],
)
result = agent.invoke({"messages": [{"role": "user", "content": "Who ranks top 3 on Google in the US for 'rag web browser'?"}]})
print(result["messages"][-1].content)

create_agent comes from the langchain package and builds a LangGraph agent, so the same code is your LangGraph quick start. The tool is named after the Actor (querystone--google-search-scraper), and timeout_secs (default 300) and dataset_items_limit (default 100) cap each run and how many results go back to the model.

Let the agent choose the Actor

The general tools let an agent run any Actor or saved task by name and read results in pages:

from langchain.agents import create_agent
from langchain_scrapely import SCRAPELY_CORE_TOOLS

agent = create_agent("openai:gpt-4o", tools=[cls() for cls in SCRAPELY_CORE_TOOLS])
for step in agent.stream(
    {"messages": [{"role": "user", "content": "Run querystone/rag-web-browser for 'LangGraph checkpointing' and summarize the top pages"}]},
    stream_mode="values",
):
    step["messages"][-1].pretty_print()

SCRAPELY_CORE_TOOLS holds five tools: run an Actor, run an Actor and get its dataset, read dataset items with limit and offset, and the same two for saved tasks. Set max_timeout_secs, max_memory_mbytes and max_items when you create them to cap what the agent may ask for. Errors such as an unknown Actor go back to the agent as text instead of crashing the graph.

Give an agent only the tools it needs. One Actor tool per job beats five general tools for accuracy, speed and tokens. The LangGraph docs show combining several Actor tools in one agent.

Which Actor for which LangChain job

For the search-then-read pattern, our list of web search MCP servers and Tavily alternatives compare the options. If your agent runs in Claude or Cursor rather than LangChain, the hosted MCP server exposes the same Actors without code.

A real crawl and its cost

We ran the Website Content Crawler on 27 October 2026 with the input from the loader above (four pages from https://docs.python.org/3/tutorial/). It saved 4 pages in 5 seconds:

Saved 4 pages. Stopped at Max pages (4); raise it to crawl more.

The Actor charges $0.0005 per crawled page plus platform usage at your plan's rates. Our run: 4 × $0.0005 = $0.002 in page fees at list price, plus $0.00023 of compute, proxy and storage; the account was charged $0.0018 because its plan has a Store discount on the page fee. That's about $0.00056 per page at list price, in line with the Store page's estimate of about $0.60 per 1,000 pages all in.

At that rate, indexing a 2,000-page documentation site costs about $1.20 per full crawl, and the Free plan's $5 of monthly usage covers roughly 8,000 pages a month.

Prompting an agent that has scraping tools

A few habits make scraping tools behave well in an agent:

• Name the Actor's job in the system prompt. "Use the Google search tool for current rankings, never guess positions" stops the model from answering from memory.
• Ask for small inputs first. Tell the agent to start with a few results and widen only if needed; dataset_items_limit enforces it.
• Return fields, not pages. Map tool output to the fields the task needs, so the model doesn't wade through full page text.
• Keep long crawls out of the loop. Let a schedule crawl and let the agent read the dataset with ScrapelyGetDatasetItemsTool, rather than starting a 2,000-page crawl mid-conversation.

What to watch out for

• Map the fields you need. Each Actor has its own output; the mapping function is where text, markdown, url and metadata become your Document. Check an Actor's output fields on its Store page first.
• Runs take time. call_actor waits for the run to finish. For long crawls, run on a schedule and load with ScrapelyDatasetLoader instead of crawling inside a request.
• Agents can spend. Cap runs with max_timeout_secs, max_items and the Actor's own limits, and use a scoped API key.
• A failed run raises. call_actor raises RuntimeError with the run's status when a run doesn't succeed; catch it in production chains.
• Respect the sites you crawl. Turn on robots.txt checks for broad crawls; our robots.txt guide explains why.

FAQ

What is the best way to scrape websites for LangChain?

Run the scraping outside your chain and load the results as documents. With langchain-scrapely, a crawler Actor does the fetching, rendering and proxies in the cloud, and call_actor returns a loader whose load() gives you Document objects.

Does langchain-scrapely work with LangGraph?

Yes. create_agent from the langchain package builds a LangGraph agent, and the Scrape.ly tools plug into it. There's no separate LangGraph package.

Can a LangChain agent scrape any website?

An agent with SCRAPELY_CORE_TOOLS can run any Actor in the Store, including general crawlers and site-specific scrapers. Limit it to the tools a task needs for better results.

How much does it cost to scrape a site for RAG?

On our crawl, about $0.00056 per page at list price with the Website Content Crawler, platform usage included. A 2,000-page site costs around $1.20 per full crawl.

Which Python version does langchain-scrapely need?

Python 3.11 or newer, with langchain-core 0.3.15 or later.