--- title: "LangChain Web Scraping: Loaders and Agent Tools | Scrape.ly" description: "LangChain web scraping with langchain-scrapely: load crawled pages as Documents for RAG and give LangGraph agents scraping tools. Tested code, a real run." url: https://scrape.ly/blog/langchain-web-scraping markdown: https://scrape.ly/blog/langchain-web-scraping.md --- [AI agents](https://scrape.ly/blog/category/ai-agents.md) # Web Scraping for LangChain and LangGraph Agents Web scraping for LangChain: load crawled pages as Documents for RAG, give LangChain and LangGraph agents tools that run scrapers, and keep runs and costs in check. Tested Python with the langchain-scrapely package and a real crawl from 27 October 2026 with its exact cost. [Scrape.ly Team The team behind Scrape.ly](https://scrape.ly/blog/author/scrapely-team.md) - Published Oct 27, 2026 - Reading time 7 min read ![Pixel-art robot with an orange >\_ on its terminal head carries the end of a metal chain that runs from a web page into a dark machine whose screen shows the light-blue LangChain logo](https://scrape.ly/media/langchain-web-scraping.png?w=1080) LangChain web scraping usually means one of two jobs: loading web pages as `Document` objects for a RAG index, or giving an agent a tool it can call to fetch live data while it works. The `langchain-scrapely` package does both. It runs a Scrape.ly Actor in the cloud, waits for it, and hands you the results as LangChain documents or as tool output, so the crawling, rendering, proxies and retries happen on Scrape.ly while your chain stays a few lines of Python. This guide covers installing the package, a tested document loader that crawls a site and splits it into chunks, building a vector index, agent tools for LangChain and LangGraph, which Actor to use for which job, and a real crawl from 27 October 2026 with its exact cost. ## Why LangChain web scraping belongs outside the chain You can fetch pages from inside a chain with a few lines of `requests`, and for a handful of static pages that's fine. Real projects hit the same walls quickly: pages that need JavaScript, sites that block datacenter addresses, crawls that take minutes rather than seconds, and agents that call a fetch tool fifty times in one conversation. Every one of those turns into infrastructure you now own inside your application process. Moving the scraping into an Actor separates the concerns. The Actor handles discovery, rendering, proxies, retries and storage in the cloud, with a maximum charge per run. Your LangChain code only says what to run and how to turn each result into a `Document`. That also makes the data reusable: the same crawl can feed a vector index, a nightly report and an agent's tool without being fetched three times. It changes how agents behave, too. A tool backed by a site-specific Actor returns structured fields (price, rating, URL) instead of raw HTML, so the model spends its tokens on the question instead of on parsing markup. ## Install `langchain-scrapely` is on [PyPI](https://pypi.org/project/langchain-scrapely/) (version 0.1.0 on 27 October 2026) and needs Python 3.11 or newer. Its source is on [GitHub](https://github.com/scrapely-app/langchain-scrapely). ```shell pip install langchain-scrapely langchain-text-splitters export SCRAPELY_TOKEN=your-api-key ``` Create the API key in the dashboard under **Settings → API & Integrations → API keys**. A scoped key needs `actors:run`, `runs:read` and `datasets:read`, per the [LangChain docs](https://scrape.ly/docs/langchain.md). ## Load a website as LangChain Documents `ScrapelyWrapper().call_actor()` starts an Actor, waits for the run to finish and returns a loader for its dataset. You decide how each result becomes a `Document`. This script crawls four pages of the Python tutorial with the [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) and splits them for embedding: ```python from langchain_core.documents import Document from langchain_scrapely import ScrapelyWrapper from langchain_text_splitters import RecursiveCharacterTextSplitter loader = ScrapelyWrapper().call_actor( actor_id="querystone/website-content-crawler", run_input={"startUrls": [{"url": "https://docs.python.org/3/tutorial/"}], "maxCrawlPages": 4}, dataset_mapping_function=lambda item: Document( page_content=item.get("text") or "", metadata={"source": item["url"], "title": item["metadata"]["title"]}, ), ) docs = loader.load() chunks = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100).split_documents(docs) for doc in docs: print(f"{len(doc.page_content):>6} chars {doc.metadata['title']}") print(len(docs), "documents,", len(chunks), "chunks") ``` Its output on the real crawl, below: ```plaintext 2318 chars The Python Tutorial — Python 3.15.2 documentation 4431 chars 1. Whetting Your Appetite — Python 3.15.2 documentation 12619 chars 3. An Informal Introduction to Python — Python 3.15.2 documentation 4389 chars 2. Using the Python Interpreter — Python 3.15.2 documentation 4 documents, 28 chunks ``` How we tested it: the crawl ran on Scrape.ly through the MCP server, then the script above ran unchanged with `langchain-scrapely` 0.1.0 against a local replay of that run's API responses. The package made the same four calls a live run makes: start the run, wait for it, and read the dataset in pages. Any Actor works the same way. Only `actor_id`, `run_input` and the mapping function change: map `markdown` instead of `text` for Markdown chunks, or build `page_content` from a product's title and description for an ecommerce index. ### Build a vector index With documents in hand, the rest is standard LangChain. With OpenAI embeddings (`pip install langchain-openai`, `OPENAI_API_KEY` set): ```python from langchain_core.vectorstores import InMemoryVectorStore from langchain_openai import OpenAIEmbeddings store = InMemoryVectorStore.from_documents(chunks, OpenAIEmbeddings()) retriever = store.as_retriever() retriever.invoke("How do I start the Python interpreter?") ``` Swap `InMemoryVectorStore` for any vector store LangChain supports. Our guide to [website to Markdown for LLMs](https://scrape.ly/blog/website-to-markdown-for-llms.md) covers chunking choices for RAG in more depth. ### Load results that already exist If a scheduled run already crawled the site, read its dataset without starting a new one: ```python from langchain_core.documents import Document from langchain_scrapely import ScrapelyDatasetLoader loader = ScrapelyDatasetLoader( dataset_id="YOUR_DATASET_ID", dataset_mapping_function=lambda item: Document(page_content=item.get("text") or ""), ) docs = loader.load() ``` `lazy_load()` yields one document at a time for large datasets. That pairs well with a [schedule](https://scrape.ly/blog/schedule-web-scraping.md): crawl nightly on Scrape.ly, rebuild the index from the latest dataset in the morning. ## Give a LangChain agent a scraping tool `ScrapelyActorsTool` turns one Actor into a tool. It reads the Actor's description and input schema, so the model knows what the tool does and how to fill in its input: ```python from langchain.agents import create_agent from langchain_scrapely import ScrapelyActorsTool agent = create_agent( "openai:gpt-4o", tools=[ScrapelyActorsTool("querystone/google-search-scraper")], ) result = agent.invoke({"messages": [{"role": "user", "content": "Who ranks top 3 on Google in the US for 'rag web browser'?"}]}) print(result["messages"][1].content) ``` `create_agent` comes from the `langchain` package and builds a LangGraph agent, so the same code is your LangGraph quick start. The tool is named after the Actor (`querystone--google-search-scraper`), and `timeout_secs` (default 300) and `dataset_items_limit` (default 100) cap each run and how many results go back to the model. ### Let the agent choose the Actor The general tools let an agent run any Actor or saved task by name and read results in pages: ```python from langchain.agents import create_agent from langchain_scrapely import SCRAPELY_CORE_TOOLS agent = create_agent("openai:gpt-4o", tools=[cls() for cls in SCRAPELY_CORE_TOOLS]) for step in agent.stream( {"messages": [{"role": "user", "content": "Run querystone/rag-web-browser for 'LangGraph checkpointing' and summarize the top pages"}]}, stream_mode="values", ): step["messages"][1].pretty_print() ``` `SCRAPELY_CORE_TOOLS` holds five tools: run an Actor, run an Actor and get its dataset, read dataset items with `limit` and `offset`, and the same two for saved tasks. Set `max_timeout_secs`, `max_memory_mbytes` and `max_items` when you create them to cap what the agent may ask for. Errors such as an unknown Actor go back to the agent as text instead of crashing the graph. Give an agent only the tools it needs. One Actor tool per job beats five general tools for accuracy, speed and tokens. The [LangGraph docs](https://scrape.ly/docs/langgraph.md) show combining several Actor tools in one agent. ## Which Actor for which LangChain job | Job | Actor | What comes back | Price (27 Oct 2026) | | --------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------------------------- | -------------------------------------------------------- | | Index a docs site or blog for RAG | [Website Content Crawler](https://scrape.ly/store/querystone/website-content-crawler.md) | Clean text and Markdown per page | $0.50 per 1,000 pages plus usage | | Search the web, then read the top pages | [RAG Web Browser](https://scrape.ly/store/querystone/rag-web-browser.md) | Markdown of the top Google results | $3.99 per 1,000 results pages, $0.00157 per fetched page | | Live Google results in a tool | [Google Search Scraper](https://scrape.ly/store/querystone/google-search-scraper.md) | Organic results, People Also Ask, related searches | $3.30 per 1,000 results pages | | Structured data from a big site | A site Actor from the [Store](https://scrape.ly/store.md) | Products, listings, reviews as JSON | Per result | For the search-then-read pattern, our list of [web search MCP servers](https://scrape.ly/blog/web-search-mcp-server.md) and [Tavily alternatives](https://scrape.ly/blog/tavily-alternatives.md) compare the options. If your agent runs in Claude or Cursor rather than LangChain, the [hosted MCP server](https://scrape.ly/mcp.md) exposes the same Actors without code. Scrape.ly Store Website Content Crawler Crawl a site into clean text and Markdown for LangChain loaders and RAG: set Start URLs and Max pages, and load the dataset with langchain-scrapely. [Try it free](https://scrape.ly/store/querystone/website-content-crawler.md) [Browse the Store](https://scrape.ly/store.md) ## A real crawl and its cost We ran the Website Content Crawler on 27 October 2026 with the input from the loader above (four pages from `https://docs.python.org/3/tutorial/`). It saved 4 pages in 5 seconds: ```plaintext Saved 4 pages. Stopped at Max pages (4); raise it to crawl more. ``` The Actor charges $0.0005 per crawled page plus platform usage at your plan's rates. **Our run:** 4 × $0.0005 = $0.002 in page fees at list price, plus $0.00023 of compute, proxy and storage; the account was charged $0.0018 because its plan has a Store discount on the page fee. That's about $0.00056 per page at list price, in line with the Store page's estimate of about $0.60 per 1,000 pages all in. At that rate, indexing a 2,000-page documentation site costs about $1.20 per full crawl, and the Free plan's $5 of monthly usage covers roughly 8,000 pages a month. ## Prompting an agent that has scraping tools A few habits make scraping tools behave well in an agent: - **Name the Actor's job in the system prompt.** "Use the Google search tool for current rankings, never guess positions" stops the model from answering from memory. - **Ask for small inputs first.** Tell the agent to start with a few results and widen only if needed; `dataset_items_limit` enforces it. - **Return fields, not pages.** Map tool output to the fields the task needs, so the model doesn't wade through full page text. - **Keep long crawls out of the loop.** Let a schedule crawl and let the agent read the dataset with `ScrapelyGetDatasetItemsTool`, rather than starting a 2,000-page crawl mid-conversation. ## What to watch out for - **Map the fields you need.** Each Actor has its own output; the mapping function is where `text`, `markdown`, `url` and metadata become your `Document`. Check an Actor's output fields on its Store page first. - **Runs take time.** `call_actor` waits for the run to finish. For long crawls, run on a schedule and load with `ScrapelyDatasetLoader` instead of crawling inside a request. - **Agents can spend.** Cap runs with `max_timeout_secs`, `max_items` and the Actor's own limits, and use a scoped API key. - **A failed run raises.** `call_actor` raises `RuntimeError` with the run's status when a run doesn't succeed; catch it in production chains. - **Respect the sites you crawl.** Turn on robots.txt checks for broad crawls; our [robots.txt guide](https://scrape.ly/blog/robots-txt-web-scraping.md) explains why. ## FAQ ### What is the best way to scrape websites for LangChain? Run the scraping outside your chain and load the results as documents. With `langchain-scrapely`, a crawler Actor does the fetching, rendering and proxies in the cloud, and `call_actor` returns a loader whose `load()` gives you `Document` objects. ### Does langchain-scrapely work with LangGraph? Yes. `create_agent` from the `langchain` package builds a LangGraph agent, and the Scrape.ly tools plug into it. There's no separate LangGraph package. ### Can a LangChain agent scrape any website? An agent with `SCRAPELY_CORE_TOOLS` can run any Actor in the Store, including general crawlers and site-specific scrapers. Limit it to the tools a task needs for better results. ### How much does it cost to scrape a site for RAG? On our crawl, about $0.00056 per page at list price with the Website Content Crawler, platform usage included. A 2,000-page site costs around $1.20 per full crawl. ### Which Python version does langchain-scrapely need? Python 3.11 or newer, with `langchain-core` 0.3.15 or later. Scrape.ly Store RAG Web Browser Search Google and get the top result pages as clean Markdown for LangChain agents and RAG, or fetch a single URL. Callable as a tool with langchain-scrapely. [Try it free](https://scrape.ly/store/querystone/rag-web-browser.md) [Browse the Store](https://scrape.ly/store.md) About the author [Scrape.ly Team](https://scrape.ly/blog/author/scrapely-team.md) The team behind Scrape.ly Scrape.ly is the web data platform from TeraShift GmbH in Switzerland, powered by Evomi: ready-made scrapers in the Scrape.ly Store, a cloud runtime, an API with JavaScript and Python SDKs, proxies, a hosted anti-bot browser and an MCP server for AI agents. - [Website](https://scrape.ly/index.md) Keep reading ## More in AI agents - [![Pixel-art robot with an orange >\_ on its terminal head, an AI sparkle in a speech bubble above it, drops a gold coin into a pay terminal beside an open gate whose sign shows the black x402 logo, a crate of data waiting behind the gate](https://scrape.ly/media/x402-for-ai-agents.png?w=750)](https://scrape.ly/blog/x402-for-ai-agents.md) ### [x402 Explained: How AI Agents Pay per API Call](https://scrape.ly/blog/x402-for-ai-agents.md) [x402 is an open standard that lets an AI agent pay for an HTTP request with a stablecoin, using the 402 Payment Required status code instead of an account and an API key. Here's how the protocol works, who runs it, and how to pay for Scrape.ly runs with USDC on Base, with tested code.](https://scrape.ly/blog/x402-for-ai-agents.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a feed of posts headed by the blue Facebook logo into a server rack labelled MCP](https://scrape.ly/media/facebook-mcp.png?w=750)](https://scrape.ly/blog/facebook-mcp.md) ### [Facebook MCP: Pages, Posts and Ads for Claude, ChatGPT and Cursor](https://scrape.ly/blog/facebook-mcp.md) [A Facebook MCP server lets your AI agent read public Facebook pages, posts, reactions and Meta ads in the chat. See how Meta's own MCP servers differ, connect Scrape.ly's hosted server without a Facebook login or Page token, and see two real runs with output and costs.](https://scrape.ly/blog/facebook-mcp.md) - [![Pixel-art robot with an orange >\_ on its terminal head carries a cable from a phone showing the TikTok note logo on a short video, with like, comment and share icons down its edge, into a server rack labelled MCP](https://scrape.ly/media/tiktok-mcp.png?w=750)](https://scrape.ly/blog/tiktok-mcp.md) ### [TikTok MCP: Videos and Comments for Claude, ChatGPT and Cursor](https://scrape.ly/blog/tiktok-mcp.md) [A TikTok MCP server lets your AI agent read public TikTok videos, play counts and comments right in the chat. See how TikTok's own ads connector differs, connect Scrape.ly's hosted server to Claude, ChatGPT or Cursor without a TikTok account, and see a real run with output and costs.](https://scrape.ly/blog/tiktok-mcp.md) ## The internet is full of data. Start owning yours. Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes. [Get Started Free](https://my.scrape.ly/register) [Chat with Support](mailto:hello@scrape.ly?subject=Question%20about%20Scrape.ly\&body=Hi%2C%20I%20have%20a%20question%20about%20Scrape.ly.) No credit card. No setup calls.