---
title: "LangChain · Scrape.ly Docs"
description: "Load Actor results as documents and give agents Scrape.ly tools."
url: https://scrape.ly/docs/langchain
markdown: https://scrape.ly/docs/langchain.md
---

# LangChain

> Load Actor results as documents and give agents Scrape.ly tools.

The `langchain-scrapely` package connects Scrape.ly to [LangChain](https://www.langchain.com). Run an Actor and load its results as documents for a vector index or RAG, or give an agent tools that run Actors and read their results. Python 3.11 or newer.

```bash
pip install langchain-scrapely
export SCRAPELY_TOKEN=your-api-key
```

Create the API key in the [dashboard](https://my.scrape.ly) under **Settings → API keys**. Every class reads it from `SCRAPELY_TOKEN`, or takes it as `scrapely_token=...`. A scoped key needs `actors:run`, `runs:read`, and `datasets:read`. See [API keys & permissions](https://scrape.ly/docs/api-keys.md).

The package is on [PyPI](https://pypi.org/project/langchain-scrapely/), and its source code is on GitHub: [scrapely-app/langchain-scrapely](https://github.com/scrapely-app/langchain-scrapely).

## Load results as documents

`ScrapelyWrapper` runs an Actor, waits for it to finish, and returns a loader for its results. `dataset_mapping_function` decides how each result becomes a LangChain `Document`.

```python
from langchain_core.documents import Document
from langchain_scrapely import ScrapelyWrapper

loader = ScrapelyWrapper().call_actor(
    actor_id='username/actor-name',
    run_input={'startUrls': [{'url': 'https://example.com'}]},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get('text') or '', metadata={'source': item.get('url')}
    ),
)
docs = loader.load()
```

| Method | What it does |
|---|---|
| `call_actor(actor_id, run_input, dataset_mapping_function)` | Runs an Actor. |
| `call_actor_task(task_id, task_input, dataset_mapping_function)` | Runs a [saved task](https://scrape.ly/docs/integrations.md#tasks). Fields in `task_input` override its saved input. |
| `acall_actor(...)`, `acall_actor_task(...)` | Async versions. |

All four take `build`, `memory_mbytes`, and `timeout_secs` for the run. A run that doesn't succeed raises `RuntimeError` with its status.

### Build a vector index

This example uses OpenAI embeddings: `pip install langchain-openai` and set `OPENAI_API_KEY`.

```python
from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings

store = InMemoryVectorStore.from_documents(loader.load(), OpenAIEmbeddings())
retriever = store.as_retriever()
retriever.invoke('What does the site say about pricing?')
```

### Load results that already exist

`ScrapelyDatasetLoader` reads a dataset, for example the results of an earlier run, without starting a new one.

```python
from langchain_scrapely import ScrapelyDatasetLoader

loader = ScrapelyDatasetLoader(
    dataset_id='YOUR_DATASET_ID',
    dataset_mapping_function=lambda item: Document(page_content=item.get('text') or ''),
)
docs = loader.load()
```

`lazy_load()` and `alazy_load()` yield one document at a time, for large datasets.

## Tools for agents

### Any Actor as a tool

`ScrapelyActorsTool` turns one Actor into a tool. It reads the Actor's description and input schema, so the agent knows what the Actor does and how to fill in its input. It looks the Actor up when you create it, and raises `ValueError` if the Actor doesn't exist or the API key can't see it.

```python
from langchain_scrapely import ScrapelyActorsTool

tool = ScrapelyActorsTool('username/actor-name')
tool.invoke({'run_input': {'query': 'coffee shops in Berlin'}})
```

The tool is named after the Actor (`username--actor-name`). It works for public Actors and your own. `timeout_secs` (default `300`) and `dataset_items_limit` (default `100`) set the run limit and how many results go back to the agent.

### General tools

These let an agent pick the Actor itself, and read results in pages.

| Class | Tool name | What it does |
|---|---|---|
| `ScrapelyRunActorTool` | `scrapely_run_actor` | Runs an Actor and returns the run, including its `dataset_id`. |
| `ScrapelyRunActorAndGetDatasetTool` | `scrapely_run_actor_and_get_dataset` | Runs an Actor and returns its results. |
| `ScrapelyGetDatasetItemsTool` | `scrapely_get_dataset_items` | Reads items from a dataset, with `limit` and `offset`. |
| `ScrapelyRunTaskTool` | `scrapely_run_task` | Runs a saved task and returns the run. |
| `ScrapelyRunTaskAndGetDatasetTool` | `scrapely_run_task_and_get_dataset` | Runs a saved task and returns its results. |

`SCRAPELY_CORE_TOOLS` holds all five as classes:

```python
from langchain_scrapely import SCRAPELY_CORE_TOOLS

tools = [cls() for cls in SCRAPELY_CORE_TOOLS]
```

The tools that return results answer with JSON:

```json
{
  "run": { "id": "...", "status": "SUCCEEDED", "status_message": null, "dataset_id": "...", "started_at": "...", "finished_at": "..." },
  "items": [ ... ]
}
```

`scrapely_run_actor` and `scrapely_run_task` return only `run`, and `scrapely_get_dataset_items` returns `"run": null`. A run that fails comes back with its status and no items. Errors, such as an unknown Actor or invalid arguments, go back to the agent as text instead of stopping it.

### Limits

Set these when you create a tool to cap what the agent may ask for:

| Option | Default | What it limits |
|---|---|---|
| `max_timeout_secs` | `600` | Longest run the agent may start. |
| `max_memory_mbytes` | `32768` | Most memory per run. Requests are rounded down to a power of two; anything under 128 MB gets 128 MB. |
| `max_items` | `1000` | Most results returned at once. |

### Use the tools in an agent

`create_agent` is in the `langchain` package, and the model below needs `langchain-openai`:

```bash
pip install langchain langchain-openai
```

```python
from langchain.agents import create_agent
from langchain_scrapely import ScrapelyActorsTool

agent = create_agent('openai:gpt-4o', tools=[ScrapelyActorsTool('username/actor-name')])
agent.invoke({'messages': [{'role': 'user', 'content': 'Find the top coffee shops in Berlin'}]})
```

Give an agent only the tools it needs. Fewer tools means better choices, faster answers, and fewer tokens. For agents with several steps or tools, see [LangGraph](https://scrape.ly/docs/langgraph.md).

---

- Previous: [LangChain ecosystem](https://scrape.ly/docs/langchain-ecosystem.md)
- Next: [LangGraph](https://scrape.ly/docs/langgraph.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)