LangChain

The langchain-scrapely package connects Scrape.ly to LangChain. Run an Actor and load its results as documents for a vector index or RAG, or give an agent tools that run Actors and read their results. Python 3.11 or newer.

pip install langchain-scrapely
export SCRAPELY_TOKEN=your-api-key

Create the API key in the dashboard under Settings → API keys. Every class reads it from SCRAPELY_TOKEN, or takes it as scrapely_token=.... A scoped key needs actors:run, runs:read, and datasets:read. See API keys & permissions.

The package is on PyPI, and its source code is on GitHub: scrapely-app/langchain-scrapely.

Load results as documents

ScrapelyWrapper runs an Actor, waits for it to finish, and returns a loader for its results. dataset_mapping_function decides how each result becomes a LangChain Document.

from langchain_core.documents import Document
from langchain_scrapely import ScrapelyWrapper

loader = ScrapelyWrapper().call_actor(
    actor_id='username/actor-name',
    run_input={'startUrls': [{'url': 'https://example.com'}]},
    dataset_mapping_function=lambda item: Document(
        page_content=item.get('text') or '', metadata={'source': item.get('url')}
    ),
)
docs = loader.load()

All four take build, memory_mbytes, and timeout_secs for the run. A run that doesn't succeed raises RuntimeError with its status.

Build a vector index

This example uses OpenAI embeddings: pip install langchain-openai and set OPENAI_API_KEY.

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_openai import OpenAIEmbeddings

store = InMemoryVectorStore.from_documents(loader.load(), OpenAIEmbeddings())
retriever = store.as_retriever()
retriever.invoke('What does the site say about pricing?')

Load results that already exist

ScrapelyDatasetLoader reads a dataset, for example the results of an earlier run, without starting a new one.

from langchain_scrapely import ScrapelyDatasetLoader

loader = ScrapelyDatasetLoader(
    dataset_id='YOUR_DATASET_ID',
    dataset_mapping_function=lambda item: Document(page_content=item.get('text') or ''),
)
docs = loader.load()

lazy_load() and alazy_load() yield one document at a time, for large datasets.

Tools for agents

Any Actor as a tool

ScrapelyActorsTool turns one Actor into a tool. It reads the Actor's description and input schema, so the agent knows what the Actor does and how to fill in its input. It looks the Actor up when you create it, and raises ValueError if the Actor doesn't exist or the API key can't see it.

from langchain_scrapely import ScrapelyActorsTool

tool = ScrapelyActorsTool('username/actor-name')
tool.invoke({'run_input': {'query': 'coffee shops in Berlin'}})

The tool is named after the Actor (username--actor-name). It works for public Actors and your own. timeout_secs (default 300) and dataset_items_limit (default 100) set the run limit and how many results go back to the agent.

General tools

These let an agent pick the Actor itself, and read results in pages.

SCRAPELY_CORE_TOOLS holds all five as classes:

from langchain_scrapely import SCRAPELY_CORE_TOOLS

tools = [cls() for cls in SCRAPELY_CORE_TOOLS]

The tools that return results answer with JSON:

{
  "run": { "id": "...", "status": "SUCCEEDED", "status_message": null, "dataset_id": "...", "started_at": "...", "finished_at": "..." },
  "items": [ ... ]
}

scrapely_run_actor and scrapely_run_task return only run, and scrapely_get_dataset_items returns "run": null. A run that fails comes back with its status and no items. Errors, such as an unknown Actor or invalid arguments, go back to the agent as text instead of stopping it.

Limits

Set these when you create a tool to cap what the agent may ask for:

Use the tools in an agent

create_agent is in the langchain package, and the model below needs langchain-openai:

pip install langchain langchain-openai

from langchain.agents import create_agent
from langchain_scrapely import ScrapelyActorsTool

agent = create_agent('openai:gpt-4o', tools=[ScrapelyActorsTool('username/actor-name')])
agent.invoke({'messages': [{'role': 'user', 'content': 'Find the top coffee shops in Berlin'}]})

Give an agent only the tools it needs. Fewer tools means better choices, faster answers, and fewer tokens. For agents with several steps or tools, see LangGraph.