MCP & AI agents

LlamaIndex

Run Actors or read datasets as LlamaIndex documents.

The llama-index-readers-scrapely package has two LlamaIndex readers: one runs an Actor and returns its results as documents, the other reads a dataset that already exists. Python 3.11 or newer.

.bash

pip install llama-index-readers-scrapely
export SCRAPELY_TOKEN=your-api-key

Create the API key in the dashboard under Settings → API keys. Both readers read it from SCRAPELY_TOKEN, or take it as scrapely_token=.... A scoped key needs actors:run, runs:read, and datasets:read. See API keys & permissions.

The package is on PyPI, and its source code is on GitHub: scrapely-app/llama-index-readers-scrapely.

Run an Actor

.python

from llama_index.core import Document, VectorStoreIndex
from llama_index.readers.scrapely import ScrapelyActor

docs = ScrapelyActor().load_data(
    actor_id='username/actor-name',
    run_input={'startUrls': [{'url': 'https://example.com'}]},
    dataset_mapping_function=lambda item: Document(
        text=item.get('text') or '', metadata={'url': item.get('url')}
    ),
)

index = VectorStoreIndex.from_documents(docs)
index.as_query_engine().query('What does the site say about pricing?')

The index and query engine use OpenAI by default: pip install llama-index (or llama-index-embeddings-openai llama-index-llms-openai) and set OPENAI_API_KEY.

load_data waits for the run to finish. build, memory_mbytes, and timeout_secs set the run options. A run that doesn't succeed raises RuntimeError with its status.

Read an existing dataset

.python

from llama_index.readers.scrapely import ScrapelyDataset

docs = ScrapelyDataset().load_data(
    dataset_id='YOUR_DATASET_ID',
    dataset_mapping_function=lambda item: Document(text=item.get('text') or ''),
)

dataset_mapping_function decides how each result becomes a Document, and must return one.