---
title: "Dify · Scrape.ly Docs"
description: "Scrape websites and run Actors in Dify workflows and agents with the Scrape.ly plugin."
url: https://scrape.ly/docs/dify
markdown: https://scrape.ly/docs/dify.md
---

# Dify

> Scrape websites and run Actors in Dify workflows and agents with the Scrape.ly plugin.

Use Scrape.ly in Dify workflows, chatflows and agents: scrape a web page, find and run Actors and tasks, and read their results. The Scrape.ly plugin is in the [Dify Marketplace](https://marketplace.dify.ai), so there is no code to write.

## Install the plugin

1. In Dify, open **Integrations**, find **Scrape.ly** in the Marketplace, and install it.
2. Create an API key in the [dashboard](https://my.scrape.ly) under **Settings → API & Integrations**.
3. In Dify, open **Integrations → Tools**, select Scrape.ly, and add the key as its credentials.

The plugin can do anything the key can, including starting paid runs, so give it a scoped key: `actors:run`, `runs:read` and `datasets:read`, with a **Max charge per day**. If you limit the key to certain Actors, include `querystone/website-content-crawler`, which **Scrape website** runs. See [API keys & permissions](https://scrape.ly/docs/api-keys.md).

## Tools

| Tool | What it does |
| --- | --- |
| **Scrape website** | Scrapes a web page, and up to 20 pages it links to if you raise **Max pages**, into title, text and markdown. |
| **Run Actor** | Runs an Actor with a JSON input, waits for it to finish, and returns its results. |
| **Run task** | Runs a saved task, optionally overriding fields of its input, and returns its results. |
| **Start run** | Starts an Actor run without waiting, for runs that take longer than 5 minutes. |
| **Get run** | Checks a run's status, optionally waiting up to 45 seconds for it to finish. |
| **Get dataset items** | Reads the results stored in a dataset, for example from an earlier run. |
| **Search Actors** | Finds ready-made scrapers (Actors) in the Store by keyword. |

**Run Actor** and **Run task** wait up to 5 minutes. For a longer run, use **Start run**, check it with **Get run** until its status is `SUCCEEDED`, then read the results with **Get dataset items** and the run's `dataset_id`. Runs are also listed on the [Runs](https://my.scrape.ly/actors/runs) page. **Max items** on Run Actor and Run task, and **Limit** on Get dataset items, cap how many results they return (100 by default, at most 1,000).

Actors are named by ID or `username/actor-name`, as shown on the Actor's page. The input is the same JSON as the Actor's **Input** tab in JSON view.

## In a workflow

Add a **Scrape.ly → Scrape website** node and set **URL** to a variable, such as a URL from the start node. Its `text` output is the scraped markdown, ready for an LLM node; its `json` output has a `pages` list with each page's `url`, `title`, `text` and `markdown`.

For a site that needs a dedicated scraper, such as Google Maps or a job board, use **Run Actor** instead, with the Actor's input as JSON. It returns one item per result, in the `json` output's `items`.

## In an agent

Add the Scrape.ly tools to an **Agent** node or an agent app. The agent picks the tool and fills in the URL, Actor and input itself, and can find a fitting Actor with **Search Actors**. Name the Actor in the instructions, for example `Use the Actor username/actor-name to look up companies`, so it doesn't guess.

**Costs**: runs started from Dify are billed like any run. Keep **Max items** and **Limit** low and a **Max charge per day** on the key.

---

- Previous: [Upsonic](https://scrape.ly/docs/upsonic.md)
- Next: [JavaScript client](https://scrape.ly/docs/client-js.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)