Dify
Scrape websites and run Actors in Dify workflows and agents with the Scrape.ly plugin.
Use Scrape.ly in Dify workflows, chatflows and agents: scrape a web page, find and run Actors and tasks, and read their results. The Scrape.ly plugin is in the Dify Marketplace, so there is no code to write.
Install the plugin
- In Dify, open Integrations, find Scrape.ly in the Marketplace, and install it.
- Create an API key in the dashboard under Settings → API & Integrations.
- In Dify, open Integrations → Tools, select Scrape.ly, and add the key as its credentials.
The plugin can do anything the key can, including starting paid runs, so give it a scoped key: actors:run, runs:read and datasets:read, with a Max charge per day. If you limit the key to certain Actors, include querystone/website-content-crawler, which Scrape website runs. See API keys & permissions.
Tools
| Tool | What it does |
|---|---|
| Scrape website | Scrapes a web page, and up to 20 pages it links to if you raise Max pages, into title, text and markdown. |
| Run Actor | Runs an Actor with a JSON input, waits for it to finish, and returns its results. |
| Run task | Runs a saved task, optionally overriding fields of its input, and returns its results. |
| Start run | Starts an Actor run without waiting, for runs that take longer than 5 minutes. |
| Get run | Checks a run's status, optionally waiting up to 45 seconds for it to finish. |
| Get dataset items | Reads the results stored in a dataset, for example from an earlier run. |
| Search Actors | Finds ready-made scrapers (Actors) in the Store by keyword. |
Run Actor and Run task wait up to 5 minutes. For a longer run, use Start run, check it with Get run until its status is SUCCEEDED, then read the results with Get dataset items and the run's dataset_id. Runs are also listed on the Runs page. Max items on Run Actor and Run task, and Limit on Get dataset items, cap how many results they return (100 by default, at most 1,000).
Actors are named by ID or username/actor-name, as shown on the Actor's page. The input is the same JSON as the Actor's Input tab in JSON view.
In a workflow
Add a Scrape.ly → Scrape website node and set URL to a variable, such as a URL from the start node. Its text output is the scraped markdown, ready for an LLM node; its json output has a pages list with each page's url, title, text and markdown.
For a site that needs a dedicated scraper, such as Google Maps or a job board, use Run Actor instead, with the Actor's input as JSON. It returns one item per result, in the json output's items.
In an agent
Add the Scrape.ly tools to an Agent node or an agent app. The agent picks the tool and fills in the URL, Actor and input itself, and can find a fitting Actor with Search Actors. Name the Actor in the instructions, for example Use the Actor username/actor-name to look up companies, so it doesn't guess.
Costs: runs started from Dify are billed like any run. Keep Max items and Limit low and a Max charge per day on the key.