GitHub

Scrape.ly works with GitHub in three ways: it commits the results of each run into a repository, the GitHub Action runs Actors from your workflows, and a connected repository rebuilds its Actor on every push.

Commit results after each run

Commits the dataset of each successful run as a file into one of your GitHub repositories. Set it up with the Commit results to GitHub recipe in New automation, or add GitHub as a delivery in the automation editor to commit the results of several Actors, after filters and an AI step. It uses the GitHub connection under Settings → Connections. This integration only adds commits; it never starts runs. To start runs from a GitHub workflow, use the GitHub Action. To rebuild an Actor on every push, see CI/CD: rebuild on git push.

1. Click New automation (in Automations, or on an Actor's Automations tab) and pick Commit results to GitHub. If GitHub isn't connected yet, click Connect GitHub.
2. Pick the Repository (owner/name) and, optionally, a Branch (the default branch if empty).
3. Choose what happens On each run:
   • Update one file (overwrite): <folder>/<actor-name>.<format> is replaced each run. The commit history keeps every version, so you can compare runs.
   • New file per run (per-run): <folder>/<actor-name>/<run-id>.<format>.
4. Set the Folder (default data), the File format (JSON, JSONL or CSV) and Max items per run (1 to 100,000, default 1,000).
5. Use Save and send test. A commit adds scrapely-test.txt to the folder.

Start your own workflow: turn on Also start a workflow in this repository to send a repository_dispatch event named scrapely-run-finished after each commit. A workflow can react to it:

on:
  repository_dispatch:
    types: [scrapely-run-finished]
jobs:
  use-results:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: echo "New results in $RESULTS_PATH ($ITEMS items)"
        env:
          RESULTS_PATH: ${{ github.event.client_payload.path }}
          ITEMS: ${{ github.event.client_payload.items }}

client_payload holds actor, runId, path, items and format.

When it commits: only for succeeded runs, and only when the run returned items. A file can be at most 25 MB; lower Max items per run if a run is bigger. The file has the same content as downloading the dataset in that format.

Limits: each GitHub integration commits at most 30 times an hour, and an account at most 500 exports a day (GitHub, S3 and Google Sheets together). If GitHub isn't connected, the repository or branch can't be written, or the connection expired, the delivery log on the integration's page says so.

API: a webhook with a github notification. Only ACTOR.RUN.SUCCEEDED is allowed, and GitHub must be connected.

curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to GitHub",
    "notification": {
      "channel": "github",
      "repo": "owner/repository",
      "branch": "main",
      "folder": "data",
      "fileMode": "overwrite",
      "format": "json",
      "maxItems": 1000,
      "dispatchWorkflow": true
    }
  }'

Run Actors from GitHub Actions

The Scrape.ly GitHub Action starts a run of an Actor or saved task from a GitHub workflow, waits for it to finish, and saves the results to a file or passes them to the next step:

- uses: scrapely-app/scrapely-action@v1
  with:
    api-token: ${{ secrets.SCRAPELY_TOKEN }}
    actor: username/actor-name
    input: '{"maxItems": 100}'
    output-file: data/results.json

Store an API key as the repository secret SCRAPELY_TOKEN. A scoped key needs actors:run, runs:read, and datasets:read. See API keys & permissions. The action's README lists all inputs and outputs, including task, output-format (JSON, JSONL or CSV), and wait.

In the dashboard, click New automation (on an Actor's Automations tab for that Actor) and pick GitHub Action under From your code and tools. It shows these steps and a complete workflow filled in for the Actor, ready to copy. The automation editor shows the same card. The results file has the same content as the GitHub integration commits and a dataset download in that format.

To have Scrape.ly commit the results into a repository after each run instead, use the GitHub integration.

To trigger a build from a workflow, see CI/CD: rebuild on git push; to run CLI commands in a pipeline, see CLI CI/CD.

Connect your GitHub account

Connect GitHub under Settings → Connections. GitHub lets you create Actors from your repositories, and registers a push webhook on each repository you link so pushes rebuild the Actor. The GitHub integration uses the same connection to commit run results. GET /v2/integrations/github/repos lists the repositories Scrape.ly can see.

GET /v2/integrations returns the connection status, and DELETE /v2/integrations/github disconnects it. See Search, alerts & utilities.

Rebuild an Actor on every push

When an Actor's source is a GitHub repository, each push to the linked branch starts a new build. See CI/CD: rebuild on git push.