---
title: "GitHub · Scrape.ly Docs"
description: "Commit results into a repository, run Actors from GitHub Actions, and connect your account."
url: https://scrape.ly/docs/github
markdown: https://scrape.ly/docs/github.md
---

# GitHub

> Commit results into a repository, run Actors from GitHub Actions, and connect your account.

Scrape.ly works with GitHub in three ways: it commits the results of each run into a repository, the GitHub Action runs Actors from your workflows, and a connected repository rebuilds its Actor on every push.

## Commit results after each run

Commits the dataset of each successful run as a file into one of your GitHub repositories. Set it up with the **Commit results to GitHub** recipe in **New automation**, or add **GitHub** as a delivery in the [automation editor](https://scrape.ly/docs/automations.md#5-send) to commit the results of several Actors, after filters and an AI step. It uses the GitHub connection under **Settings** → **Connections**. This integration only adds commits; it never starts runs. To start runs from a GitHub workflow, use the [GitHub Action](#run-actors-from-github-actions). To rebuild an Actor on every push, see [CI/CD: rebuild on git push](https://scrape.ly/docs/integrations.md#cicd-rebuild-on-git-push).

1. Click **New automation** (in **Automations**, or on an Actor's **Automations** tab) and pick **Commit results to GitHub**. If GitHub isn't connected yet, click **Connect GitHub**.
2. Pick the **Repository** (`owner/name`) and, optionally, a **Branch** (the default branch if empty).
3. Choose what happens **On each run**:
   - **Update one file** (`overwrite`): `<folder>/<actor-name>.<format>` is replaced each run. The commit history keeps every version, so you can compare runs.
   - **New file per run** (`per-run`): `<folder>/<actor-name>/<run-id>.<format>`.
4. Set the **Folder** (default `data`), the **File format** (JSON, JSONL or CSV) and **Max items per run** (1 to 100,000, default 1,000).
5. Use **Save and send test**. A commit adds `scrapely-test.txt` to the folder.

**Start your own workflow**: turn on **Also start a workflow in this repository** to send a `repository_dispatch` event named `scrapely-run-finished` after each commit. A workflow can react to it:

```yaml
on:
  repository_dispatch:
    types: [scrapely-run-finished]
jobs:
  use-results:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: echo "New results in $RESULTS_PATH ($ITEMS items)"
        env:
          RESULTS_PATH: ${{ github.event.client_payload.path }}
          ITEMS: ${{ github.event.client_payload.items }}
```

`client_payload` holds `actor`, `runId`, `path`, `items` and `format`.

**When it commits**: only for succeeded runs, and only when the run returned items. A file can be at most 25 MB; lower **Max items per run** if a run is bigger. The file has the same content as downloading the dataset in that format.

**Limits**: each GitHub integration commits at most 30 times an hour, and an account at most 500 exports a day (GitHub, S3 and Google Sheets together). If GitHub isn't connected, the repository or branch can't be written, or the connection expired, the delivery log on the integration's page says so.

**API**: a webhook with a `github` notification. Only `ACTOR.RUN.SUCCEEDED` is allowed, and GitHub must be connected.

```bash
curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to GitHub",
    "notification": {
      "channel": "github",
      "repo": "owner/repository",
      "branch": "main",
      "folder": "data",
      "fileMode": "overwrite",
      "format": "json",
      "maxItems": 1000,
      "dispatchWorkflow": true
    }
  }'
```

## Run Actors from GitHub Actions

The [Scrape.ly GitHub Action](https://github.com/scrapely-app/scrapely-action) starts a run of an Actor or saved task from a GitHub workflow, waits for it to finish, and saves the results to a file or passes them to the next step:

```yaml
- uses: scrapely-app/scrapely-action@v1
  with:
    api-token: ${{ secrets.SCRAPELY_TOKEN }}
    actor: username/actor-name
    input: '{"maxItems": 100}'
    output-file: data/results.json
```

Store an API key as the repository secret `SCRAPELY_TOKEN`. A scoped key needs `actors:run`, `runs:read`, and `datasets:read`. See [API keys & permissions](https://scrape.ly/docs/api-keys.md). The action's README lists all inputs and outputs, including `task`, `output-format` (JSON, JSONL or CSV), and `wait`.

In the dashboard, click **New automation** (on an Actor's **Automations** tab for that Actor) and pick **GitHub Action** under **From your code and tools**. It shows these steps and a complete workflow filled in for the Actor, ready to copy. The automation editor shows the same card. The results file has the same content as the [GitHub integration](#commit-results-after-each-run) commits and a dataset download in that format.

To have Scrape.ly commit the results into a repository after each run instead, use the [GitHub integration](#commit-results-after-each-run).

To trigger a build from a workflow, see [CI/CD: rebuild on git push](https://scrape.ly/docs/integrations.md#cicd-rebuild-on-git-push); to run CLI commands in a pipeline, see [CLI CI/CD](https://scrape.ly/docs/cli-config.md).

## Connect your GitHub account

Connect GitHub under **Settings** → **Connections**. **GitHub** lets you create Actors from your repositories, and registers a push webhook on each repository you link so pushes rebuild the Actor. The [GitHub integration](#commit-results-after-each-run) uses the same connection to commit run results. `GET /v2/integrations/github/repos` lists the repositories Scrape.ly can see.

`GET /v2/integrations` returns the connection status, and `DELETE /v2/integrations/github` disconnects it. See [Search, alerts & utilities](https://scrape.ly/docs/api-misc.md#integrations).

## Rebuild an Actor on every push

When an Actor's source is a GitHub repository, each push to the linked branch starts a new build. See [CI/CD: rebuild on git push](https://scrape.ly/docs/integrations.md#cicd-rebuild-on-git-push).

---

- Previous: [Amazon S3 & R2](https://scrape.ly/docs/amazon-s3.md)
- Next: [LangChain](https://scrape.ly/docs/langchain.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)