---
title: "BigQuery · Scrape.ly Docs"
description: "Add the results of each successful run as rows to a BigQuery table."
url: https://scrape.ly/docs/bigquery
markdown: https://scrape.ly/docs/bigquery.md
---

# BigQuery

> Add the results of each successful run as rows to a BigQuery table.

Adds the dataset items of each successful run as rows to a BigQuery table. Scrape.ly writes as its own Google service account, which you give access to one dataset: no key of yours is stored, and you can take the access back at any time.

## Give Scrape.ly access to a dataset

1. In the automation form, under **Give this account access**, copy the service account email.
2. In the [BigQuery console](https://console.cloud.google.com/bigquery), open the dataset's sharing permissions and add that email as a principal with the **BigQuery Data Editor** role.

Scrape.ly can then add rows to the tables of that dataset and create new tables in it; it can't see or change your other datasets. To take the access back, remove the email from the dataset's permissions.

With the `bq` tool, the same grant is:

```bash
bq add-iam-policy-binding \
  --member='serviceAccount:<the email from the form>' \
  --role='roles/bigquery.dataEditor' \
  my-project:scraping
```

## Link the dataset to your account

Scrape.ly only writes to a dataset that carries your account's label, so nobody else can send rows into a dataset you shared.

1. In the automation form, under **Link the dataset to your account**, copy the label (it looks like `scrapely:1a2b3c4d5e6f7a8b`).
2. In the BigQuery console, open the dataset's details, click **Edit details**, and add the label under **Labels** (key `scrapely`, value the part after the colon).

With the `bq` tool:

```bash
bq update --set_label scrapely:<the value from the form> my-project:scraping
```

Only the dataset's owner can set labels; the BigQuery Data Editor role Scrape.ly gets can't change them.

## Send results to a table

1. Click **New automation** (in **Automations**, or on an Actor's **Automations** tab), click **Or create manually** and, under **Send**, click **BigQuery**.
2. Enter the **Table** as `project.dataset.table`, for example `my-project.scraping.results`.
3. Optionally set **Max rows per run** (1 to 100,000, default 1,000).
4. Use **Save and send test**. The test checks that the dataset is shared and carries your label and, when the table exists, that Scrape.ly may add rows to it. It writes nothing.

**New tables**: if the table doesn't exist yet, the first run creates it, and BigQuery picks the column types from those results. To choose the types yourself, create the table first.

**Columns**: a result field is written to the column with the same name (upper or lower case doesn't matter). Characters other than letters, digits, and `_` in field names become `_`, so `price.amount` as a field name goes into `price_amount`. Nested objects go into `RECORD` columns, and lists into `REPEATED` columns; `null` values inside lists are left out. Any value for a `STRING` column is written as text, objects as JSON. Fields without a column are skipped, and the delivery history lists them. A row BigQuery can't take, such as text for a `NUMERIC` column, is skipped and counted; the other rows are still written.

**When it writes**: only for succeeded runs, and only when the run returned items. Each run is one BigQuery load job, so its rows appear together. Load jobs are free; you pay BigQuery only for storing the rows. Each run adds new rows; existing rows are never changed. An export that takes longer than 8 minutes is stopped; for very large runs, lower **Max rows per run**.

**Limits**: each BigQuery automation of this kind writes at most 30 times an hour, and an account at most 500 exports a day (Google Sheets, S3, GitHub, Airtable, HubSpot, Salesforce, Notion, Dropbox, OneDrive, Postgres, Azure Blob, BigQuery, OpenAI Vector Store, and Snowflake together). If the dataset isn't shared or doesn't carry your label, the table can't be written, BigQuery refuses every row, or none of the fields match a column, the delivery history on the automation's page says so.

## API

A webhook with a `bigquery` notification. Only `ACTOR.RUN.SUCCEEDED` is allowed. `GET https://api.scrape.ly/v2/integrations/bigquery` returns the `serviceAccountEmail` to share the dataset with and the `datasetLabel` (`scrapely:<value>`) the dataset must carry.

```bash
curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to BigQuery",
    "notification": {
      "channel": "bigquery",
      "table": "my-project.scraping.results",
      "maxRows": 1000
    }
  }'
```

Responses carry `notification.bigquerySettings` with `table` and `maxRows`. In an automation, use a delivery with `"channel": "bigquery"` and the same fields.

---

- Previous: [Postgres](https://scrape.ly/docs/postgres.md)
- Next: [Snowflake](https://scrape.ly/docs/snowflake.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)