BigQuery

Adds the dataset items of each successful run as rows to a BigQuery table. Scrape.ly writes as its own Google service account, which you give access to one dataset: no key of yours is stored, and you can take the access back at any time.

Give Scrape.ly access to a dataset

1. In the automation form, under Give this account access, copy the service account email.
2. In the BigQuery console, open the dataset's sharing permissions and add that email as a principal with the BigQuery Data Editor role.

Scrape.ly can then add rows to the tables of that dataset and create new tables in it; it can't see or change your other datasets. To take the access back, remove the email from the dataset's permissions.

With the bq tool, the same grant is:

bq add-iam-policy-binding \
  --member='serviceAccount:<the email from the form>' \
  --role='roles/bigquery.dataEditor' \
  my-project:scraping

Link the dataset to your account

Scrape.ly only writes to a dataset that carries your account's label, so nobody else can send rows into a dataset you shared.

1. In the automation form, under Link the dataset to your account, copy the label (it looks like scrapely:1a2b3c4d5e6f7a8b).
2. In the BigQuery console, open the dataset's details, click Edit details, and add the label under Labels (key scrapely, value the part after the colon).

With the bq tool:

bq update --set_label scrapely:<the value from the form> my-project:scraping

Only the dataset's owner can set labels; the BigQuery Data Editor role Scrape.ly gets can't change them.

Send results to a table

1. Click New automation (in Automations, or on an Actor's Automations tab), click Or create manually and, under Send, click BigQuery.
2. Enter the Table as project.dataset.table, for example my-project.scraping.results.
3. Optionally set Max rows per run (1 to 100,000, default 1,000).
4. Use Save and send test. The test checks that the dataset is shared and carries your label and, when the table exists, that Scrape.ly may add rows to it. It writes nothing.

New tables: if the table doesn't exist yet, the first run creates it, and BigQuery picks the column types from those results. To choose the types yourself, create the table first.

Columns: a result field is written to the column with the same name (upper or lower case doesn't matter). Characters other than letters, digits, and _ in field names become _, so price.amount as a field name goes into price_amount. Nested objects go into RECORD columns, and lists into REPEATED columns; null values inside lists are left out. Any value for a STRING column is written as text, objects as JSON. Fields without a column are skipped, and the delivery history lists them. A row BigQuery can't take, such as text for a NUMERIC column, is skipped and counted; the other rows are still written.

When it writes: only for succeeded runs, and only when the run returned items. Each run is one BigQuery load job, so its rows appear together. Load jobs are free; you pay BigQuery only for storing the rows. Each run adds new rows; existing rows are never changed. An export that takes longer than 8 minutes is stopped; for very large runs, lower Max rows per run.

Limits: each BigQuery automation of this kind writes at most 30 times an hour, and an account at most 500 exports a day (Google Sheets, S3, GitHub, Airtable, HubSpot, Salesforce, Notion, Dropbox, OneDrive, Postgres, Azure Blob, BigQuery, OpenAI Vector Store, and Snowflake together). If the dataset isn't shared or doesn't carry your label, the table can't be written, BigQuery refuses every row, or none of the fields match a column, the delivery history on the automation's page says so.

API

A webhook with a bigquery notification. Only ACTOR.RUN.SUCCEEDED is allowed. GET https://api.scrape.ly/v2/integrations/bigquery returns the serviceAccountEmail to share the dataset with and the datasetLabel (scrapely:<value>) the dataset must carry.

curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to BigQuery",
    "notification": {
      "channel": "bigquery",
      "table": "my-project.scraping.results",
      "maxRows": 1000
    }
  }'

Responses carry notification.bigquerySettings with table and maxRows. In an automation, use a delivery with "channel": "bigquery" and the same fields.