Postgres

Adds the dataset items of each successful run as rows to a table in your own Postgres database: Supabase, Neon, Amazon RDS, or any other Postgres server reachable from the internet. Scrape.ly connects with a connection string you paste. It is stored encrypted and never shown again.

Prepare the table

Scrape.ly writes into a table that already exists; it never creates or changes tables. Create one with a column for each result field you want, and give Scrape.ly a database user that may only write to it:

CREATE TABLE public.results (
  url        text PRIMARY KEY,
  title      text,
  price      numeric,
  tags       jsonb,
  scraped_at timestamptz DEFAULT now()
);

CREATE USER scrapely WITH PASSWORD 'a-long-random-password';
GRANT USAGE ON SCHEMA public TO scrapely;
GRANT INSERT, UPDATE ON public.results TO scrapely;

UPDATE is only needed for Update existing rows. On Supabase, run this in the SQL editor, then use the connection string for the user you created.

Send results to a table

1. Click New automation (in Automations, or on an Actor's Automations tab) and pick Send results to Postgres, or click Or create manually and, under Send, click Postgres.
2. Paste the Connection string, such as postgres://scrapely:password@db.example.com:5432/app.
3. Enter the Table: results, or schema.results for a schema other than public.
4. Choose the Write mode: Add new rows, or Update existing rows (upsert) with a Key column.
5. Optionally set Max rows per run (1 to 100,000, default 1,000).
6. Use Save and send test. The test connects, checks that the table exists and that the user may write to it, and writes nothing.

Connection: Scrape.ly connects from the internet, so the database must accept outside connections. The connection always uses TLS. The server's certificate is checked only when the connection string ends in ?sslmode=verify-full, because many hosted databases use certificates that only their own tools trust. Connection strings with a host name are accepted; IP addresses, localhost, and private network addresses are not. On Supabase, use the Connection pooler string from Connect (session or transaction mode both work); the direct connection host only has an IPv6 address.

Columns: a result field is written to the column with the same name (upper or lower case doesn't matter). A nested field whose parent has no column of its own goes into the column named parent and child joined with _, so address.city goes into address_city. Objects and lists go into json or jsonb columns as JSON, and lists into array columns as arrays. Fields without a column are skipped, and the delivery history lists them. Columns without a field get their default value. Generated columns and GENERATED ALWAYS identity columns are never written. Postgres converts each value to the column's type; a value it can't convert, such as abc for an integer column, skips that row.

Upsert: with Update existing rows, a row whose key column already holds the item's value is updated instead of added (INSERT … ON CONFLICT … DO UPDATE). The key column needs a primary key, a unique constraint, or a unique index of its own. Only the columns the item has a field for are updated; the others keep their values. Items without a value for the key column are skipped and counted. When one run has several items with the same key, the last one wins.

When it writes: only for succeeded runs, and only when the run returned items. Rows go in up to 500 at a time, each batch in its own transaction, so a failure partway leaves the earlier batches written and the delivery history says how many rows are in the table. A row Postgres refuses is skipped and counted; the other rows are still written.

Limits: each Postgres automation of this kind writes at most 30 times an hour, and an account at most 500 exports a day (Google Sheets, S3, GitHub, Airtable, HubSpot, Salesforce, Notion, Dropbox, OneDrive, Postgres, Azure Blob, BigQuery, OpenAI Vector Store, and Snowflake together). A statement that takes longer than 60 seconds is stopped. If the database can't be reached, the login is refused, the table doesn't exist, the user may not write to it, or none of the fields match a column, the delivery history on the automation's page says so.

API

A webhook with a postgres notification. Only ACTOR.RUN.SUCCEEDED is allowed.

curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to Postgres",
    "notification": {
      "channel": "postgres",
      "connectionString": "postgres://scrapely:password@db.example.com:5432/app",
      "table": "public.results",
      "mode": "upsert",
      "keyColumn": "url",
      "maxRows": 1000
    }
  }'

mode is insert (default) or upsert; keyColumn is only used with upsert. Responses carry notification.postgresSettings with table, mode, keyColumn, maxRows, and a connectionHint (user@host:port/database); the connection string is never returned. On PUT, leaving out connectionString keeps the saved one. In an automation, use a delivery with "channel": "postgres" and the same fields.