MySQL

Adds the dataset items of each successful run as rows to a table in your own MySQL database: PlanetScale, Amazon RDS or Aurora MySQL, Google Cloud SQL, Azure Database for MySQL, DigitalOcean, or any other MySQL server reachable from the internet. Scrape.ly connects with a connection string you paste. It is stored encrypted and never shown again.

Prepare the table

Scrape.ly writes into a table that already exists; it never creates or changes tables. Create one with a column for each result field you want, and give Scrape.ly a database user that may only write to it:

CREATE TABLE app.results (
  url        VARCHAR(768) PRIMARY KEY,
  title      TEXT,
  price      DECIMAL(10,2),
  tags       JSON,
  scraped_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE USER 'scrapely'@'%' IDENTIFIED BY 'a-long-random-password';
GRANT INSERT, UPDATE ON app.results TO 'scrapely'@'%';

UPDATE is only needed for Update existing rows.

Send results to a table

1. Click New automation (in Automations, or on an Actor's Automations tab) and pick Send results to MySQL, or click Or create manually and, under Send, click MySQL.
2. Paste the Connection string, such as mysql://scrapely:password@db.example.com:3306/app.
3. Enter the Table: results, a table in the connection string's database.
4. Choose the Write mode: Add new rows, or Update existing rows (upsert) with a Key column.
5. Optionally set Max rows per run (1 to 100,000, default 1,000).
6. Use Save and send test. The test connects, checks that the table exists and that the user may write to it, and writes nothing.

Connection: Scrape.ly connects from the internet, so the database must accept outside connections. The connection always uses TLS, and ssl-mode=DISABLED is refused. The server's certificate is checked only when the connection string has ssl-mode=VERIFY_IDENTITY or ssl-mode=VERIFY_CA, because many hosted databases use certificates that only their own tools trust. Connection strings with a host name are accepted; IP addresses, localhost, and private network addresses are not.

Columns: a result field is written to the column with the same name (upper or lower case doesn't matter). A nested field whose parent has no column of its own goes into the column named parent and child joined with _, so address.city goes into address_city. Objects and lists go into JSON columns as JSON. Fields without a column are skipped, and the delivery history lists them. Columns without a field get their default value. Generated columns are never written. A row MySQL refuses, such as abc for an INT column, is skipped.

Upsert: with Update existing rows, a row whose key column already holds the item's value is updated instead of added (INSERT … ON DUPLICATE KEY UPDATE). The key column needs a primary key or a unique index of its own. MySQL also matches on any other unique index of the table, so a row that repeats another unique value is updated too. Only the columns the item has a field for are updated; the others keep their values. Items without a value for the key column are skipped and counted. When one run has several items with the same key, the last one wins.

When it writes: only for succeeded runs, and only when the run returned items. Rows go in up to 500 at a time, each batch in its own transaction, so a failure partway leaves the earlier batches written and the delivery history says how many rows are in the table. A row MySQL refuses is skipped and counted; the other rows are still written.

Limits: each MySQL automation of this kind writes at most 30 times an hour, and an account at most 500 exports a day (Google Sheets, S3, GitHub, Airtable, HubSpot, Salesforce, Notion, Dropbox, OneDrive, Postgres, MySQL, MongoDB, SQL Server, ClickHouse, Azure Blob, BigQuery, Snowflake, and OpenAI Vector Store together). A statement that takes longer than 60 seconds is stopped. If the database can't be reached, the login is refused, the table doesn't exist, the user may not write to it, or none of the fields match a column, the delivery history on the automation's page says so.

API

A webhook with a mysql notification. Only ACTOR.RUN.SUCCEEDED is allowed.

curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to MySQL",
    "notification": {
      "channel": "mysql",
      "connectionString": "mysql://scrapely:password@db.example.com:3306/app",
      "table": "results",
      "mode": "upsert",
      "keyColumn": "url",
      "maxRows": 1000
    }
  }'

mode is insert (default) or upsert; keyColumn is only used with upsert. Responses carry notification.mysqlSettings with table, mode, keyColumn, maxRows, and a connectionHint (user@host:port/database); the connection string is never returned. On PUT, leaving out connectionString keeps the saved one. In an automation, use a delivery with "channel": "mysql" and the same fields.