--- title: "ClickHouse · Scrape.ly Docs" description: "Add the results of each successful run as rows to your own ClickHouse table." url: https://scrape.ly/docs/clickhouse markdown: https://scrape.ly/docs/clickhouse.md --- # ClickHouse > Add the results of each successful run as rows to your own ClickHouse table. Adds the dataset items of each successful run as rows to a table in your own ClickHouse database: ClickHouse Cloud, or any other ClickHouse server reachable over HTTPS from the internet. Scrape.ly connects with a connection string you paste. It is stored encrypted and never shown again. ## Prepare the table Scrape.ly writes into a table that already exists; it never creates or changes tables. Create one with a column for each result field you want, and give Scrape.ly a user that may only write to it. On ClickHouse Cloud, run this in the service's **SQL console**: ```sql CREATE TABLE results ( url String, title String, price Nullable(Float64), tags Array(String), scraped_at DateTime DEFAULT now() ) ENGINE = MergeTree ORDER BY url; CREATE USER scrapely IDENTIFIED BY 'Long-Random-Passw0rd-42'; GRANT INSERT, SELECT ON default.results TO scrapely; ``` `SELECT` lets Scrape.ly read the table's column list; it never reads your rows. ## Find the connection string On ClickHouse Cloud, select your service, click **Connect**, and choose **HTTPS**. The example `curl` command shows the host and port, usually 8443. Put them together with the user and password, and the database after the port: `https://scrapely:Long-Random-Passw0rd-42@abc123.eu-central-1.aws.clickhouse.cloud:8443/default` The port defaults to 8443 and the database to `default` when left out. A self-hosted server needs its HTTPS port enabled (`https_port`) with a valid certificate and a public host name. ## Send results to a table 1. Click **New automation** (in **Automations**, or on an Actor's **Automations** tab) and pick **Send results to ClickHouse**, or click **Or create manually** and, under **Send**, click **ClickHouse**. 2. Paste the **Connection string**. 3. Enter the **Table**, such as `results`. It must be in the connection string's database. 4. Optionally set **Max rows per run** (1 to 100,000, default 1,000). 5. Use **Save and send test**. The test signs in, checks that the table exists and that the user may insert into it, and writes nothing. **Connection**: Scrape.ly connects from the internet over HTTPS and always checks the server's certificate. New ClickHouse Cloud services accept connections from anywhere. If you restricted yours, Scrape.ly has no fixed IP address: under the service's **Settings**, find the IP access list, click **Add IPs**, choose **Allow incoming traffic from anywhere to the service**, and click **Save**. The user and password still protect the database. The user and password are sent in request headers, never in the URL. Connection strings with a host name are accepted; IP addresses, `localhost`, and private network addresses are not. **Columns**: a result field is written to the column with the same name (upper or lower case doesn't matter). A nested field whose parent has no column of its own goes into the column named parent and child joined with `_`, so `address.city` goes into `address_city`. Objects and lists go into `String` columns as JSON; lists go into `Array` columns as they are. Dates in common formats such as `2026-10-30T14:05:00Z` are read into `DateTime` columns. Fields without a column are skipped, and the delivery history lists them. Columns without a field get their default value. `MATERIALIZED` and `ALIAS` columns are never written. **Rows are only added**: ClickHouse has no update in place, so every run adds new rows. To keep only the latest row per key, create the table with `ENGINE = ReplacingMergeTree(scraped_at)` and `ORDER BY` the key (such as `url`); ClickHouse then merges duplicates in the background, and `SELECT … FINAL` shows one row per key. **When it writes**: only for succeeded runs, and only when the run returned items. Rows go in up to 10,000 at a time, each insert all or nothing, so a failure partway leaves the earlier inserts written and the delivery history says how many rows are in the table. When ClickHouse refuses an insert because of a value, such as `abc` for a `Float64` column, the insert is split until the refused rows are found; the other rows are still written and the refused ones are counted. After more than 50 refused rows the export stops, since the columns' types likely don't fit the results. An insert is never sent twice after a dropped connection, so rows aren't added twice. **Limits**: each ClickHouse automation of this kind writes at most 30 times an hour, and an account at most 500 exports a day (Google Sheets, S3, GitHub, Airtable, HubSpot, Salesforce, Notion, Dropbox, OneDrive, Postgres, MySQL, MongoDB, SQL Server, ClickHouse, Azure Blob, BigQuery, Snowflake, and OpenAI Vector Store together). A request that takes longer than 80 seconds is stopped, and an export stops after 8 minutes. If the server can't be reached, the login is refused, the table doesn't exist, the user may not insert into it, or none of the fields match a column, the delivery history on the automation's page says so. ## API A webhook with a `clickhouse` notification. Only `ACTOR.RUN.SUCCEEDED` is allowed. ```bash curl -X POST https://api.scrape.ly/v2/webhooks \ -H "Authorization: Bearer $SCRAPELY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "eventTypes": ["ACTOR.RUN.SUCCEEDED"], "condition": { "actorId": "YOUR_ACTOR_ID" }, "description": "Results to ClickHouse", "notification": { "channel": "clickhouse", "connectionString": "https://scrapely:password@abc123.eu-central-1.aws.clickhouse.cloud:8443/default", "table": "results", "maxRows": 1000 } }' ``` Responses carry `notification.clickhouseSettings` with `table`, `maxRows`, and a `connectionHint` (`user@host:port/database`); the connection string is never returned. On `PUT`, leaving out `connectionString` keeps the saved one. In an automation, use a delivery with `"channel": "clickhouse"` and the same fields. --- - Previous: [SQL Server](https://scrape.ly/docs/sqlserver.md) - Next: [BigQuery](https://scrape.ly/docs/bigquery.md) - All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)