Amazon S3 & R2

Uploads the dataset of each successful run as one file to your own bucket: AWS S3, Cloudflare R2, or any other S3-compatible storage. Scrape.ly uses access keys you create, stored encrypted and never shown again.

Set it up with the Upload results to Amazon S3 / R2 recipe in New automation, or add Amazon S3 / R2 as a delivery in the automation editor to upload the results of several Actors, after filters and an AI step.

1. Click New automation (in Automations, or on an Actor's Automations tab) and pick Upload results to Amazon S3 / R2.
2. Choose the Storage: AWS S3, Cloudflare R2, or Other S3-compatible.
3. Enter the Bucket and, for AWS S3 or other storage, its Region. For Cloudflare R2 or other storage, enter the Endpoint, for example https://<account-id>.r2.cloudflarestorage.com.
4. Enter an Access key ID and Secret access key. Use keys that can only upload to this bucket (see the policy below).
5. Optionally set a Folder, the File format (JSON, JSONL or CSV), and Max items per run (empty uploads every item).
6. Use Save and send test. A scrapely-test.txt file appears in the folder.

File names: <folder>/<actor-name>/<run-id>.<format>, for example exports/my-actor/a1b2c3d4.csv. Each run gets its own file. The file has the same content as downloading the dataset in that format, except that CSV files have no byte-order mark; CSV cells that start with =, +, - or @ are prefixed with ' so spreadsheets don't run them as formulas.

When it uploads: only for succeeded runs, and only when the run returned items. Large datasets are uploaded in parts. An upload that takes longer than 8 minutes is stopped; for very large runs, set Max items per run.

Keys: an AWS IAM policy that allows only uploads into one folder (s3:AbortMultipartUpload lets Scrape.ly clean up a large upload that fails partway):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:AbortMultipartUpload"],
      "Resource": "arn:aws:s3:::my-bucket/exports/*"
    }
  ]
}

On Cloudflare R2, create an API token under R2 → Manage API tokens with Object Read & Write for the bucket, and use its access key ID and secret. To change the keys later, enter new ones on the integration's page; leaving the fields empty keeps the saved keys.

Limits: each S3 integration uploads at most 30 times an hour, and an account at most 500 exports a day (S3, Google Sheets, and GitHub together). If the bucket doesn't exist, the keys are wrong, or they can't write to the bucket, the delivery log on the integration's page says so.

API: a webhook with an s3 notification. Only ACTOR.RUN.SUCCEEDED is allowed.

curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to R2",
    "notification": {
      "channel": "s3",
      "bucket": "my-bucket",
      "endpoint": "https://<account-id>.r2.cloudflarestorage.com",
      "region": "auto",
      "prefix": "exports",
      "format": "jsonl",
      "accessKeyId": "…",
      "secretAccessKey": "…"
    }
  }'

Leave out endpoint for AWS and set region to the bucket's region. maxItems is optional; leave it out or set it to null to upload every item. Responses carry notification.bucketSettings with the bucket, its settings and a masked accessKeyHint; the keys are never returned. On PUT, leaving out both keys keeps the saved ones.