MongoDB

Adds the dataset items of each successful run as documents to a collection in your own MongoDB database: MongoDB Atlas, or any other MongoDB server reachable from the internet. Scrape.ly connects with a connection string you paste. It is stored encrypted and never shown again.

Prepare MongoDB Atlas

In MongoDB Atlas:

1. Under Database & Network Access, open the Database Users tab and click Add New Database User. Choose Password as the authentication method and set a user name and password. Expand Specific Privileges and give the user readWrite on your database only, then click Add User.
2. Under Network Access, open IP Access List and click Add IP Address. Scrape.ly has no fixed IP address, so enter 0.0.0.0/0 (access from anywhere) and click Confirm; the user and password still protect the database. Atlas emails the project's users when this entry is added.
3. Under Clusters, click Connect on your cluster, pick Drivers, and copy the connection string under Use this connection string in your application. Put in the user's password, and add the database name after the host, before ?: mongodb+srv://scrapely:password@cluster0.ab1cd.mongodb.net/app?retryWrites=true&w=majority.

A self-hosted server needs TLS and a public host name.

Send results to a collection

1. Click New automation (in Automations, or on an Actor's Automations tab) and pick Send results to MongoDB, or click Or create manually and, under Send, click MongoDB.
2. Paste the Connection string. It must name the database, such as mongodb+srv://scrapely:password@cluster0.ab1cd.mongodb.net/app.
3. Enter the Collection, such as results. If it doesn't exist yet, MongoDB creates it on the first write.
4. Choose the Write mode: Add new documents, or Update existing documents (upsert) with a Key field.
5. Optionally set Max documents per run (1 to 100,000, default 1,000).
6. Use Save and send test. The test connects, checks that the user may write to the collection, and writes nothing.

Connection: Scrape.ly connects from the internet, so the database must accept outside connections. The connection always uses TLS, and tls=false is refused. Both mongodb+srv:// and mongodb:// connection strings work. Connection strings with a host name are accepted; IP addresses, localhost, and private network addresses are not.

Documents: each result item becomes one document, with nested objects and lists kept as they are. MongoDB doesn't allow field names that start with $ or contain ., so those characters are replaced with _: price.amount as a field name becomes price_amount.

Upsert: with Update existing documents, a document whose key field already holds the item's value is updated instead of added: the item's fields are set, and the document's other fields keep their values. When no document matches, a new one is added. The key field needs no unique index, but one makes upserts faster on large collections. Items without a value for the key field are skipped and counted. When one run has several items with the same key, the last one wins.

When it writes: only for succeeded runs, and only when the run returned items. Documents go in up to 500 at a time, so a failure partway leaves the earlier documents written, and the delivery history says how many were written. A document MongoDB refuses is skipped and counted; the other documents are still written.

Limits: each MongoDB automation of this kind writes at most 30 times an hour, and an account at most 500 exports a day (Google Sheets, S3, GitHub, Airtable, HubSpot, Salesforce, Notion, Dropbox, OneDrive, Postgres, MySQL, MongoDB, SQL Server, ClickHouse, Azure Blob, BigQuery, Snowflake, and OpenAI Vector Store together). A write that takes longer than 60 seconds is stopped, and an export stops after 8 minutes. If the database can't be reached, the login is refused, or the user may not write to the collection, the delivery history on the automation's page says so.

API

A webhook with a mongodb notification. Only ACTOR.RUN.SUCCEEDED is allowed.

curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to MongoDB",
    "notification": {
      "channel": "mongodb",
      "connectionString": "mongodb+srv://scrapely:password@cluster0.ab1cd.mongodb.net/app",
      "collection": "results",
      "mode": "upsert",
      "keyField": "url",
      "maxRows": 1000
    }
  }'

mode is insert (default) or upsert; keyField is only used with upsert. Responses carry notification.mongodbSettings with collection, mode, keyField, maxRows, and a connectionHint (user@host/database); the connection string is never returned. On PUT, leaving out connectionString keeps the saved one. In an automation, use a delivery with "channel": "mongodb" and the same fields.