---
title: "MongoDB · Scrape.ly Docs"
description: "Add or upsert the results of each successful run as documents in your own MongoDB collection."
url: https://scrape.ly/docs/mongodb
markdown: https://scrape.ly/docs/mongodb.md
---

# MongoDB

> Add or upsert the results of each successful run as documents in your own MongoDB collection.

Adds the dataset items of each successful run as documents to a collection in your own MongoDB database: MongoDB Atlas, or any other MongoDB server reachable from the internet. Scrape.ly connects with a connection string you paste. It is stored encrypted and never shown again.

## Prepare MongoDB Atlas

In [MongoDB Atlas](https://cloud.mongodb.com):

1. Under **Database & Network Access**, open the **Database Users** tab and click **Add New Database User**. Choose **Password** as the authentication method and set a user name and password. Expand **Specific Privileges** and give the user `readWrite` on your database only, then click **Add User**.
2. Under **Network Access**, open **IP Access List** and click **Add IP Address**. Scrape.ly has no fixed IP address, so enter `0.0.0.0/0` (access from anywhere) and click **Confirm**; the user and password still protect the database. Atlas emails the project's users when this entry is added.
3. Under **Clusters**, click **Connect** on your cluster, pick **Drivers**, and copy the connection string under **Use this connection string in your application**. Put in the user's password, and add the database name after the host, before `?`: `mongodb+srv://scrapely:password@cluster0.ab1cd.mongodb.net/app?retryWrites=true&w=majority`.

A self-hosted server needs TLS and a public host name.

## Send results to a collection

1. Click **New automation** (in **Automations**, or on an Actor's **Automations** tab) and pick **Send results to MongoDB**, or click **Or create manually** and, under **Send**, click **MongoDB**.
2. Paste the **Connection string**. It must name the database, such as `mongodb+srv://scrapely:password@cluster0.ab1cd.mongodb.net/app`.
3. Enter the **Collection**, such as `results`. If it doesn't exist yet, MongoDB creates it on the first write.
4. Choose the **Write mode**: **Add new documents**, or **Update existing documents (upsert)** with a **Key field**.
5. Optionally set **Max documents per run** (1 to 100,000, default 1,000).
6. Use **Save and send test**. The test connects, checks that the user may write to the collection, and writes nothing.

**Connection**: Scrape.ly connects from the internet, so the database must accept outside connections. The connection always uses TLS, and `tls=false` is refused. Both `mongodb+srv://` and `mongodb://` connection strings work. Connection strings with a host name are accepted; IP addresses, `localhost`, and private network addresses are not.

**Documents**: each result item becomes one document, with nested objects and lists kept as they are. MongoDB doesn't allow field names that start with `$` or contain `.`, so those characters are replaced with `_`: `price.amount` as a field name becomes `price_amount`.

**Upsert**: with **Update existing documents**, a document whose key field already holds the item's value is updated instead of added: the item's fields are set, and the document's other fields keep their values. When no document matches, a new one is added. The key field needs no unique index, but one makes upserts faster on large collections. Items without a value for the key field are skipped and counted. When one run has several items with the same key, the last one wins.

**When it writes**: only for succeeded runs, and only when the run returned items. Documents go in up to 500 at a time, so a failure partway leaves the earlier documents written, and the delivery history says how many were written. A document MongoDB refuses is skipped and counted; the other documents are still written.

**Limits**: each MongoDB automation of this kind writes at most 30 times an hour, and an account at most 500 exports a day (Google Sheets, S3, GitHub, Airtable, HubSpot, Salesforce, Notion, Dropbox, OneDrive, Postgres, MySQL, MongoDB, SQL Server, ClickHouse, Azure Blob, BigQuery, Snowflake, and OpenAI Vector Store together). A write that takes longer than 60 seconds is stopped, and an export stops after 8 minutes. If the database can't be reached, the login is refused, or the user may not write to the collection, the delivery history on the automation's page says so.

## API

A webhook with a `mongodb` notification. Only `ACTOR.RUN.SUCCEEDED` is allowed.

```bash
curl -X POST https://api.scrape.ly/v2/webhooks \
  -H "Authorization: Bearer $SCRAPELY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
    "condition": { "actorId": "YOUR_ACTOR_ID" },
    "description": "Results to MongoDB",
    "notification": {
      "channel": "mongodb",
      "connectionString": "mongodb+srv://scrapely:password@cluster0.ab1cd.mongodb.net/app",
      "collection": "results",
      "mode": "upsert",
      "keyField": "url",
      "maxRows": 1000
    }
  }'
```

`mode` is `insert` (default) or `upsert`; `keyField` is only used with `upsert`. Responses carry `notification.mongodbSettings` with `collection`, `mode`, `keyField`, `maxRows`, and a `connectionHint` (`user@host/database`); the connection string is never returned. On `PUT`, leaving out `connectionString` keeps the saved one. In an automation, use a delivery with `"channel": "mongodb"` and the same fields.

---

- Previous: [MySQL](https://scrape.ly/docs/mysql.md)
- Next: [SQL Server](https://scrape.ly/docs/sqlserver.md)
- All docs: [Docs index](https://scrape.ly/docs.md), [llms.txt](https://scrape.ly/llms.txt)