Skip to content

Dataset Merge & Dedupe

Try for free

Merge any number of datasets, JSON or CSV links and pasted items into one clean dataset and remove duplicates by the fields you choose. Keep only unique items, list the duplicates, or just count them. Streams millions of items in order.

🧩 Dataset Merge & Dedupe

Merge any number of datasets, JSON or CSV files and pasted items into one clean dataset, and remove duplicates by the fields you choose. Items stream through in their original order: two 10,000-item datasets merge in about 8 seconds and 100,000 items in about 25. No code and no setup: $0.0042 per 1,000 saved items plus $0.00034 per 1,000 items read on the Free plan (less on higher plans), with platform usage included.

✨ Why use Dataset Merge & Dedupe?

  • 🔗 Merge everything into one — combine the datasets of many scraper runs, JSON / JSON Lines / CSV links and pasted items in a single run, in the order you list them.
  • 🧹 Remove duplicates your way — compare one field (url), several fields together (name + city), nested fields (seller.id) or whole items (*), optionally ignoring letter case and extra spaces.
  • 🆕 Keep only what is new — exclude every item that is already in earlier datasets, so each run gives you only the listings, leads or products you have not seen before.
  • 🔍 See the duplicates or just count them — save the unique items, only the duplicates that were dropped, or a single summary row with the counts.
  • ⚡ Fast and light on memory — datasets are read page by page with read-ahead and rows are saved while the next page loads; only a small fingerprint per unique key stays in memory.
  • ✂️ Trim the output — keep or drop fields (nested ones too) and tag each item with the dataset or link it came from.

🎯 Use cases

  • 🛒 E-commerce monitoring — merge the product datasets of 20 category runs into one catalogue with each product URL once.
  • 🧲 Lead generation — combine leads from several directories and searches, dedupe them by email or by company name + city, and drop the ones already in your CRM export.
  • 🏠 Listings and jobs — run a scraper every day and keep only the listings that were not in any earlier day's dataset.
  • 📊 Data quality checks — count how many duplicates a scraper produces, or list them to see why.
  • 🔄 Automations — start it after each scraper run with the run's dataset and send the clean result on to Google Sheets, a webhook or your database.

🚀 How to use

  1. Pick the datasets to merge in Datasets (or paste their IDs), add JSON or CSV links, or paste items as JSON.
  2. Enter the Unique key fields that identify an item, for example url. Leave it empty to merge without removing anything.
  3. Choose the Output: unique items, duplicate items only, or a summary.
  4. Optional: exclude items already in other datasets, keep or drop fields, add the source of each item, or read only part of the data with Skip first items and Max items to read.
  5. Click Start and download the merged dataset as JSON, CSV, Excel or XML, or read it through the API.

📥 Input

FieldWhat it does
datasetIdsDatasets to merge, read in this order
dataUrlsPublic JSON, JSON Lines or CSV links, read after the datasets
itemsItems pasted as a JSON array, read last
fieldsUnique key fields; * compares whole items; empty merges without deduplication
outputunique-items (default), duplicate-items or summary
ignoreCaseAndSpacesCompare key values without letter case and extra spaces
nullAsUniqueKeep items whose key field is missing or null instead of comparing them
excludeDatasetIds, excludeDataUrls, excludeItemsItems whose key is already here are not saved
fieldsToKeep, fieldsToOmitFields to keep or drop in the saved items
appendSourceAdd _source with the dataset ID, link or input
offset, limitSkip the first items and stop after a number of items, counted across all sources
{
  "datasetIds": ["<dataset ID of the first run>", "<dataset ID of the second run>"],
  "fields": ["url"],
  "output": "unique-items",
  "excludeDatasetIds": ["<dataset ID of last week's merged result>"],
  "fieldsToOmit": ["description"],
  "appendSource": true
}

To merge right after a scraper finishes, start this actor from an automation with {"datasetIds": ["{{resource.defaultDatasetId}}"], "fields": ["url"]}.

📤 Output

The dataset gets your own items, unchanged except for the fields you keep or drop and the optional _source field. Unique items are the first copy of each key, in source order. A run with the example input (seven products from two stores, two of them listed twice) saves five items:

[
  {"url": "https://shop.example.com/p/desk-lamp", "title": "Desk lamp", "price": 29.9, "category": "Lighting", "store": "North"},
  {"url": "https://shop.example.com/p/office-chair", "title": "Office chair", "price": 149.0, "category": "Furniture", "store": "North"}
]

With Summary only the dataset gets one row with the counts, and every run also stores the same counts in the OUTPUT record of its key-value store:

FieldMeaning
itemsReadItems read from all sources (after Skip first items / Max items to read)
uniqueItemsItems kept as unique
duplicateItemsItems whose key appeared earlier in this run
excludedItemsItems whose key is in the exclude sources
duplicateShareShare of items read that were duplicates (0 to 1)
itemsWithoutKeyItems that have none of the key fields (always kept)
itemsWithEmptyKeyItems kept because a key field was empty and Keep items with an empty key field is on
invalidItemsSkippedEntries that were not JSON objects
keyFields, sourcesThe key fields used and how many items each source gave
{
  "itemsRead": 7, "uniqueItems": 5, "duplicateItems": 2, "excludedItems": 0,
  "itemsWithoutKey": 0, "itemsWithEmptyKey": 0, "invalidItemsSkipped": 0,
  "duplicateShare": 0.2857, "keyFields": ["url"],
  "sources": [{"source": "items from input", "items": 7}]
}

💰 Pricing

Pay per event, platform usage included:

EventFreeStarterScaleBusiness
Item saved to the output (per 1,000)$0.0042$0.0042$0.0038$0.0034
Item read from your data (per 1,000)$0.00034$0.00034$0.0003$0.00027
Run start$0.00013$0.00013$0.0001$0.000085

Example on the Free plan: merging two datasets of 10,000 items each into 15,000 unique items costs 15,000 × $0.0000042 + 20,000 × $0.00000034 + $0.00013 ≈ $0.07. Counting the duplicates of 1,000,000 items with Summary only costs about $0.34. You can cap any run with a maximum charge; the run then stops cleanly and keeps everything saved so far.

❓ FAQ

Which datasets can I merge? Your own: datasets of your runs and datasets you created. Pick them with the dataset picker or paste their IDs (Storage tab of a run, or defaultDatasetId from the API). Other people's datasets need a public link to their export.

How are duplicates decided? Two items are duplicates when all key fields have the same value. Numbers like 1 and 1.0 are equal; objects and lists are compared by their whole content. The first copy is kept. With Ignore letter case and extra spaces, "ACME Ltd " and "acme ltd" match.

What happens to items without the key fields? Items that have none of the key fields are always kept and counted in itemsWithoutKey, so a typo in a field name never throws your data away (the log warns when no item has the fields). Items that have some key fields but an empty one are compared like any other value, unless you turn on Keep items with an empty key field.

Is there a limit on the number of items? No. Datasets are streamed page by page. Memory grows only with the number of unique keys (about 70 bytes each): the default 256 MB holds about 1.5 million unique keys; give the run 1 GB for about 10 million, and more beyond that.

Can I run it on a schedule or from the API? Yes. Start it from an automation after a scraper run, on a schedule, or through the API with the same input JSON.

Is it legal? It only processes data you already have. You are responsible for how you use the data you scraped.

⚠️ Limits

  • No custom code: shaping the output is limited to keeping or dropping fields and adding the source. Transform the data in a later step if you need more.
  • Datasets are read by ID and must be your own (a dataset name is not enough); links must be public.
  • JSON and CSV links are downloaded whole before they are processed, so very large files (several hundred MB) need more memory than the default.
  • The result always goes to this run's dataset; copy or export it from there.
  • Rows keep the order of the sources; a different order (for example by date) needs a sort in a later step.

Related actors

//From the blog\\

Latest articles

All articles
G2
4.7/5
Excellent
Trustpilot

The internet is full of data.
Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes.

Get Started FreeChat with Support
No credit card. No setup calls.