Skip to content

Duplicate Image Finder & Image Similarity

Try for free

Find duplicate and near-duplicate images in a list of links or a dataset, or compare images with your reference images. Perceptual hashes (pHash, dHash) catch resized and recompressed copies; a vision model scores visual similarity. Duplicate groups, the copy to keep, and a verdict per image.

🖼️ Duplicate Image Finder & Image Similarity

Find duplicate and near-duplicate images in a list of links or in a scraper's dataset, or compare a batch of images with your own reference images. Every image gets perceptual hashes (pHash and dHash) that prove whether two files are the same picture, a visual similarity score from a vision model, its duplicate group and the copy worth keeping. No login, no setup, and $0.425 per 1,000 results with platform usage included.

✨ Why use Duplicate Image Finder?

  • 🧬 Two signals, one row: a vision-model similarity score says what the images show; perceptual hash distances say whether they are literally the same picture, even after resizing, recompression or a format change.
  • 🗂️ Duplicate groups out of the box: give it a list and get groups, the copy to keep (the largest one) and, for every other copy, which image it duplicates.
  • 🎯 Reference mode: compare hundreds of images with one or several originals (a product master photo, your logo, your portfolio) and get one scored row per pair.
  • ⚡ Fast duplicate-only mode: switch to hashes only to sweep big catalogues without the model.
  • 📦 Works with your scraper results: point it at a dataset and the image links are picked up from the usual fields, with your own ID copied onto every row.
  • 🆓 Failed images are free: links that are broken, blocked or not an image get a clear error row and no charge.

🎯 Use cases

  • 🛒 Clean product catalogues: find the same product photo uploaded under several SKUs, or supplier photos that duplicate your own.
  • 🏷️ Spot relistings and copies: compare marketplace or classified listing photos with your originals to catch re-uploads.
  • 🖼️ Tidy photo libraries: group resized, recompressed and re-exported copies and keep only the best one.
  • 🛡️ Brand and content monitoring: check whether images found online are copies of your assets.
  • 🧪 Visual regression checks: compare new screenshots or renders with an approved baseline.
  • 🤖 Data pipelines and AI agents: deduplicate images before training, indexing or publishing.

🚀 How to use

  1. Paste image links into Images, one per line (or pick a dataset with image links).
  2. To find duplicates within that list, leave Reference images empty. To compare every image with your originals, add them under Reference images.
  3. Optionally choose what to compare (both, hashes only, or visual similarity only) and adjust the duplicate distance or similarity threshold.
  4. Click Start. In reference mode results appear as the images are read; duplicate groups are saved once every image is read.
  5. Download the results as JSON, CSV, Excel or HTML, or read them through the API.

📥 Input

FieldWhat it does
imageUrlsImage links (JPEG, PNG, WebP, GIF, AVIF, HEIC, TIFF, BMP; up to 100 MB each). Through the API also data:image base64 URIs or {"url", "customId", "headers"} objects.
sourceImagesOptional reference images. Empty: duplicates are searched within imageUrls. Set: every image is compared with each reference image.
comparisonModeboth (default), hash (duplicate hashes only, fastest) or semantic (visual similarity only).
duplicateDistanceMost pHash bits (of 64) that may differ for a duplicate. Default 8.
similarityThresholdSimilarity score from which isSimilar is true. Default 0.85.
groupSimilarAlso put visually similar images (score at or above the threshold) in one group.
datasetId, datasetImageField, datasetIdFieldRead image links from one of your datasets, and copy one of its fields into customId.
customId, sourceImageId, headersYour labels on the results, and extra download headers such as a Referer.
proxyConfigurationProxy for the downloads (datacenter by default).
{
  "imageUrls": [
    "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/250px-Cat03.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/2/25/Siam_lilacpoint.jpg/960px-Siam_lilacpoint.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/9/90/Labrador_Retriever_portrait.jpg/960px-Labrador_Retriever_portrait.jpg",
    "https://upload.wikimedia.org/wikipedia/commons/thumb/9/90/Labrador_Retriever_portrait.jpg/500px-Labrador_Retriever_portrait.jpg"
  ],
  "comparisonMode": "both",
  "duplicateDistance": 8,
  "similarityThreshold": 0.85
}

📤 Output

One row per image when duplicates are searched within the list; one row per image and reference image in reference mode.

FieldMeaning
image, index, customIdThe image, its position in the input and your label
width, height, format, bytesUpright pixel size, file format and file size
sha256, phash, dhashFile hash and 64-bit perceptual hashes (hex), to store and match later
groupId, groupSizeDuplicate group (empty when the image has no duplicate) and how many images it holds
keep, duplicateOf, duplicateOfIndextrue for the copy to keep (most pixels, then largest file); other copies name it
comparedWith, comparedWithIndex, comparedWithIdThe reference image, or the most similar other image in the list
similarityScoreVision-model similarity, 0 to 1
phashDistance, dhashDistanceHash distances, 0 to 64 (0 = same picture structure)
isNearDuplicate, isSimilar, isExactDuplicateDuplicate by hash, similar by score, byte-identical file
verdictduplicate, very-similar (score 0.92+), related (0.75+) or different
error, errorCodeWhy an image could not be read (free rows)

Real rows from a run of the example above:

[
  {
    "mode": "find-duplicates",
    "index": 1,
    "image": "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg",
    "width": 1600,
    "height": 1598,
    "format": "jpeg",
    "bytes": 279603,
    "sha256": "ec8ff0a6954a5f54e4ad246f0a788d076f675e457bca16a04248718cca3fe631",
    "phash": "bcca96c0f1f1b1b0",
    "dhash": "1427361f9636363e",
    "groupId": 1,
    "groupSize": 2,
    "keep": true,
    "duplicateOf": null,
    "comparedWith": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/250px-Cat03.jpg",
    "comparedWithIndex": 2,
    "similarityScore": 0.9413,
    "phashDistance": 2,
    "dhashDistance": 0,
    "isSimilar": true,
    "isNearDuplicate": true,
    "isExactDuplicate": false,
    "verdict": "duplicate",
    "comparisonMode": "both",
    "embeddingModel": "clip-vit-b-32"
  },
  {
    "mode": "find-duplicates",
    "index": 2,
    "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/250px-Cat03.jpg",
    "width": 250,
    "height": 250,
    "format": "jpeg",
    "bytes": 16629,
    "phash": "bcc896c0f1f1b9b0",
    "dhash": "1427361f9636363e",
    "groupId": 1,
    "groupSize": 2,
    "keep": false,
    "duplicateOf": "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg",
    "duplicateOfIndex": 1,
    "comparedWithIndex": 1,
    "similarityScore": 0.9413,
    "phashDistance": 2,
    "dhashDistance": 0,
    "verdict": "duplicate"
  }
]

How to read the two signals:

SignalValueMeaning
phashDistance0 to 8The same picture: resized, recompressed, re-exported or lightly edited
phashDistance9 and moreNot the same picture, whatever the similarity score says
similarityScore0.95 and moreNear-identical content
similarityScore0.85 to 0.95Very similar scenes or subjects
similarityScore0.75 to 0.85Related subjects, visibly different images

Two photos of different cats score high on similarity but far apart on hashes (similar, not duplicates); a thumbnail of your photo stays within a few hash bits of the original (a duplicate).

A run summary (images read, duplicate groups, duplicates, errors by code) is saved as OUTPUT in the run's key-value store.

💰 Pricing

You pay per result row: $0.000425 per result on the Free and Starter plans ($0.425 per 1,000), $0.0003825 on Scale and $0.00034 on Business. Platform usage (compute, proxy) is included, so the result price is all you pay.

  • Finding duplicates among 1,000 images: 1,000 results ≈ $0.425.
  • Comparing 500 images with 2 reference images: 1,000 results ≈ $0.425.
  • Error rows for images that cannot be read are free.

❓ FAQ

Is it legal to use? It reads images from links you provide, the way a browser does. Make sure you may use the images you process.

Do I need a login or a proxy? No login. Downloads go through the datacenter proxy by default; switch to the Residential group if an image host refuses it, or add a Referer under Request headers for hosts that block hotlinking.

How many images can I check? As many as you like: there is no limit on the number of images per run. Very large lists simply take longer; finding duplicates compares every image with every other one, which stays fast because it runs on compact fingerprints.

How fast is it? In our test runs at the default 2 GB memory, both signals together handled about 2.5 images per second (1,000 images in under 7 minutes) and the run memory stayed below 900 MB; with 8 GB memory (more CPU) it was about 4 times faster. "Duplicate hashes only" checked 1,000 images in 8 seconds with 512 MB. Download time from slow image hosts comes on top.

What is the difference between the duplicate distance and the similarity threshold? The duplicate distance works on perceptual hashes and answers "is this the same picture?". The similarity threshold works on the vision model's score and answers "do these images show something similar?". Use hashes for deduplication, the score for lookalikes.

Which copy is marked to keep? The one with the most pixels, then the largest file, then the first in your list. All other copies point to it in duplicateOf.

What if a field is empty? similarityScore is empty in hash-only mode and the hash fields are empty in similarity-only mode. Images without a duplicate have no groupId. Images that could not be read have only error and errorCode.

Can I run it on a schedule or through the API? Yes. Start it from the API, an integration or a schedule, and pass image objects with your own customId to join results back to your records.

⚠️ Limits

  • Images up to 100 MB and 200 megapixels each. Very large non-JPEG images (roughly above 100 megapixels) need more run memory; such an image gets a clear error row telling you so.
  • SVG, PDF and web pages are not images: use the direct image link.
  • Animated GIF and WebP are compared by their first frame.
  • Rotations, mirror images and heavy crops are not duplicates by hash; the similarity score still rates them as similar.
  • With "Group similar images too" on, groups are linked step by step, so a long chain of gradually different images can end up in one group; raise the threshold if groups grow too broad.
  • With a maximum charge per run, a duplicate search reads only as many images as that charge covers and groups those; the status message says so.
  • Hash-only runs need little memory: you can start them with 512 MB. Runs with the vision model use the default 2 GB; more memory also brings more CPU and speed.
  • The default run timeout is 4 hours, enough for about 30,000 images with both signals at the default memory (fewer when the images come from slow hosts); raise it, or the memory, for bigger lists.

Related actors

//From the blog\\

Latest articles

All articles
G2
4.7/5
Excellent
Trustpilot

The internet is full of data.
Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes.

Get Started FreeChat with Support
No credit card. No setup calls.