Duplicate & Similar Image Finder (Perceptual Hash)

Find exact and near-duplicate images in a list of URLs. It groups copies that were resized, recompressed, converted to another format or lightly edited, picks the best copy to keep in each group, and flags the rest.

What you get

• Groups of the same picture even when the file differs: other size, JPEG vs PNG vs WebP, different compression.
• A keep / duplicate / unique status per image, with the URL of the copy to keep.
• The reason for each pick: the copy with the most pixels wins, then the sharpest, then the largest file.
• Per image: width, height, bytes, format, a sharpness score, SHA-256, hash distance to the kept copy, and whether it is a byte-identical copy.
• No AI model and no external service: perceptual hashes (a DCT hash and a gradient hash that must both agree) computed locally.

Use cases

• Clean up AI-generated batches where similar seeds and re-saves pile up.
• De-duplicate product photos, stock sets and scraped image collections.
• Find lower-resolution copies of the same asset across CDNs and sites.
• Check a gallery for repeated uploads before you publish it.

Quick start

Keep the example URLs (a photo, an identical copy of it, and two different photos) and press Start. The first two end up in one group; the others are unique.

Input

{
  "imageUrls": ["https://raw.githubusercontent.com/ianare/exif-samples/master/jpg/gps/DSCN0010.jpg", "https://raw.githubusercontent.com/ianare/exif-samples/master/jpg/gps/DSCN0010.jpg?copy=1"],
  "similarityThreshold": 6
}

Field	Default	Notes
imageUrls	(required)	2 to 2,000 direct links.
similarityThreshold	6	0-20, out of 64 hash bits. 0 = identical pixels only. About 6 catches resized and recompressed copies. Above 12 starts to group different pictures.
onlyReportDuplicates	false	Leave unique images out of the output.
maxImageSizeMb	30	Larger files are skipped with an error.
maxConcurrency	4	Parallel downloads.

Output

{
  "url": "https://raw.githubusercontent.com/ianare/exif-samples/master/jpg/gps/DSCN0010.jpg?copy=1",
  "status": "duplicate",
  "group_id": 1,
  "group_size": 2,
  "keep_url": "https://raw.githubusercontent.com/ianare/exif-samples/master/jpg/gps/DSCN0010.jpg",
  "distance_to_keep": 0,
  "exact_copy_of_keep": true,
  "width": 640, "height": 480, "bytes": 161713, "format": "jpeg",
  "sharpness": 3007.03
}

Rows are ordered group by group (the kept copy first in its group), then unique images, then failures.

Limits

• Images smaller than 16 px on a side carry too little information for perceptual comparison; they only match byte-identical copies.
• Crops, mirrored copies and heavy edits are different pictures to the hash and are not grouped.
• All images are compared with each other, so very large lists take longer (2,000 images = about 2 million comparisons, done in memory).
• Each site's robots.txt is checked before downloading. Use it on images you are allowed to process.

Pricing

Pay per event: you are charged once per image that was downloaded and analysed. Failed downloads are free.

Related Actors

• Bulk Image Compressor & WebP / AVIF Converter: shrink the images you keep.
• Image Metadata Remover: strip EXIF, GPS and AI prompts before publishing.
• Stable Diffusion Prompt Extractor: read the prompt and settings embedded in AI images.