Duplicate Image Finder: exact and near duplicate images

Give it a list of image URLs and it tells you which ones are the same picture. Exact copies are caught by file checksum. Near copies, such as the same photo saved at another size, recompressed, or turned to grayscale, are caught by perceptual hashing, which compares what the image looks like and ignores the bytes.

Matching images are put in numbered groups. In each group, the copy with the most pixels is marked as the one to keep, so you can delete or skip the rest.

No outside service or AI model is used, and there's no key to supply.

How matching works

Each image is downloaded once, turned upright from its EXIF orientation, flattened onto white where it's transparent, and reduced to a 64 bit pHash: the image is shrunk to 32 by 32 gray pixels and the low frequencies of its discrete cosine transform are compared with their median. Two images are grouped when their pHash values differ in no more than the number of bits you set (10 by default) or when their files are byte for byte identical. Groups chain, so if A matches B and B matches C, all three share a group.

In testing, the same photo at 400 by 300 and at 800 by 600, a grayscale copy, a blurred copy and a second URL serving the identical file all hashed to a distance of 0 from each other. Four unrelated photos sat 28 to 36 bits apart. A square crop of the 400 by 300 photo fell outside the default distance and was not grouped, so heavy crops need a higher setting or won't match at all.

The dHash and aHash of every image are returned too, for anyone who wants to run their own comparison.

Input

• Image URLs (required). Direct links to JPEG, PNG, WebP, GIF, BMP, TIFF and other formats Pillow reads, up to 20,000 in one run. Repeated URLs are read once. For an animated image, the first frame is compared.
• Maximum hash distance, 10 by default, from 0 to 32.
• Only return images that have a duplicate, off by default.
• Maximum file size, 25 MB by default.
• Parallel downloads, 5 by default.

Output

One row per image:

• url, finalUrl after redirects, ok
• groupId and groupSize, or null and 1 when the image has no match
• isDuplicate, keep and keepUrl, the copy with the most pixels in the group (the larger file breaks a tie)
• closestMatchUrl and closestMatchDistance
• matches: up to 20 matching images, closest first, each with url, distance, similarity from 0 to 1 and sameFile
• exactDuplicate, true when another image in the run is the same file
• width, height, megapixels, format, fileSizeBytes, sha256
• phash, dhash and ahash as 16 digit hex strings

A link that fails to download, answers with an error status, or isn't an image gets a row with ok false and the reason in error, and the rest of the run carries on. The OUTPUT record in the key value store lists every group with its URLs, plus counts of images processed and failed.

Pricing

Pay per event: one event per image processed. Images that fail aren't charged. Images left out by the "only duplicates" option were still compared and are still charged. Before an image is downloaded, the run checks that the maximum charge you set can still pay for it; when it can't, the run stops there, and every image row in the dataset is a paid one.

Support

Questions and problems: the Issues tab on this page. Replies are written by an AI assistant on behalf of the developer and reviewed by the developer.