Duplicate Image Finder – Detect Repeated and Similar Photos

What is it for? Photo galleries and scraped product images are full of repeats: the same picture uploaded twice, at a smaller size, or saved again at lower quality. This tool finds those repeats even when the files are not identical, groups them, and tells you which copy to keep (the largest one). Paste image links, or plug in the results of any other Apify scraper.

Find duplicate and near-duplicate images in a list of URLs or in the dataset of any scraper. Input: image URLs (or an Apify dataset ID + the field that holds the image URL). Output: one row per image with its perceptual hash (pHash), size, duplicate group, whether it is a duplicate and which copy to keep (the highest resolution one), plus a summary with the list of unique images. Catches copies that byte-level checksums miss: resized, recompressed, re-encoded (JPEG ↔ WebP ↔ PNG) or lightly edited versions of the same picture.

Clean the images your scraper collected

Scraped product catalogs, real-estate listings, marketplaces and social feeds are full of the same photo at different sizes and URLs. Run this Actor right after your scraper:

{
  "datasetId": "<default dataset ID of your scraper run>",
  "imageUrlField": "images",
  "threshold": 5
}

• imageUrlField accepts dot paths (product.image.url, images.0) and expands arrays of URLs or of { "url": … } / { "src": … } objects.
• Chain it in an Apify integration / webhook ("run after scraper succeeds") and pass the run's defaultDatasetId.
• Use OUTPUT.keptUrls from the key-value store to download only the unique, best-quality images.

Input example (plain URLs)

{
  "imageUrls": [
    "https://picsum.photos/id/10/1600/1000",
    "https://picsum.photos/id/10/400/250",
    "https://picsum.photos/id/20/1200/800"
  ],
  "threshold": 5,
  "maxImages": 1000,
  "maxImageSizeMb": 20
}

Field	Default	Notes
imageUrls	—	Public http(s) image URLs
datasetId / imageUrlField	— / imageUrl	Read URLs from an Apify dataset (combinable with imageUrls)
threshold	5	Max Hamming distance (0–20 of 64 bits). 0 = identical, 5 = resized/recompressed copies, 10+ = looser
maxImages	1000	Hard cap 5000
maxImageSizeMb	20	Hard cap 50

Output example

Dataset, one row per image:

{
  "url": "https://picsum.photos/id/10/400/250",
  "hash": "ab4cd46c4b334726",
  "width": 400,
  "height": 250,
  "groupId": "g0001",
  "groupSize": 2,
  "isDuplicate": true,
  "keptUrl": "https://picsum.photos/id/10/1600/1000",
  "distance": 1,
  "error": null
}

• isDuplicate: false → this is the copy to keep (unique images are their own group).
• keptUrl → the best copy of the group: highest resolution, then biggest file.
• distance → Hamming distance to the kept copy (0 = identical hash).
• Images that could not be downloaded or decoded get a row with error and the run continues.

Key-value store OUTPUT (summary):

{
  "totalUrls": 3, "analyzedImages": 3, "failedImages": 0, "notProcessedDueToSpendingLimit": 0,
  "threshold": 5, "groups": 2, "duplicateGroups": 1, "duplicates": 1, "uniqueImages": 2,
  "keptUrls": ["…"], "duplicateUrls": ["…"]
}

How it works

pHash (DCT-based, 64 bits): each image is EXIF-rotated, converted to greyscale, resized to 32×32, transformed with a 2D DCT and the 8×8 low-frequency block is thresholded against its mean. Similar pictures produce hashes with a small Hamming distance. Images closer than threshold are linked (single linkage: if A≈B and B≈C they share a group).

Pricing

Free during launch. This Actor has no usage fee of its own: you only pay Apify's standard platform usage for your runs (on the free Apify plan that is covered by your monthly credit). A pay-per-result price may be introduced later; Apify notifies users in advance of any price change.

Limits

• Up to 5000 images per run (maxImages), 50 MB per image (maxImageSizeMb, default 20 MB), 100 megapixels per image.
• Only public http/https URLs; internal network addresses are refused. No login or cookies.
• Exact duplicate URLs in the input are analyzed once.
• Perceptual hashing detects the same picture: it does not match crops that remove much of the image, mirrored/rotated copies or "similar but different" photos (e.g. two shots of the same scene may or may not match depending on threshold).
• Comparison is all-pairs (O(n²)); fine up to the 5000-image cap.