Remove Duplicates from Datasets, CSV & JSON - Smart Dedupe
Pricing
from $0.60 / 1,000 per input item deduplicateds
Remove Duplicates from Datasets, CSV & JSON - Smart Dedupe
Deduplicate Apify datasets, CSV or JSON files: match on chosen fields, normalize URLs, emails and whitespace, keep first or last, merge several datasets in one run. $0.0009 per input item.
Pricing
from $0.60 / 1,000 per input item deduplicateds
Rating
0.0
(0)
Developer
Ja doei
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
16 hours ago
Last modified
Categories
Share
✅ Tested daily with its default input (Apify's automated QA plus our own smoke test before every release).
Remove duplicates from Apify datasets, CSV files or JSON files: merge one or more sources and keep only unique rows, matched on the fields you choose with URL, email and whitespace normalization. $0.0009 per input item, no start fee.
What it does
- Match on the fields you choose (for example
url,email,name), or on the whole item. - Normalizes before comparing (optional, on by default): trims and lowercases text, strips
http(s)://,www., trailing slashes and tracking parameters (utm_*,fbclid,gclid). Sohttps://www.example.com/a/?utm_source=xandexample.com/acount as the same. - Keep the first or the last occurrence of each duplicate.
- Merges several sources in one run: dataset IDs, CSV/JSON URLs and inline items together.
Output
- Unique items in the default dataset, in their original order.
- A summary in the key-value store record
OUTPUT: input items, unique items, duplicates removed.
Pricing
Pay per event: per input item. Lower prices on Silver and Gold plans. No start fee. Platform usage is included.
Input example
Notes
- Works only on data you provide (your datasets or files). It does not collect any data itself.
- Large datasets are streamed from the Apify API; output is written in batches of 1,000.
Use with AI agents (MCP)
This Actor can be used as a tool by AI agents (Claude, Cursor, VS Code, other MCP clients) through the official Apify MCP server:
- Add only this Actor as a tool:
https://mcp.apify.com?tools=admirala/smart-dedupe(sign in with OAuth or an Apify API token). - Or connect
https://mcp.apify.comand let the agent find it withsearch-actors(e.g. "remove duplicates"), read the input schema withfetch-actor-detailsand run it withcall-actor.
Example input an agent can pass:
Cost per call: $0.0009 per input item (lower on higher Apify plans). Results come back as dataset items (get-dataset-items).
FAQ
How do I remove duplicate rows from an Apify dataset?
Put the dataset ID in datasetIds and the field(s) to compare in fields, e.g. ["url"].
Can I deduplicate a CSV file?
Yes, give a direct CSV or JSON file URL in urls.
Are https://www.example.com/a/ and example.com/a duplicates?
Yes with normalize: true (default): scheme, www, trailing slash and utm parameters are ignored.
