Remove Duplicates: Deduplicate Datasets, CSV & Excel Data

Use this Actor to remove duplicates from Apify datasets, CSV and Excel files, JSON arrays and plain text lists, in exact or fuzzy mode. It deduplicates by the fields you choose (or by the whole row), cleans values before comparing (case, spaces, accents, emails, URLs, phone numbers), keeps the first, last or most complete row, and can merge each group of duplicates into one enriched row. You get the clean rows, a report of every dropped row with the key it matched, and CSV, Excel or JSON downloads.

What you get

• Clean rows in the dataset, in their original order, with optional duplicateCount and clusterId columns.
• A duplicates report (duplicates-report.json and .csv): which rows were dropped, why (exact or fuzzy with a score), the key they matched, and which row was kept.
• A SUMMARY record (rows in and out, percent removed) and optional CSV, XLSX, JSON or JSONL files of the clean data with download links.

Exact mode is a hash lookup, so matching is linear in the number of rows (500,000 rows matched in about 3 seconds in a local test, before reading and writing). Fuzzy mode finds typos and word-order differences and uses blocking to keep big data practical.

Input example

Copy this into the Input tab (JSON view). It is a real run: the second row is a duplicate of the first (different case and spacing), the third is a near-duplicate (a typo), and the phone number found in the second row is merged into the kept row.

{
  "items": [
    {"name": "Anna Smith", "email": "anna@acme.com", "phone": ""},
    {"name": "ANNA  smith", "email": "Anna@Acme.com", "phone": "+1 555 010 1000"},
    {"name": "Ana Smith", "email": "ana@acme.com", "phone": ""},
    {"name": "Ben Jones", "email": "ben@globex.com", "phone": ""}
  ],
  "keyFields": ["name", "email"],
  "matchMode": "fuzzy",
  "fuzzyThreshold": 90,
  "fieldTypes": {"email": "email"},
  "keepStrategy": "first",
  "mergeDuplicates": true,
  "addDuplicateCount": true,
  "addClusterId": true,
  "exports": ["csv"]
}

Output example

Dataset (2 of 4 rows are left):

{"name": "Anna Smith", "email": "anna@acme.com", "phone": "+1 555 010 1000", "duplicateCount": 2, "clusterId": 1}
{"name": "Ben Jones", "email": "ben@globex.com", "phone": "", "duplicateCount": 0, "clusterId": 2}

duplicates-report.json (first entries):

[
  {"rowNumber": 2, "source": "items", "clusterId": 1, "reason": "exact", "score": 100.0,
   "matchedKey": "anna smith | anna@acme.com", "keptRowNumber": 1, "keptSource": "items",
   "row": {"name": "ANNA  smith", "email": "Anna@Acme.com", "phone": "+1 555 010 1000"}},
  {"rowNumber": 3, "source": "items", "clusterId": 1, "reason": "fuzzy", "score": 95.7,
   "matchedKey": "ana smith | ana@acme.com", "keptRowNumber": 1, "keptSource": "items",
   "row": {"name": "Ana Smith", "email": "ana@acme.com", "phone": ""}}
]

SUMMARY record (shortened):

{"rowsIn": 4, "rowsOut": 2, "duplicatesRemoved": 2, "percentRemoved": 50.0, "exactDuplicates": 1, "fuzzyDuplicates": 1,
 "duplicateGroups": 1, "largestGroup": 3, "rowsCharged": 4, "exportFilesCharged": 1,
 "exports": [{"format": "csv", "key": "clean-data.csv", "url": "https://api.apify.com/v2/key-value-stores/.../records/clean-data.csv", "rows": 2}],
 "report": {"entries": 2, "truncated": false}, "warnings": []}

How to remove duplicates from an Apify dataset

1. Open the Actor and, under Sources, pick your dataset in Apify datasets (or paste its ID). Datasets of other Actors and runs that you can read all work.
2. Under Match, enter the Key fields that identify a row, for example url or email. Leave it empty to compare the whole row.
3. Run it. The dataset of this run holds the unique rows; your original dataset is never modified.
4. Download the clean data from the Output tab, or add csv or xlsx under Download files.

Several datasets at once are deduplicated across each other: a row in dataset B that already exists in dataset A is dropped.

How to remove duplicate rows from a CSV or Excel file

1. Put the file link in File URLs (.csv, .tsv, .xlsx, .json, .jsonl). A Google Sheet works through its "publish to web" CSV link.
2. Choose the key fields by column name, or leave them empty to remove rows that are identical in every column.
3. Add xlsx or csv under Download files. You get a link to the cleaned file in the SUMMARY record.

To clean a list you pasted, use Plain text lines (one value per line) or Items (a JSON array).

How to dedupe a CSV file by email, URL or phone

Put the CSV link in File URLs, set the key field (for example email), then set Smart normaliser per field, for example {"email": "email", "website": "url", "phone": "phone"}:

• email: case-insensitive; Name <a@b.com> and mailto: forms reduce to the address. Turn on emailGmailDots and emailPlusTags to treat first.last+news@gmail.com and firstlast@gmail.com as one address.
• url: ignores http versus https, www., trailing slash, #fragment, default ports, and tracking parameters (utm_*, fbclid, gclid and similar). Paths stay case-sensitive.
• phone: digits only. Use phoneLastDigits (for example 9) to ignore country codes.
• domain: the host of a URL or the part after @ in an email.

The general text clean-up (trim, ignore case, collapse spaces, ignore punctuation, ignore accents) is set in Clean text before comparing.

How to find duplicate rows without deleting them

1. Set What to put in the dataset to duplicates. The dataset then holds only the rows that would be removed, each with clusterId, duplicateOf (the row number that was kept), matchReason and matchScore.
2. Or keep the default (unique rows) and open duplicates-report.csv in the key-value store: one line per dropped row with its source, row number, why it matched, the normalised key, and the row that was kept.
3. Row numbers count rows in the order the sources were read (datasets, dataset links, files, pasted items, text lines), starting at 1.

How to deduplicate a dataset with fuzzy matching (typos, word order, extra words)

1. Set Match mode to fuzzy and choose the key fields, for example company or name and email.
2. Pick the algorithm: tokenSet ignores word order and extra words (good for names and companies), jaroWinkler suits short strings with typos, levenshtein counts character edits.
3. Set the threshold. 90 is a safe start, 80 finds more (and more false ones). The report shows the score of every match so you can tune it.
4. For very large data keep Fuzzy blocking on prefix (fastest) or use phonetic or tokens to catch pairs that start with different letters.

All inputs

Input	What it does	Default
items	Rows pasted as a JSON array of objects.	(8-row sample)
datasetIds	Datasets to deduplicate.	empty
datasetUrls	Dataset IDs, "username~name" names or Apify links, one per line.	empty
fileUrls	Direct links to .csv, .tsv, .xlsx, .json or .jsonl files, one per line.	empty
textLines	One value per line, for example a pasted list of emails, URLs, names or phone numbers.	empty
keyFields	Rows with the same value in all of these fields are duplicates.	empty
matchMode	exact: rows are duplicates when their normalised keys are identical (fast, any size).	exact
fuzzyAlgorithm	tokenSet: ignores word order, repeated words and extra words (best for names, companies, addresses).	tokenSet
fuzzyThreshold	Two rows are near-duplicates when their similarity score is at least this value (100 = identical).	90
blocking	Rows are only compared with rows in the same block, which keeps 100,000+ rows fast.	prefix
blockingChars	For prefix blocking: how many leading characters of the key must match for two rows to be compared.	2
normalise	Applied to every key value before comparing; your original data is never changed.	["trim", "caseFold", "collapseSpaces"]
fieldTypes	Pick a normaliser for fields that hold emails, URLs, phones or domains, as {"field": "type"}.	empty
emailGmailDots	Treat first.last@gmail.com and firstlast@gmail.com as the same address (Gmail ignores dots).	false
emailPlusTags	Treat anna+news@acme.com and anna@acme.com as the same address.	false
phoneLastDigits	0 compares all digits.	0
keepStrategy	first: keep the earliest row (lowest memory use, streams).	first
keepField	Only for the longestField strategy: the field whose longest text wins, e.g.	empty
mergeDuplicates	Fill empty fields of the kept row with values from its duplicates (a phone number found in a second copy, for example).	false
outputMode	unique: the clean rows without duplicates.	unique
addDuplicateCount	Add a duplicateCount column: how many duplicates were folded into each unique row (0 = it had none).	false
addClusterId	Add a clusterId column (1, 2, 3...) that identifies each group of duplicates; the same ID appears in the duplicates report.	false
duplicatesReport	Save duplicates-report.json and duplicates-report.csv in the key-value store: every dropped row with its source, row number, why it matched (exact or fuzzy with...	true
maxReportRows	Longest the duplicates report can get.	10000
exports	Also write the clean data as files in the key-value store, each with a download link in the SUMMARY record.	[]
exportName	Base name of the files, e.g.	clean-data
escapeFormulas	In the CSV export and report, text that starts with = + - or @ gets a leading apostrophe so Excel and Sheets cannot run it as a formula.	true
maxItems	Hard cap on input rows across all sources.	500000
maxItemsPerSource	Optional cap on rows taken from each source, so one huge source cannot use up the whole budget.	empty
maxFileSizeMb	Files larger than this are rejected (that source fails, the others continue).	50
csvInferTypes	Turn CSV text like 42, 3.5 or true into numbers and booleans.	true
xlsxSheet	Sheet name or number (1 = first) to read from every Excel file.	empty

Not in the input form but accepted through the API: inlineSources, a list of {"name", "format": "csv"|"json"|"jsonl"|"lines", "text"} objects for pasted CSV or JSON Lines text.

How much does it cost

Pay per event, usage included:

Event	Price	When
row-processed	$0.0008 per row ($0.80 per 1,000)	each input row in exact mode
fuzzy-row-processed	$0.001 per row ($1.00 per 1,000)	each input row in fuzzy mode (more compute per row)
export-file	$0.004 per file	each CSV, XLSX, JSON or JSONL download file you ask for

The dataset, the duplicates report and the SUMMARY are free. Rows are charged on the way in, whatever the number of duplicates found.

Job	Rows	Files	Price
Exact dedupe of a 1,000-row contact list, CSV download	1,000	1	$0.01 + $0.004 = $0.014
Fuzzy dedupe of 10,000 company records, no files	10,000	0	$0.20
Exact dedupe of 100,000 scraped items, CSV and Excel download	100,000	2	$80.00 + $0.008 = $80.008

The maxItems input caps the rows (default 500,000) and therefore the cost. If you set a maximum cost per run, the Actor stops reading when the budget is spent, still returns the clean result for the rows it did process, and says so in the status message and SUMMARY.

Use cases

• Clean a scraped dataset before delivery: remove repeated products, jobs, listings or profiles by URL or ID.
• Dedupe a CRM or newsletter export by email, ignoring case, spaces, Gmail dots and plus tags.
• Merge several exports (CSV, Excel, other Actor runs) and keep one enriched row per contact, filling empty fields from the copies.
• Find near-duplicate company or person names (typos, "Ltd" versus none, swapped words) before importing to a CRM.
• Normalise a list of URLs, domains or phone numbers and keep each one once, with a report of what was dropped.

Use with the Apify API, MCP, n8n and Make

Python:

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("<username>/remove-duplicates").call(run_input={
    "datasetIds": ["<DATASET_ID>"],
    "keyFields": ["url"],
    "fieldTypes": {"url": "url"},
    "exports": ["csv"],
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)
summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("SUMMARY")["value"]
print(summary["rowsIn"], summary["rowsOut"], summary["percentRemoved"])

JavaScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<APIFY_TOKEN>' });
const run = await client.actor('<username>/remove-duplicates').call({
    items: [{ email: 'A@x.com' }, { email: ' a@x.com' }, { email: 'b@x.com' }],
    keyFields: ['email'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items); // [{ email: 'A@x.com' }, { email: 'b@x.com' }]

In n8n use the Apify node (Run Actor, then Get Dataset Items) or an HTTP Request node on the run-sync endpoint. In Make use the Apify module "Run an Actor" then "Get Dataset Items". For AI agents the Actor works through the Apify MCP server: it needs no required field, and keyFields plus one source (items, datasetIds, fileUrls or textLines) is enough. To dedupe the output of another Actor automatically, chain it to run on "Actor run succeeded" and pass datasetIds: [defaultDatasetId].

FAQ

How is this different from Merge Datasets & Dedupe? Merge Datasets combines sources and filters, reshapes and sorts them, with exact dedupe by key or whole row. Remove Duplicates is built for the dedupe step itself: fuzzy matching, smart email, URL, phone and domain normalisers, most-complete and longest-field strategies, merging a group into one row, a duplicates report with the reason for every dropped row, and a cheap per-row price. Use both together if you need to merge and reshape first.

Does it change my original dataset or file? No. It reads the sources and writes a new dataset for the run. Original datasets and files stay as they are.

How accurate is fuzzy matching, and can it be wrong? Fuzzy matching is a similarity score, so it can merge rows that are different (two people called "John Smith") or miss a pair that is far apart. Use fields that identify the record (name and email together), start at threshold 90, and check duplicates-report.json, which lists every match with its score. Each row is compared with the first row of each group, never chained pair to pair, so a long chain of "almost the same" rows does not collapse into one.

How fast is it, and what are the limits? Exact mode is a hash lookup: matching 500,000 rows took about 3 seconds in a local test (reading and writing the data adds to that). Fuzzy mode is slower. In local tests on synthetic company names with the built-in pure Python scorers it processed 20,000 rows in 8 seconds and 100,000 rows in about 3 minutes (the rapidfuzz library is installed on Apify and should be faster; that is expected, not measured). Crowded data is slower: 20,000 synthetic person names that share many first names took 19 seconds. Blocking is what keeps it practical: rows are only compared when they share a block, by default the first 2 characters of the name's words (legal suffixes such as Inc, Ltd, Corp and filler words are ignored when choosing the block, and word order does not matter). That also means a pair is missed when its first letters differ ("Katherine" and "Catherine", or a typo in the first word of a name); use phonetic or tokens blocking for that, and 3 or 4 blocking characters to go faster. Each block compares a row with at most the 1,000 most recent group leaders, and the SUMMARY warns when that cap was hit. Other limits: 5,000,000 rows per run, 500 MB per file (JSON arrays 100 MB; use JSONL), old .xls files are not supported (save as .xlsx).

What does it do with rows where the key is empty? A row whose key fields are all empty is never collapsed with other empty-key rows. It is only removed if it is an exact copy of another whole row, and the SUMMARY counts these rows (rowsWithoutKey).

What does it not do? It does not match across languages or scripts (no transliteration beyond removing accents), it does not parse addresses or names into parts, it does not use AI or look anything up online, and the kept row keeps its original values (cleaning is only used for matching, never written to your data). The domain normaliser removes www. but does not know public suffixes, so a.example.co.uk and example.co.uk are different domains. The merge step fills empty top-level fields only; it does not combine lists or nested objects. Memory: plain keep-first runs stream and use little memory; keep last, most complete, longest field, merge and duplicates-only output hold the kept rows in memory, so memory grows with the number of kept rows (the run is given 2 GB).

Related tools

• Merge Datasets & Dedupe: Combine, Filter, Export CSV & Excel: combine datasets and files, filter, reshape and sort before or after deduplicating.
• Excel to JSON Converter: XLSX, XLS & CSV to JSON API: turn spreadsheets into clean JSON first.
• Email Extractor from Text: Emails, Phones & URLs from Any Text: pull emails, phones and URLs out of text, then dedupe them here.