# Remove Duplicates: Deduplicate Datasets, CSV & Excel (`data-done-right/remove-duplicates`) Actor Remove duplicates from Apify datasets, pasted JSON, CSV, Excel and JSONL files or text lines. Deduplicate by chosen fields or whole rows, exact or fuzzy, with email, URL, phone and domain normalisers. Keep first, last or most complete, or merge. Get clean rows and a duplicates report. - **URL**: https://apify.com/data-done-right/remove-duplicates.md - **Developed by:** [Data Done Right](https://apify.com/data-done-right) (community) - **Categories:** Developer tools, Automation, Integrations - **Security:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.64 / 1,000 row processeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Remove Duplicates: Deduplicate Datasets, CSV & Excel Data Use this Actor to **remove duplicates** from Apify datasets, CSV and Excel files, JSON arrays and plain text lists, in exact or fuzzy mode. It deduplicates by the fields you choose (or by the whole row), cleans values before comparing (case, spaces, accents, emails, URLs, phone numbers), keeps the first, last or most complete row, and can merge each group of duplicates into one enriched row. You get the clean rows, a report of every dropped row with the key it matched, and CSV, Excel or JSON downloads. **What you get** - Clean rows in the dataset, in their original order, with optional `duplicateCount` and `clusterId` columns. - A duplicates report (`duplicates-report.json` and `.csv`): which rows were dropped, why (exact or fuzzy with a score), the key they matched, and which row was kept. - A SUMMARY record (rows in and out, percent removed) and optional CSV, XLSX, JSON or JSONL files of the clean data with download links. Exact mode is a hash lookup, so matching is linear in the number of rows (500,000 rows matched in about 3 seconds in a local test, before reading and writing). Fuzzy mode finds typos and word-order differences and uses blocking to keep big data practical. ### Input example Copy this into the Input tab (JSON view). It is a real run: the second row is a duplicate of the first (different case and spacing), the third is a near-duplicate (a typo), and the phone number found in the second row is merged into the kept row. ```json { "items": [ {"name": "Anna Smith", "email": "anna@acme.com", "phone": ""}, {"name": "ANNA smith", "email": "Anna@Acme.com", "phone": "+1 555 010 1000"}, {"name": "Ana Smith", "email": "ana@acme.com", "phone": ""}, {"name": "Ben Jones", "email": "ben@globex.com", "phone": ""} ], "keyFields": ["name", "email"], "matchMode": "fuzzy", "fuzzyThreshold": 90, "fieldTypes": {"email": "email"}, "keepStrategy": "first", "mergeDuplicates": true, "addDuplicateCount": true, "addClusterId": true, "exports": ["csv"] } ``` ### Output example Dataset (2 of 4 rows are left): ```json {"name": "Anna Smith", "email": "anna@acme.com", "phone": "+1 555 010 1000", "duplicateCount": 2, "clusterId": 1} {"name": "Ben Jones", "email": "ben@globex.com", "phone": "", "duplicateCount": 0, "clusterId": 2} ``` `duplicates-report.json` (first entries): ```json [ {"rowNumber": 2, "source": "items", "clusterId": 1, "reason": "exact", "score": 100.0, "matchedKey": "anna smith | anna@acme.com", "keptRowNumber": 1, "keptSource": "items", "row": {"name": "ANNA smith", "email": "Anna@Acme.com", "phone": "+1 555 010 1000"}}, {"rowNumber": 3, "source": "items", "clusterId": 1, "reason": "fuzzy", "score": 95.7, "matchedKey": "ana smith | ana@acme.com", "keptRowNumber": 1, "keptSource": "items", "row": {"name": "Ana Smith", "email": "ana@acme.com", "phone": ""}} ] ``` `SUMMARY` record (shortened): ```json {"rowsIn": 4, "rowsOut": 2, "duplicatesRemoved": 2, "percentRemoved": 50.0, "exactDuplicates": 1, "fuzzyDuplicates": 1, "duplicateGroups": 1, "largestGroup": 3, "rowsCharged": 4, "exportFilesCharged": 1, "exports": [{"format": "csv", "key": "clean-data.csv", "url": "https://api.apify.com/v2/key-value-stores/.../records/clean-data.csv", "rows": 2}], "report": {"entries": 2, "truncated": false}, "warnings": []} ``` ### How to remove duplicates from an Apify dataset 1. Open the Actor and, under Sources, pick your dataset in **Apify datasets** (or paste its ID). Datasets of other Actors and runs that you can read all work. 2. Under Match, enter the **Key fields** that identify a row, for example `url` or `email`. Leave it empty to compare the whole row. 3. Run it. The dataset of this run holds the unique rows; your original dataset is never modified. 4. Download the clean data from the Output tab, or add `csv` or `xlsx` under Download files. Several datasets at once are deduplicated across each other: a row in dataset B that already exists in dataset A is dropped. ### How to remove duplicate rows from a CSV or Excel file 1. Put the file link in **File URLs** (`.csv`, `.tsv`, `.xlsx`, `.json`, `.jsonl`). A Google Sheet works through its "publish to web" CSV link. 2. Choose the key fields by column name, or leave them empty to remove rows that are identical in every column. 3. Add `xlsx` or `csv` under **Download files**. You get a link to the cleaned file in the SUMMARY record. To clean a list you pasted, use **Plain text lines** (one value per line) or **Items** (a JSON array). ### How to dedupe a CSV file by email, URL or phone Put the CSV link in **File URLs**, set the key field (for example `email`), then set **Smart normaliser per field**, for example `{"email": "email", "website": "url", "phone": "phone"}`: - `email`: case-insensitive; `Name ` and `mailto:` forms reduce to the address. Turn on `emailGmailDots` and `emailPlusTags` to treat `first.last+news@gmail.com` and `firstlast@gmail.com` as one address. - `url`: ignores http versus https, `www.`, trailing slash, `#fragment`, default ports, and tracking parameters (`utm_*`, `fbclid`, `gclid` and similar). Paths stay case-sensitive. - `phone`: digits only. Use `phoneLastDigits` (for example 9) to ignore country codes. - `domain`: the host of a URL or the part after `@` in an email. The general text clean-up (trim, ignore case, collapse spaces, ignore punctuation, ignore accents) is set in **Clean text before comparing**. ### How to find duplicate rows without deleting them 1. Set **What to put in the dataset** to `duplicates`. The dataset then holds only the rows that would be removed, each with `clusterId`, `duplicateOf` (the row number that was kept), `matchReason` and `matchScore`. 2. Or keep the default (unique rows) and open `duplicates-report.csv` in the key-value store: one line per dropped row with its source, row number, why it matched, the normalised key, and the row that was kept. 3. Row numbers count rows in the order the sources were read (datasets, dataset links, files, pasted items, text lines), starting at 1. ### How to deduplicate a dataset with fuzzy matching (typos, word order, extra words) 1. Set **Match mode** to fuzzy and choose the key fields, for example `company` or `name` and `email`. 2. Pick the algorithm: `tokenSet` ignores word order and extra words (good for names and companies), `jaroWinkler` suits short strings with typos, `levenshtein` counts character edits. 3. Set the threshold. 90 is a safe start, 80 finds more (and more false ones). The report shows the score of every match so you can tune it. 4. For very large data keep **Fuzzy blocking** on `prefix` (fastest) or use `phonetic` or `tokens` to catch pairs that start with different letters. ### All inputs | Input | What it does | Default | |---|---|---| | `items` | Rows pasted as a JSON array of objects. | (8-row sample) | | `datasetIds` | Datasets to deduplicate. | empty | | `datasetUrls` | Dataset IDs, "username~name" names or Apify links, one per line. | empty | | `fileUrls` | Direct links to .csv, .tsv, .xlsx, .json or .jsonl files, one per line. | empty | | `textLines` | One value per line, for example a pasted list of emails, URLs, names or phone numbers. | empty | | `keyFields` | Rows with the same value in all of these fields are duplicates. | empty | | `matchMode` | exact: rows are duplicates when their normalised keys are identical (fast, any size). | `exact` | | `fuzzyAlgorithm` | tokenSet: ignores word order, repeated words and extra words (best for names, companies, addresses). | `tokenSet` | | `fuzzyThreshold` | Two rows are near-duplicates when their similarity score is at least this value (100 = identical). | `90` | | `blocking` | Rows are only compared with rows in the same block, which keeps 100,000+ rows fast. | `prefix` | | `blockingChars` | For prefix blocking: how many leading characters of the key must match for two rows to be compared. | `2` | | `normalise` | Applied to every key value before comparing; your original data is never changed. | `["trim", "caseFold", "collapseSpaces"]` | | `fieldTypes` | Pick a normaliser for fields that hold emails, URLs, phones or domains, as {"field": "type"}. | empty | | `emailGmailDots` | Treat first.last@gmail.com and firstlast@gmail.com as the same address (Gmail ignores dots). | `false` | | `emailPlusTags` | Treat anna+news@acme.com and anna@acme.com as the same address. | `false` | | `phoneLastDigits` | 0 compares all digits. | `0` | | `keepStrategy` | first: keep the earliest row (lowest memory use, streams). | `first` | | `keepField` | Only for the longestField strategy: the field whose longest text wins, e.g. | empty | | `mergeDuplicates` | Fill empty fields of the kept row with values from its duplicates (a phone number found in a second copy, for example). | `false` | | `outputMode` | unique: the clean rows without duplicates. | `unique` | | `addDuplicateCount` | Add a duplicateCount column: how many duplicates were folded into each unique row (0 = it had none). | `false` | | `addClusterId` | Add a clusterId column (1, 2, 3...) that identifies each group of duplicates; the same ID appears in the duplicates report. | `false` | | `duplicatesReport` | Save duplicates-report.json and duplicates-report.csv in the key-value store: every dropped row with its source, row number, why it matched (exact or fuzzy with... | `true` | | `maxReportRows` | Longest the duplicates report can get. | `10000` | | `exports` | Also write the clean data as files in the key-value store, each with a download link in the SUMMARY record. | `[]` | | `exportName` | Base name of the files, e.g. | `clean-data` | | `escapeFormulas` | In the CSV export and report, text that starts with = + - or @ gets a leading apostrophe so Excel and Sheets cannot run it as a formula. | `true` | | `maxItems` | Hard cap on input rows across all sources. | `500000` | | `maxItemsPerSource` | Optional cap on rows taken from each source, so one huge source cannot use up the whole budget. | empty | | `maxFileSizeMb` | Files larger than this are rejected (that source fails, the others continue). | `50` | | `csvInferTypes` | Turn CSV text like 42, 3.5 or true into numbers and booleans. | `true` | | `xlsxSheet` | Sheet name or number (1 = first) to read from every Excel file. | empty | Not in the input form but accepted through the API: `inlineSources`, a list of `{"name", "format": "csv"|"json"|"jsonl"|"lines", "text"}` objects for pasted CSV or JSON Lines text. ### How much does it cost Pay per event, usage included: | Event | Price | When | |---|---|---| | `row-processed` | $0.0008 per row ($0.80 per 1,000) | each input row in exact mode | | `fuzzy-row-processed` | $0.001 per row ($1.00 per 1,000) | each input row in fuzzy mode (more compute per row) | | `export-file` | $0.004 per file | each CSV, XLSX, JSON or JSONL download file you ask for | The dataset, the duplicates report and the SUMMARY are free. Rows are charged on the way in, whatever the number of duplicates found. | Job | Rows | Files | Price | |---|---|---|---| | Exact dedupe of a 1,000-row contact list, CSV download | 1,000 | 1 | $0.01 + $0.004 = $0.014 | | Fuzzy dedupe of 10,000 company records, no files | 10,000 | 0 | $0.20 | | Exact dedupe of 100,000 scraped items, CSV and Excel download | 100,000 | 2 | $80.00 + $0.008 = $80.008 | The `maxItems` input caps the rows (default 500,000) and therefore the cost. If you set a maximum cost per run, the Actor stops reading when the budget is spent, still returns the clean result for the rows it did process, and says so in the status message and SUMMARY. ### Use cases - Clean a scraped dataset before delivery: remove repeated products, jobs, listings or profiles by URL or ID. - Dedupe a CRM or newsletter export by email, ignoring case, spaces, Gmail dots and plus tags. - Merge several exports (CSV, Excel, other Actor runs) and keep one enriched row per contact, filling empty fields from the copies. - Find near-duplicate company or person names (typos, "Ltd" versus none, swapped words) before importing to a CRM. - Normalise a list of URLs, domains or phone numbers and keep each one once, with a report of what was dropped. ### Use with the Apify API, MCP, n8n and Make Python: ```python from apify_client import ApifyClient client = ApifyClient("") run = client.actor("/remove-duplicates").call(run_input={ "datasetIds": [""], "keyFields": ["url"], "fieldTypes": {"url": "url"}, "exports": ["csv"], }) for row in client.dataset(run["defaultDatasetId"]).iterate_items(): print(row) summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("SUMMARY")["value"] print(summary["rowsIn"], summary["rowsOut"], summary["percentRemoved"]) ``` JavaScript: ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: '' }); const run = await client.actor('/remove-duplicates').call({ items: [{ email: 'A@x.com' }, { email: ' a@x.com' }, { email: 'b@x.com' }], keyFields: ['email'], }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items); // [{ email: 'A@x.com' }, { email: 'b@x.com' }] ``` In **n8n** use the Apify node (Run Actor, then Get Dataset Items) or an HTTP Request node on the run-sync endpoint. In **Make** use the Apify module "Run an Actor" then "Get Dataset Items". For AI agents the Actor works through the Apify MCP server: it needs no required field, and `keyFields` plus one source (`items`, `datasetIds`, `fileUrls` or `textLines`) is enough. To dedupe the output of another Actor automatically, chain it to run on "Actor run succeeded" and pass `datasetIds: [defaultDatasetId]`. ### FAQ **How is this different from Merge Datasets & Dedupe?** Merge Datasets combines sources and filters, reshapes and sorts them, with exact dedupe by key or whole row. Remove Duplicates is built for the dedupe step itself: fuzzy matching, smart email, URL, phone and domain normalisers, most-complete and longest-field strategies, merging a group into one row, a duplicates report with the reason for every dropped row, and a cheap per-row price. Use both together if you need to merge and reshape first. **Does it change my original dataset or file?** No. It reads the sources and writes a new dataset for the run. Original datasets and files stay as they are. **How accurate is fuzzy matching, and can it be wrong?** Fuzzy matching is a similarity score, so it can merge rows that are different (two people called "John Smith") or miss a pair that is far apart. Use fields that identify the record (name and email together), start at threshold 90, and check `duplicates-report.json`, which lists every match with its score. Each row is compared with the first row of each group, never chained pair to pair, so a long chain of "almost the same" rows does not collapse into one. **How fast is it, and what are the limits?** Exact mode is a hash lookup: matching 500,000 rows took about 3 seconds in a local test (reading and writing the data adds to that). Fuzzy mode is slower. In local tests on synthetic company names with the built-in pure Python scorers it processed 20,000 rows in 8 seconds and 100,000 rows in about 3 minutes (the rapidfuzz library is installed on Apify and should be faster; that is expected, not measured). Crowded data is slower: 20,000 synthetic person names that share many first names took 19 seconds. Blocking is what keeps it practical: rows are only compared when they share a block, by default the first 2 characters of the name's words (legal suffixes such as Inc, Ltd, Corp and filler words are ignored when choosing the block, and word order does not matter). That also means a pair is missed when its first letters differ ("Katherine" and "Catherine", or a typo in the first word of a name); use `phonetic` or `tokens` blocking for that, and 3 or 4 blocking characters to go faster. Each block compares a row with at most the 1,000 most recent group leaders, and the SUMMARY warns when that cap was hit. Other limits: 5,000,000 rows per run, 500 MB per file (JSON arrays 100 MB; use JSONL), old `.xls` files are not supported (save as `.xlsx`). **What does it do with rows where the key is empty?** A row whose key fields are all empty is never collapsed with other empty-key rows. It is only removed if it is an exact copy of another whole row, and the SUMMARY counts these rows (`rowsWithoutKey`). **What does it not do?** It does not match across languages or scripts (no transliteration beyond removing accents), it does not parse addresses or names into parts, it does not use AI or look anything up online, and the kept row keeps its original values (cleaning is only used for matching, never written to your data). The domain normaliser removes `www.` but does not know public suffixes, so `a.example.co.uk` and `example.co.uk` are different domains. The merge step fills empty top-level fields only; it does not combine lists or nested objects. Memory: plain keep-first runs stream and use little memory; keep last, most complete, longest field, merge and duplicates-only output hold the kept rows in memory, so memory grows with the number of kept rows (the run is given 2 GB). ### Related tools - Merge Datasets & Dedupe: Combine, Filter, Export CSV & Excel: combine datasets and files, filter, reshape and sort before or after deduplicating. - Excel to JSON Converter: XLSX, XLS & CSV to JSON API: turn spreadsheets into clean JSON first. - Email Extractor from Text: Emails, Phones & URLs from Any Text: pull emails, phones and URLs out of text, then dedupe them here. # Actor input Schema ## `items` (type: `array`): Rows pasted as a JSON array of objects. Example: \[{"name": "Ann Lee", "email": "ann@x.com"}, {"name": "ann lee", "email": "ANN@x.com"}]. Works offline, so it is the quickest way to try the Actor. You can combine it with the other sources below. Leave empty if you only use a dataset, file links or text lines. ## `datasetIds` (type: `array`): Datasets to deduplicate. Pick them from the list, or paste dataset IDs. Reads go through your run, so datasets of other Actors and runs that you can read all work. Example: \["aBcD1234eFgH5678i", "username~my-named-dataset"]. ## `datasetUrls` (type: `array`): Dataset IDs, "username~name" names or Apify links, one per line. Example: https://console.apify.com/storage/datasets/aBcD1234eFgH5678i. A dataset that is only given here may be unreadable when the run has limited permissions: pick it in "Apify datasets" instead. ## `fileUrls` (type: `array`): Direct links to .csv, .tsv, .xlsx, .json or .jsonl files, one per line. The type is detected from the extension or the content. Example: https://example.com/export/contacts.csv. Links to localhost and private networks are blocked. ## `textLines` (type: `string`): One value per line, for example a pasted list of emails, URLs, names or phone numbers. Each non-empty line becomes a row with one field called text. Example: anna@acme.com (one per line). Keep the key fields empty to compare the whole line. ## `keyFields` (type: `array`): Rows with the same value in all of these fields are duplicates. Example: email, or first name and last name on two lines. Dots reach nested fields: author.id. Leave empty to compare ALL fields of each row. A row where all key fields are empty is only removed if it is an exact copy of another row. ## `matchMode` (type: `string`): exact: rows are duplicates when their normalised keys are identical (fast, any size). fuzzy: exact copies first, then near matches above the similarity threshold (typos, word order, extra words). Fuzzy is slower; speed it up with blocking. ## `fuzzyAlgorithm` (type: `string`): tokenSet: ignores word order, repeated words and extra words (best for names, companies, addresses). tokenSort: ignores word order only. jaroWinkler: typos in short strings with the same start. levenshtein: character edits. Only used in fuzzy mode. ## `fuzzyThreshold` (type: `integer`): Two rows are near-duplicates when their similarity score is at least this value (100 = identical). 90 is a good start; lower finds more matches and more false ones. Example: 88. Only used in fuzzy mode. ## `blocking` (type: `string`): Rows are only compared with rows in the same block, which keeps 100,000+ rows fast. prefix: same first characters (misses pairs that differ in their first characters). phonetic: words that sound alike (Smith/Smyth). tokens: any shared word (most thorough, slower). none: compare everything (small data only). ## `blockingChars` (type: `integer`): For prefix blocking: how many leading characters of the key must match for two rows to be compared. 1-2 finds more pairs, 3-4 is faster on big data. Example: 2. ## `normalise` (type: `array`): Applied to every key value before comparing; your original data is never changed. trim removes spaces at the ends; caseFold makes upper and lower case equal (also ss for the German sharp s); collapseSpaces turns runs of spaces into one; stripPunctuation ignores dots, commas, dashes; stripAccents makes e and the accented letter equal. Default: trim, caseFold, collapseSpaces. ## `fieldTypes` (type: `object`): Pick a normaliser for fields that hold emails, URLs, phones or domains, as {"field": "type"}. Types: email (case, Name forms), url (no http/https, www, trailing slash, utm\_ and click-id parameters), phone (digits only), domain (host of a URL or email), text. Example: {"email": "email", "website": "url", "phone": "phone"}. ## `emailGmailDots` (type: `boolean`): Treat first.last@gmail.com and firstlast@gmail.com as the same address (Gmail ignores dots). Only for gmail.com and googlemail.com. Applies to fields with the email normaliser. ## `emailPlusTags` (type: `boolean`): Treat anna+news@acme.com and anna@acme.com as the same address. Many providers (Gmail, Outlook, Fastmail) deliver both to the same inbox. Applies to fields with the email normaliser. ## `phoneLastDigits` (type: `integer`): 0 compares all digits. Set 9 or 10 to ignore country codes and trunk prefixes, so +44 20 7946 0958 and 020 7946 0958 match on the last 9 digits. Example: 9. Applies to fields with the phone normaliser. ## `keepStrategy` (type: `string`): first: keep the earliest row (lowest memory use, streams). last: keep the latest. mostComplete: keep the row with the most non-empty fields. longestField: keep the row whose Keep-field value is longest. Ties go to the earliest row. Kept rows stay at the position of the group's first row. ## `keepField` (type: `string`): Only for the longestField strategy: the field whose longest text wins, e.g. description. Dots reach nested fields. ## `mergeDuplicates` (type: `boolean`): Fill empty fields of the kept row with values from its duplicates (a phone number found in a second copy, for example). Never overwrites a value that is already there. Holds the kept rows in memory. ## `outputMode` (type: `string`): unique: the clean rows without duplicates. duplicates: only the rows that were removed, each with clusterId, duplicateOf (the row number that was kept), matchReason and matchScore, to inspect what was found. ## `addDuplicateCount` (type: `boolean`): Add a duplicateCount column: how many duplicates were folded into each unique row (0 = it had none). If your data already has a column of that name, the new one is called \_duplicateCount. ## `addClusterId` (type: `boolean`): Add a clusterId column (1, 2, 3...) that identifies each group of duplicates; the same ID appears in the duplicates report. If your data already has that column, the new one is called \_clusterId. ## `duplicatesReport` (type: `boolean`): Save duplicates-report.json and duplicates-report.csv in the key-value store: every dropped row with its source, row number, why it matched (exact or fuzzy with score), the key it matched, and the row number that was kept. Free of charge. ## `maxReportRows` (type: `integer`): Longest the duplicates report can get. The summary counts always cover all rows. Example: 10000. ## `exports` (type: `array`): Also write the clean data as files in the key-value store, each with a download link in the SUMMARY record. Each file is charged once (export-file). Leave empty for the dataset only. ## `exportName` (type: `string`): Base name of the files, e.g. clean-data gives clean-data.csv and clean-data.xlsx. ## `escapeFormulas` (type: `boolean`): In the CSV export and report, text that starts with = + - or @ gets a leading apostrophe so Excel and Sheets cannot run it as a formula. Excel files always store such text as plain text. Turn off only if you need the raw values. ## `maxItems` (type: `integer`): Hard cap on input rows across all sources. Reading stops at the cap and the run still succeeds. You are charged per row processed, so this also caps cost. Example: 100000. ## `maxItemsPerSource` (type: `integer`): Optional cap on rows taken from each source, so one huge source cannot use up the whole budget. Leave empty for no per-source cap. ## `maxFileSizeMb` (type: `integer`): Files larger than this are rejected (that source fails, the others continue). A JSON array file is loaded whole and is limited to 100 MB; use JSONL or CSV for bigger files. ## `csvInferTypes` (type: `boolean`): Turn CSV text like 42, 3.5 or true into numbers and booleans. Values with leading zeros (007), plus signs or very long digit strings stay text. Matching treats 42 and "42" as equal either way. ## `xlsxSheet` (type: `string`): Sheet name or number (1 = first) to read from every Excel file. Leave empty for the first sheet. ## Actor input object example ```json { "items": [ { "id": 1, "name": "Anna Smith", "email": "anna@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin" }, { "id": 2, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston" }, { "id": 3, "name": "Carla Diaz", "email": "carla@initech.com", "company": "Initech", "phone": "", "city": "Denver" }, { "id": 4, "name": " ANNA SMITH ", "email": "Anna@Acme.com", "company": "Acme Inc", "phone": "+1 555 010 1000", "city": "" }, { "id": 5, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston" }, { "id": 6, "name": "Carla Diaz", "email": "CARLA@initech.com ", "company": "Initech", "phone": "", "city": "Denver" }, { "id": 7, "name": "Ana Smith", "email": "ana@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin" }, { "id": 8, "name": "Ben Jonse", "email": "ben@globex.com", "company": "Globex", "phone": "", "city": "Boston" } ], "keyFields": [ "name", "email" ], "matchMode": "fuzzy", "fuzzyAlgorithm": "tokenSet", "fuzzyThreshold": 90, "blocking": "prefix", "blockingChars": 2, "normalise": [ "trim", "caseFold", "collapseSpaces" ], "fieldTypes": { "email": "email" }, "emailGmailDots": false, "emailPlusTags": false, "phoneLastDigits": 0, "keepStrategy": "first", "mergeDuplicates": true, "outputMode": "unique", "addDuplicateCount": true, "addClusterId": true, "duplicatesReport": true, "maxReportRows": 10000, "exports": [ "csv" ], "exportName": "clean-data", "escapeFormulas": true, "maxItems": 500000, "maxFileSizeMb": 50, "csvInferTypes": true } ``` # Actor output Schema ## `results` (type: `string`): The rows left after duplicates were removed (or, in outputMode duplicates, the removed rows), as JSON, CSV, Excel and more. ## `report` (type: `string`): Key-value store records: SUMMARY (rows in and out, percent removed, exact and fuzzy counts, per-source results, download links), duplicates-report.json and duplicates-report.csv (every dropped row with the key it matched and the row that was kept), and the export files of the clean data. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "items": [ { "id": 1, "name": "Anna Smith", "email": "anna@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin" }, { "id": 2, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston" }, { "id": 3, "name": "Carla Diaz", "email": "carla@initech.com", "company": "Initech", "phone": "", "city": "Denver" }, { "id": 4, "name": " ANNA SMITH ", "email": "Anna@Acme.com", "company": "Acme Inc", "phone": "+1 555 010 1000", "city": "" }, { "id": 5, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston" }, { "id": 6, "name": "Carla Diaz", "email": "CARLA@initech.com ", "company": "Initech", "phone": "", "city": "Denver" }, { "id": 7, "name": "Ana Smith", "email": "ana@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin" }, { "id": 8, "name": "Ben Jonse", "email": "ben@globex.com", "company": "Globex", "phone": "", "city": "Boston" } ], "keyFields": [ "name", "email" ], "matchMode": "fuzzy", "fuzzyThreshold": 90, "fieldTypes": { "email": "email" }, "keepStrategy": "first", "mergeDuplicates": true, "addDuplicateCount": true, "addClusterId": true, "exports": [ "csv" ] }; // Run the Actor and wait for it to finish const run = await client.actor("data-done-right/remove-duplicates").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "items": [ { "id": 1, "name": "Anna Smith", "email": "anna@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin", }, { "id": 2, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston", }, { "id": 3, "name": "Carla Diaz", "email": "carla@initech.com", "company": "Initech", "phone": "", "city": "Denver", }, { "id": 4, "name": " ANNA SMITH ", "email": "Anna@Acme.com", "company": "Acme Inc", "phone": "+1 555 010 1000", "city": "", }, { "id": 5, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston", }, { "id": 6, "name": "Carla Diaz", "email": "CARLA@initech.com ", "company": "Initech", "phone": "", "city": "Denver", }, { "id": 7, "name": "Ana Smith", "email": "ana@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin", }, { "id": 8, "name": "Ben Jonse", "email": "ben@globex.com", "company": "Globex", "phone": "", "city": "Boston", }, ], "keyFields": [ "name", "email", ], "matchMode": "fuzzy", "fuzzyThreshold": 90, "fieldTypes": { "email": "email" }, "keepStrategy": "first", "mergeDuplicates": True, "addDuplicateCount": True, "addClusterId": True, "exports": ["csv"], } # Run the Actor and wait for it to finish run = client.actor("data-done-right/remove-duplicates").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "items": [ { "id": 1, "name": "Anna Smith", "email": "anna@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin" }, { "id": 2, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston" }, { "id": 3, "name": "Carla Diaz", "email": "carla@initech.com", "company": "Initech", "phone": "", "city": "Denver" }, { "id": 4, "name": " ANNA SMITH ", "email": "Anna@Acme.com", "company": "Acme Inc", "phone": "+1 555 010 1000", "city": "" }, { "id": 5, "name": "Ben Jones", "email": "ben@globex.com", "company": "Globex", "phone": "+1 555 010 2000", "city": "Boston" }, { "id": 6, "name": "Carla Diaz", "email": "CARLA@initech.com ", "company": "Initech", "phone": "", "city": "Denver" }, { "id": 7, "name": "Ana Smith", "email": "ana@acme.com", "company": "Acme Inc", "phone": "", "city": "Austin" }, { "id": 8, "name": "Ben Jonse", "email": "ben@globex.com", "company": "Globex", "phone": "", "city": "Boston" } ], "keyFields": [ "name", "email" ], "matchMode": "fuzzy", "fuzzyThreshold": 90, "fieldTypes": { "email": "email" }, "keepStrategy": "first", "mergeDuplicates": true, "addDuplicateCount": true, "addClusterId": true, "exports": [ "csv" ] }' | apify call data-done-right/remove-duplicates --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,data-done-right/remove-duplicates" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/TKOQWqRiU98EG0Rni/builds/uqI5vDE1h0IYZZYzI/openapi.json