Merge & Dedupe Datasets: Combine Runs, Remove Duplicates
Pricing
from $0.32 / 1,000 merged rows
Merge & Dedupe Datasets: Combine Runs, Remove Duplicates
Merge Apify datasets from many runs, plus CSV, JSON or Excel files, into one table and remove duplicates by any field. Ignore case, spaces and URL tracking; keep the first, last or most complete row; get only rows new since earlier runs, or keep a master list. Runs with limited permissions.
Pricing
from $0.32 / 1,000 merged rows
Rating
0.0
(0)
Developer
Michael Costa
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
12 days ago
Last modified
Categories
Features
Share
What does Merge & Dedupe Datasets do?
Merge & Dedupe Datasets combines Apify datasets from many runs, and CSV, JSON or Excel files, into one table with the duplicates removed. Say which fields make a row unique, which differences to ignore (case, spaces, URL tracking, phone formatting) and which copy to keep: the first, the last, the most complete, or all of them combined.
It's the step between a scraper you run again and again and the sheet or CRM the rows end up in. Connect it to a scraper's run, and each run adds only the rows you've never had to a growing master list.
It runs with limited permissions: no approval screen, and it reads only the datasets you pick, nothing else in your account.
Input fields · API · Use it from Claude, ChatGPT or Cursor (MCP)
Jump to: Price · How to use · Across runs · Input · Output · Switching · Limits · FAQ
Try it in one click: the input comes pre-filled with the US Treasury's official exchange rates on 2025-12-31 and 2024-12-31, merged by currency: 372 rows go in and 196 come out, one per currency, with the newer rate where there is one. About $0.09 (196 × $0.00045, plus $0.00005 for the run start). Then pick your own datasets.
How much does it cost to merge datasets?
You pay per row in the result: $0.45 per 1,000 rows, plus $0.00005 each time a run starts. Duplicates and rows skipped because an earlier run returned them are free, however many are read, and there's no usage bill on top.
It's cheaper on paid Apify plans: $0.40 per 1,000 rows on Starter, $0.36 on Scale and $0.32 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show.
- The example: 196 rows × $0.00045 = about $0.09, plus the start fee.
- For example: 30 runs of a scraper, 10,000 rows read, 2,500 of them unique: 2,500 × $0.00045 = $1.13.
- Caps: Max rows per run in the input, and Maximum cost per run in the run options. The run stops cleanly at whichever comes first.
Never charged: duplicates, rows already returned by earlier runs, empty rows, rows too large to write, and the optional file.
How to merge datasets and remove duplicates
- Open Merge & Dedupe Datasets and click Try for free (or Start if you're signed in).
- Pick your datasets in Apify datasets to merge, in the order to read them (or paste links in file URLs).
- In Dedupe by, name the fields that identify a row, such as
emailorname, city. Leave it empty to remove only rows identical in every field. - Optional: choose what to ignore when comparing, and which duplicate to keep.
- Click Start and open the Output tab (export as JSON, CSV or Excel). The
SUMMARYrecord has the counts per source.
Example: one exchange rate per currency
The pre-filled input (two public CSV files, the newer one first, so the newer rate is the one kept):
{"fileUrls": ["https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300","https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2024-12-31&page[size]=300"],"dedupeBy": ["country_currency_desc"]}
A typical job, leads from two scrapes, matched on the website however it was written, with blanks filled in:
{"datasetIds": ["<January's dataset>", "<February's dataset>"],"dedupeBy": ["website"],"normalize": ["trim", "lowercase", "url"],"keep": "merge"}
Ready-to-run examples
Each example opens with the input already filled in. Run it as it is, or change the input first.
- Merge CSV files and remove duplicates by a key column: One table with one row per key, also saved as a CSV.
- Combine runs and keep only the newest row per key: Read oldest to newest and keep each record's latest values.
Remove duplicates across runs of a scraper
Listing every dataset by hand works once. For a scraper on a schedule, let the actor remember instead:
- Open the scraper or its saved task, go to the Integrations tab and add Apify actor → Merge & Dedupe Datasets, triggered when a run succeeds.
- Set
"datasetIds": ["{{resource.defaultDatasetId}}"](Apify fills in the finished run's dataset). This line is needed: Apify gives a limited-permissions run access only to the datasets named in that field, so the dataset in the integration's own payload can't be read, and a run with the field empty tells you to add it. - Set Dedupe by, and one of:
- Only rows not returned by earlier runs: each run's dataset holds just the rows that are new, ready to send on.
- Master dataset, e.g.
all-leads: the same, and new rows are also added to that one dataset, which grows into your full list without duplicates.
What's remembered is a fingerprint of each key, not your data, in a key-value store in your own account. A row is remembered only once it's in your dataset, so rows a cost limit cut off come back next run. Changing Dedupe by or what's ignored starts a new memory, because the same keys no longer mean the same thing.
Input
| Field | What it does |
|---|---|
| Apify datasets to merge | Your datasets, read in order. Read-only, exactly these. Up to 200 with the files. |
| Or: file URLs | CSV/TSV, JSON, JSON Lines or Excel (first sheet) links, read after the datasets. |
| Dedupe by | email, or name, city; dots reach nested fields. Empty or *: only identical rows. |
| Ignore these differences | trim (the default), lowercase, accents, url, digits. The output keeps the original values. |
| Which duplicate to keep | first (the default), last, mostComplete or merge. |
| Only rows not returned by earlier runs | Skips keys earlier runs returned, free. |
| Memory name | Shares one memory between several tasks or scrapers. |
| Master dataset | A dataset that keeps every unique row across runs; turns the option above on. |
| Add a field saying where each row came from | Adds mergeSource. |
| Also save as a file | OUTPUT.csv or OUTPUT.jsonl, free. |
| Max rows per run | Cap the rows written. |
url compares web addresses without the scheme, www., default port, fragment, trailing slash and tracking
parameters (utm_*, gclid, fbclid and similar), so https://www.acme.com/?utm_source=x and http://acme.com are one
row. digits keeps only the digits, so +1 (555) 010-2000 and 15550102000 match; use it when the key is a phone
number, since it would empty a name. A row with every dedupe field empty is always kept, since nothing says it repeats.
Output
One dataset row per unique key: the row as it was, with its own fields (plus mergeSource if you asked for it). The
dataset can be downloaded as CSV, JSON, Excel, XML or HTML, or passed on to the next actor.
The run's SUMMARY record has the rows read and written, duplicates removed, rows already returned by earlier runs, and
the same per source:
{"rowsRead": 372, "rowsWritten": 196, "duplicatesRemoved": 176, "alreadyReturnedInEarlierRuns": 0,"sources": [{"source": "https://api.fiscaldata.treasury.gov/…/rates_of_exchange", "rowsRead": 174, "emitted": 167, "duplicates": 7, "alreadyReturned": 0, "format": "csv", "error": null},{"source": "https://api.fiscaldata.treasury.gov/…/rates_of_exchange #2", "rowsRead": 198, "emitted": 29, "duplicates": 169, "alreadyReturned": 0, "format": "csv", "error": null}]}
File links appear here, in the log and in mergeSource without their query string or password, so a signed link's
secret stays in your input alone. In the optional CSV, text a spreadsheet would run as a formula (starting with =,
+, -, @, a tab or a line break, even after spaces) gets an apostrophe in front so it opens as text; plain numbers and the dataset itself
are unchanged.
Can I use Merge & Dedupe Datasets from an AI agent (MCP)?
Yes, through Apify's MCP server, from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify:
{"mcpServers": {"apify": {"url": "https://mcp.apify.com?tools=humble-echidna/dataset-merge"}}}
To use an Apify API token instead of signing in, add
"headers": {"Authorization": "Bearer <APIFY_TOKEN>"} next to url.
Who it's for
Anyone who runs the same scraper repeatedly or splits a big job across runs: lead lists built up week by week, job or product monitors that should alert only on what's new, and one table from several scrapers' results.
Why this one?
- No permission errors or approval screens. The form's dataset picker gives the run read access to exactly the datasets you choose; no account-wide access is asked for.
- A deleted dataset doesn't lose the merge. One source that can't be read is reported with the reason, and the rest are still merged.
- Compare the way your data really differs. Case, spaces, accents, URL tracking parameters and phone formatting, no code to write.
- Keep the best copy. First, last, most complete, or combined, so a later scrape can fill the blanks of an earlier one.
- Across runs without listing them. Only new rows, or a master list, from one integration.
- Merged rows stay within limits. A combined row that would pass 5 MB keeps its previous version, and is counted.
- You pay for the result. Duplicates are free, and there's no usage bill.
- Fails fast on a typo. If no row has the dedupe field, the run stops before writing or charging anything and lists the fields the first row has.
Switching from another actor?
Most merge and dedupe actors take the same ideas under other names, so moving takes minutes:
| Elsewhere | Here |
|---|---|
| A list of dataset ids | Apify datasets to merge (datasetIds); the same ids work through the API |
fields (what must be unique) | Dedupe by (dedupeBy); no fields means identical rows are removed, not everything kept |
| A dataset of earlier items to filter against | Only rows not returned by earlier runs (onlyNew), or Master dataset |
| An output dataset name | Master dataset (masterDataset) |
| Add the dataset id to each row | Add a field saying where each row came from (addSourceField) |
| JavaScript transform functions | Not here: chain Dataset Transformer to filter, rename and reshape, or AI for Every Row |
Loading every run of an actor by its name isn't here, on purpose: that requires an actor with full access to your account, since Apify doesn't let a limited-permissions run list your runs. Use an integration with Master dataset from now on; for your past runs, pick their datasets once.
Limits
- Up to 200 datasets and files per run, and 2,000,000 rows read per run.
firstholds only keys in memory.last,mostCompleteandmergealso hold one row per key until the end. Both are limited to 60% of the run's memory; the status tells you to give the run more memory if it's not enough (about 300,000 rows of 2 KB at the default 1 GB with these three modes).- Files: up to 200 MB (after decompression), public web addresses on ports 80 and 443; Excel's first sheet only.
- A single row can be up to 5 MB; larger ones are skipped and counted.
- It finds exact matches after the differences you ignore, not similar-looking text (
Acme IncandAcme Incorporatedare two rows). - Fingerprints are 64 bits: a false match between two different keys is about a 1-in-a-million chance at 6 million distinct keys, so don't use it where a single lost row among millions matters.
FAQ
Why is a dataset reported as unreadable?
Pick it in the form's dataset picker, or pass its id in datasetIds: those are what give the run access. A dataset
named anywhere else in the input can't be read by a limited-permissions run. It must also be in your account or shared
with you, and not expired (unnamed datasets are deleted after your retention period).
Why didn't my duplicates go away?
Check that Dedupe by names the fields as written (case-sensitive; dots for nested fields), and whether the values
differ in something you aren't ignoring: compare two rows, then add lowercase or url. The SUMMARY counts rows with
no key, which are always kept.
Which order are rows in?
The order they were first read, datasets first and then files, each in its listed order. With last, mostComplete or
merge, the kept row stays where its key first appeared.
Can I start the memory again?
Yes: delete the seen-… records in your dataset-merge-memory key-value store, or use a new Memory name.
Is it legal to merge datasets and files with it?
It only processes data you choose: your own Apify datasets, and files at addresses you supply, fetched politely (robots.txt honoured, honest User-Agent, public addresses on ports 80 and 443). Make sure you're allowed to use what you point it at. The example data is the US Treasury's Fiscal Data API, which is "offered free, without restriction" for commercial and non-commercial use.
Something that used to work now fails. Why?
The run log and the status say what went wrong. Please open an issue with the input you used.
Related actors
| Actor | Use it when |
|---|---|
| Dataset Transformer | One source to filter, sort, rename or flatten, as well as dedupe. |
| Dataset Diff | You want what changed or was removed between two runs, not only what's new. |
| Join Datasets | You want to add columns from another dataset by a key, not stack rows. |
| Google Sheets Import & Export | The merged rows should end up in a sheet. |
Feedback and support
Found a bug, or need a comparison that isn't here? Open an issue on the Issues tab with the input you used.
Versions
Current version: 0.1. See the Changelog tab for what changed in each version.