Merge & Dedupe Datasets: Combine Runs, Remove Duplicates avatar

Merge & Dedupe Datasets: Combine Runs, Remove Duplicates

Pricing

from $0.32 / 1,000 merged rows

Go to Apify Store
Merge & Dedupe Datasets: Combine Runs, Remove Duplicates

Merge & Dedupe Datasets: Combine Runs, Remove Duplicates

Merge Apify datasets from many runs, plus CSV, JSON or Excel files, into one table and remove duplicates by any field. Ignore case, spaces and URL tracking; keep the first, last or most complete row; get only rows new since earlier runs, or keep a master list. Runs with limited permissions.

Pricing

from $0.32 / 1,000 merged rows

Rating

0.0

(0)

Developer

Michael Costa

Michael Costa

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

12 days ago

Last modified

Features

Limited permissions

Share

What does Merge & Dedupe Datasets do?

Merge & Dedupe Datasets combines Apify datasets from many runs, and CSV, JSON or Excel files, into one table with the duplicates removed. Say which fields make a row unique, which differences to ignore (case, spaces, URL tracking, phone formatting) and which copy to keep: the first, the last, the most complete, or all of them combined.

It's the step between a scraper you run again and again and the sheet or CRM the rows end up in. Connect it to a scraper's run, and each run adds only the rows you've never had to a growing master list.

It runs with limited permissions: no approval screen, and it reads only the datasets you pick, nothing else in your account.

Input fields · API · Use it from Claude, ChatGPT or Cursor (MCP)

Jump to: Price · How to use · Across runs · Input · Output · Switching · Limits · FAQ

Try it in one click: the input comes pre-filled with the US Treasury's official exchange rates on 2025-12-31 and 2024-12-31, merged by currency: 372 rows go in and 196 come out, one per currency, with the newer rate where there is one. About $0.09 (196 × $0.00045, plus $0.00005 for the run start). Then pick your own datasets.

How much does it cost to merge datasets?

You pay per row in the result: $0.45 per 1,000 rows, plus $0.00005 each time a run starts. Duplicates and rows skipped because an earlier run returned them are free, however many are read, and there's no usage bill on top.

It's cheaper on paid Apify plans: $0.40 per 1,000 rows on Starter, $0.36 on Scale and $0.32 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show.

  • The example: 196 rows × $0.00045 = about $0.09, plus the start fee.
  • For example: 30 runs of a scraper, 10,000 rows read, 2,500 of them unique: 2,500 × $0.00045 = $1.13.
  • Caps: Max rows per run in the input, and Maximum cost per run in the run options. The run stops cleanly at whichever comes first.

Never charged: duplicates, rows already returned by earlier runs, empty rows, rows too large to write, and the optional file.

How to merge datasets and remove duplicates

  1. Open Merge & Dedupe Datasets and click Try for free (or Start if you're signed in).
  2. Pick your datasets in Apify datasets to merge, in the order to read them (or paste links in file URLs).
  3. In Dedupe by, name the fields that identify a row, such as email or name, city. Leave it empty to remove only rows identical in every field.
  4. Optional: choose what to ignore when comparing, and which duplicate to keep.
  5. Click Start and open the Output tab (export as JSON, CSV or Excel). The SUMMARY record has the counts per source.

Example: one exchange rate per currency

The pre-filled input (two public CSV files, the newer one first, so the newer rate is the one kept):

{"fileUrls": [
"https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300",
"https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2024-12-31&page[size]=300"],
"dedupeBy": ["country_currency_desc"]}

A typical job, leads from two scrapes, matched on the website however it was written, with blanks filled in:

{"datasetIds": ["<January's dataset>", "<February's dataset>"],
"dedupeBy": ["website"],
"normalize": ["trim", "lowercase", "url"],
"keep": "merge"}

Ready-to-run examples

Each example opens with the input already filled in. Run it as it is, or change the input first.

Remove duplicates across runs of a scraper

Listing every dataset by hand works once. For a scraper on a schedule, let the actor remember instead:

  1. Open the scraper or its saved task, go to the Integrations tab and add Apify actor → Merge & Dedupe Datasets, triggered when a run succeeds.
  2. Set "datasetIds": ["{{resource.defaultDatasetId}}"] (Apify fills in the finished run's dataset). This line is needed: Apify gives a limited-permissions run access only to the datasets named in that field, so the dataset in the integration's own payload can't be read, and a run with the field empty tells you to add it.
  3. Set Dedupe by, and one of:
    • Only rows not returned by earlier runs: each run's dataset holds just the rows that are new, ready to send on.
    • Master dataset, e.g. all-leads: the same, and new rows are also added to that one dataset, which grows into your full list without duplicates.

What's remembered is a fingerprint of each key, not your data, in a key-value store in your own account. A row is remembered only once it's in your dataset, so rows a cost limit cut off come back next run. Changing Dedupe by or what's ignored starts a new memory, because the same keys no longer mean the same thing.

Input

FieldWhat it does
Apify datasets to mergeYour datasets, read in order. Read-only, exactly these. Up to 200 with the files.
Or: file URLsCSV/TSV, JSON, JSON Lines or Excel (first sheet) links, read after the datasets.
Dedupe byemail, or name, city; dots reach nested fields. Empty or *: only identical rows.
Ignore these differencestrim (the default), lowercase, accents, url, digits. The output keeps the original values.
Which duplicate to keepfirst (the default), last, mostComplete or merge.
Only rows not returned by earlier runsSkips keys earlier runs returned, free.
Memory nameShares one memory between several tasks or scrapers.
Master datasetA dataset that keeps every unique row across runs; turns the option above on.
Add a field saying where each row came fromAdds mergeSource.
Also save as a fileOUTPUT.csv or OUTPUT.jsonl, free.
Max rows per runCap the rows written.

url compares web addresses without the scheme, www., default port, fragment, trailing slash and tracking parameters (utm_*, gclid, fbclid and similar), so https://www.acme.com/?utm_source=x and http://acme.com are one row. digits keeps only the digits, so +1 (555) 010-2000 and 15550102000 match; use it when the key is a phone number, since it would empty a name. A row with every dedupe field empty is always kept, since nothing says it repeats.

Output

One dataset row per unique key: the row as it was, with its own fields (plus mergeSource if you asked for it). The dataset can be downloaded as CSV, JSON, Excel, XML or HTML, or passed on to the next actor.

The run's SUMMARY record has the rows read and written, duplicates removed, rows already returned by earlier runs, and the same per source:

{"rowsRead": 372, "rowsWritten": 196, "duplicatesRemoved": 176, "alreadyReturnedInEarlierRuns": 0,
"sources": [{"source": "https://api.fiscaldata.treasury.gov/…/rates_of_exchange", "rowsRead": 174, "emitted": 167, "duplicates": 7, "alreadyReturned": 0, "format": "csv", "error": null},
{"source": "https://api.fiscaldata.treasury.gov/…/rates_of_exchange #2", "rowsRead": 198, "emitted": 29, "duplicates": 169, "alreadyReturned": 0, "format": "csv", "error": null}]}

File links appear here, in the log and in mergeSource without their query string or password, so a signed link's secret stays in your input alone. In the optional CSV, text a spreadsheet would run as a formula (starting with =, +, -, @, a tab or a line break, even after spaces) gets an apostrophe in front so it opens as text; plain numbers and the dataset itself are unchanged.

Can I use Merge & Dedupe Datasets from an AI agent (MCP)?

Yes, through Apify's MCP server, from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify:

{
"mcpServers": {
"apify": {
"url": "https://mcp.apify.com?tools=humble-echidna/dataset-merge"
}
}
}

To use an Apify API token instead of signing in, add "headers": {"Authorization": "Bearer <APIFY_TOKEN>"} next to url.

Who it's for

Anyone who runs the same scraper repeatedly or splits a big job across runs: lead lists built up week by week, job or product monitors that should alert only on what's new, and one table from several scrapers' results.

Why this one?

  • No permission errors or approval screens. The form's dataset picker gives the run read access to exactly the datasets you choose; no account-wide access is asked for.
  • A deleted dataset doesn't lose the merge. One source that can't be read is reported with the reason, and the rest are still merged.
  • Compare the way your data really differs. Case, spaces, accents, URL tracking parameters and phone formatting, no code to write.
  • Keep the best copy. First, last, most complete, or combined, so a later scrape can fill the blanks of an earlier one.
  • Across runs without listing them. Only new rows, or a master list, from one integration.
  • Merged rows stay within limits. A combined row that would pass 5 MB keeps its previous version, and is counted.
  • You pay for the result. Duplicates are free, and there's no usage bill.
  • Fails fast on a typo. If no row has the dedupe field, the run stops before writing or charging anything and lists the fields the first row has.

Switching from another actor?

Most merge and dedupe actors take the same ideas under other names, so moving takes minutes:

ElsewhereHere
A list of dataset idsApify datasets to merge (datasetIds); the same ids work through the API
fields (what must be unique)Dedupe by (dedupeBy); no fields means identical rows are removed, not everything kept
A dataset of earlier items to filter againstOnly rows not returned by earlier runs (onlyNew), or Master dataset
An output dataset nameMaster dataset (masterDataset)
Add the dataset id to each rowAdd a field saying where each row came from (addSourceField)
JavaScript transform functionsNot here: chain Dataset Transformer to filter, rename and reshape, or AI for Every Row

Loading every run of an actor by its name isn't here, on purpose: that requires an actor with full access to your account, since Apify doesn't let a limited-permissions run list your runs. Use an integration with Master dataset from now on; for your past runs, pick their datasets once.

Limits

  • Up to 200 datasets and files per run, and 2,000,000 rows read per run.
  • first holds only keys in memory. last, mostComplete and merge also hold one row per key until the end. Both are limited to 60% of the run's memory; the status tells you to give the run more memory if it's not enough (about 300,000 rows of 2 KB at the default 1 GB with these three modes).
  • Files: up to 200 MB (after decompression), public web addresses on ports 80 and 443; Excel's first sheet only.
  • A single row can be up to 5 MB; larger ones are skipped and counted.
  • It finds exact matches after the differences you ignore, not similar-looking text (Acme Inc and Acme Incorporated are two rows).
  • Fingerprints are 64 bits: a false match between two different keys is about a 1-in-a-million chance at 6 million distinct keys, so don't use it where a single lost row among millions matters.

FAQ

Why is a dataset reported as unreadable?

Pick it in the form's dataset picker, or pass its id in datasetIds: those are what give the run access. A dataset named anywhere else in the input can't be read by a limited-permissions run. It must also be in your account or shared with you, and not expired (unnamed datasets are deleted after your retention period).

Why didn't my duplicates go away?

Check that Dedupe by names the fields as written (case-sensitive; dots for nested fields), and whether the values differ in something you aren't ignoring: compare two rows, then add lowercase or url. The SUMMARY counts rows with no key, which are always kept.

Which order are rows in?

The order they were first read, datasets first and then files, each in its listed order. With last, mostComplete or merge, the kept row stays where its key first appeared.

Can I start the memory again?

Yes: delete the seen-… records in your dataset-merge-memory key-value store, or use a new Memory name.

It only processes data you choose: your own Apify datasets, and files at addresses you supply, fetched politely (robots.txt honoured, honest User-Agent, public addresses on ports 80 and 443). Make sure you're allowed to use what you point it at. The example data is the US Treasury's Fiscal Data API, which is "offered free, without restriction" for commercial and non-commercial use.

Something that used to work now fails. Why?

The run log and the status say what went wrong. Please open an issue with the input you used.

ActorUse it when
Dataset TransformerOne source to filter, sort, rename or flatten, as well as dedupe.
Dataset DiffYou want what changed or was removed between two runs, not only what's new.
Join DatasetsYou want to add columns from another dataset by a key, not stack rows.
Google Sheets Import & ExportThe merged rows should end up in a sheet.

Feedback and support

Found a bug, or need a comparison that isn't here? Open an issue on the Issues tab with the input you used.

Versions

Current version: 0.1. See the Changelog tab for what changed in each version.