# Merge & Dedupe Datasets: Combine Runs, Remove Duplicates (`humble-echidna/dataset-merge`) Actor

Merge Apify datasets from many runs, plus CSV, JSON or Excel files, into one table and remove duplicates by any field. Ignore case, spaces and URL tracking; keep the first, last or most complete row; get only rows new since earlier runs, or keep a master list. Runs with limited permissions.

- **URL**: https://apify.com/humble-echidna/dataset-merge.md
- **Developed by:** [Michael Costa](https://apify.com/humble-echidna) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.32 / 1,000 merged rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Merge & Dedupe Datasets do?

**Merge & Dedupe Datasets** combines **Apify datasets from many runs**, and CSV, JSON or Excel files, into **one table
with the duplicates removed**. Say which fields make a row unique, which differences to ignore (case, spaces, URL
tracking, phone formatting) and which copy to keep: the first, the last, the most complete, or all of them combined.

It's the step between a scraper you run again and again and the sheet or CRM the rows end up in. Connect it to a
scraper's run, and each run adds only the rows you've never had to a growing master list.

It runs with **limited permissions**: no approval screen, and it reads only the datasets you pick, nothing else in your
account.

[Input fields](https://apify.com/humble-echidna/dataset-merge/input-schema) ·
[API](https://apify.com/humble-echidna/dataset-merge/api) ·
[Use it from Claude, ChatGPT or Cursor (MCP)](#can-i-use-merge-and-dedupe-datasets-from-an-ai-agent-mcp)

**Jump to:** [Price](#how-much-does-it-cost-to-merge-datasets) · [How to use](#how-to-merge-datasets-and-remove-duplicates) ·
[Across runs](#remove-duplicates-across-runs-of-a-scraper) · [Input](#input) · [Output](#output) ·
[Switching](#switching-from-another-actor) · [Limits](#limits) · [FAQ](#faq)

**Try it in one click:** the input comes pre-filled with the US Treasury's official exchange rates on 2025-12-31 and
2024-12-31, merged by currency: 372 rows go in and 196 come out, one per currency, with the newer rate where there is
one. About $0.09 (196 × $0.00045, plus $0.00005 for the run start). **Then pick your own datasets.**

### How much does it cost to merge datasets?

You pay per row in the result: **$0.45 per 1,000 rows**, plus $0.00005 each time a run starts. **Duplicates and rows
skipped because an earlier run returned them are free**, however many are read, and there's no usage bill on top.

**It's cheaper on paid Apify plans:** $0.40 per 1,000 rows on Starter, $0.36 on Scale and $0.32 on Business. The prices
on this page are the Free-plan price, so on a paid plan you pay less than the examples show.

- **The example:** 196 rows × $0.00045 = about $0.09, plus the start fee.
- **For example:** 30 runs of a scraper, 10,000 rows read, 2,500 of them unique: 2,500 × $0.00045 = **$1.13**.
- **Caps:** **Max rows per run** in the input, and **Maximum cost per run** in the run options. The run stops cleanly
  at whichever comes first.

**Never charged:** duplicates, rows already returned by earlier runs, empty rows, rows too large to write, and the
optional file.

### How to merge datasets and remove duplicates

1. Open Merge & Dedupe Datasets and click **Try for free** (or **Start** if you're signed in).
2. Pick your datasets in **Apify datasets to merge**, in the order to read them (or paste links in **file URLs**).
3. In **Dedupe by**, name the fields that identify a row, such as `email` or `name, city`. Leave it empty to remove only
   rows identical in every field.
4. Optional: choose what to ignore when comparing, and **which duplicate to keep**.
5. Click **Start** and open the **Output** tab (export as JSON, CSV or Excel). The `SUMMARY` record has the counts per
   source.

### Example: one exchange rate per currency

The pre-filled input (two public CSV files, the newer one first, so the newer rate is the one kept):

```json
{"fileUrls": [
   "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300",
   "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2024-12-31&page[size]=300"],
 "dedupeBy": ["country_currency_desc"]}
```

A typical job, leads from two scrapes, matched on the website however it was written, with blanks filled in:

```json
{"datasetIds": ["<January's dataset>", "<February's dataset>"],
 "dedupeBy": ["website"],
 "normalize": ["trim", "lowercase", "url"],
 "keep": "merge"}
```

### Ready-to-run examples

Each example opens with the input already filled in. Run it as it is, or change the input first.

- **[Merge CSV files and remove duplicates by a key column](https://apify.com/humble-echidna/dataset-merge/examples/merge-csv-files-remove-duplicates)**: One table with one row per key, also saved as a CSV.
- **[Combine runs and keep only the newest row per key](https://apify.com/humble-echidna/dataset-merge/examples/keep-newest-row-per-key)**: Read oldest to newest and keep each record's latest values.

### Remove duplicates across runs of a scraper

Listing every dataset by hand works once. For a scraper on a schedule, let the actor remember instead:

1. Open the scraper or its saved task, go to the **Integrations** tab and add **Apify actor** → **Merge & Dedupe
   Datasets**, triggered when a run **succeeds**.
2. Set `"datasetIds": ["{{resource.defaultDatasetId}}"]` (Apify fills in the finished run's dataset). This line is
   needed: Apify gives a limited-permissions run access only to the datasets named in that field, so the dataset in
   the integration's own payload can't be read, and a run with the field empty tells you to add it.
3. Set **Dedupe by**, and one of:
   - **Only rows not returned by earlier runs**: each run's dataset holds just the rows that are new, ready to send on.
   - **Master dataset**, e.g. `all-leads`: the same, and new rows are also added to that one dataset, which grows into
     your full list without duplicates.

What's remembered is a fingerprint of each key, not your data, in a key-value store in your own account. A row is
remembered only once it's in your dataset, so rows a cost limit cut off come back next run. Changing **Dedupe by** or
what's ignored starts a new memory, because the same keys no longer mean the same thing.

### Input

| Field | What it does |
|---|---|
| **Apify datasets to merge** | Your datasets, read in order. Read-only, exactly these. Up to 200 with the files. |
| Or: file URLs | CSV/TSV, JSON, JSON Lines or Excel (first sheet) links, read after the datasets. |
| **Dedupe by** | `email`, or `name, city`; dots reach nested fields. Empty or `*`: only identical rows. |
| Ignore these differences | `trim` (the default), `lowercase`, `accents`, `url`, `digits`. The output keeps the original values. |
| Which duplicate to keep | `first` (the default), `last`, `mostComplete` or `merge`. |
| Only rows not returned by earlier runs | Skips keys earlier runs returned, free. |
| Memory name | Shares one memory between several tasks or scrapers. |
| Master dataset | A dataset that keeps every unique row across runs; turns the option above on. |
| Add a field saying where each row came from | Adds `mergeSource`. |
| Also save as a file | `OUTPUT.csv` or `OUTPUT.jsonl`, free. |
| Max rows per run | Cap the rows written. |

**`url`** compares web addresses without the scheme, `www.`, default port, fragment, trailing slash and tracking
parameters (`utm_*`, `gclid`, `fbclid` and similar), so `https://www.acme.com/?utm_source=x` and `http://acme.com` are one
row. **`digits`** keeps only the digits, so `+1 (555) 010-2000` and `15550102000` match; use it when the key is a phone
number, since it would empty a name. A row with every dedupe field empty is always kept, since nothing says it repeats.

### Output

One dataset row per unique key: the row as it was, with its own fields (plus `mergeSource` if you asked for it). The
dataset can be downloaded as CSV, JSON, Excel, XML or HTML, or passed on to the next actor.

The run's `SUMMARY` record has the rows read and written, duplicates removed, rows already returned by earlier runs, and
the same per source:

```json
{"rowsRead": 372, "rowsWritten": 196, "duplicatesRemoved": 176, "alreadyReturnedInEarlierRuns": 0,
 "sources": [{"source": "https://api.fiscaldata.treasury.gov/…/rates_of_exchange", "rowsRead": 174, "emitted": 167, "duplicates": 7, "alreadyReturned": 0, "format": "csv", "error": null},
             {"source": "https://api.fiscaldata.treasury.gov/…/rates_of_exchange #2", "rowsRead": 198, "emitted": 29, "duplicates": 169, "alreadyReturned": 0, "format": "csv", "error": null}]}
```

File links appear here, in the log and in `mergeSource` without their query string or password, so a signed link's
secret stays in your input alone. In the optional CSV, text a spreadsheet would run as a formula (starting with `=`,
`+`, `-`, `@`, a tab or a line break, even after spaces) gets an apostrophe in front so it opens as text; plain numbers and the dataset itself
are unchanged.

#### Can I use Merge & Dedupe Datasets from an AI agent (MCP)?

Yes, through [Apify's MCP server](https://docs.apify.com/platform/integrations/mcp), from Claude, ChatGPT, Cursor or
any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor
search); your client signs you in to Apify:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=humble-echidna/dataset-merge"
    }
  }
}
```

To use an [Apify API token](https://console.apify.com/settings/integrations) instead of signing in, add
`"headers": {"Authorization": "Bearer <APIFY_TOKEN>"}` next to `url`.

### Who it's for

Anyone who runs the same scraper repeatedly or splits a big job across runs: lead lists built up week by week, job or
product monitors that should alert only on what's new, and one table from several scrapers' results.

### Why this one?

- **No permission errors or approval screens.** The form's dataset picker gives the run read access to exactly the
  datasets you choose; no account-wide access is asked for.
- **A deleted dataset doesn't lose the merge.** One source that can't be read is reported with the reason, and the rest
  are still merged.
- **Compare the way your data really differs.** Case, spaces, accents, URL tracking parameters and phone formatting, no
  code to write.
- **Keep the best copy.** First, last, most complete, or combined, so a later scrape can fill the blanks of an earlier
  one.
- **Across runs without listing them.** Only new rows, or a master list, from one integration.
- **Merged rows stay within limits.** A combined row that would pass 5 MB keeps its previous version, and is counted.
- **You pay for the result.** Duplicates are free, and there's no usage bill.
- **Fails fast on a typo.** If no row has the dedupe field, the run stops before writing or charging anything and lists
  the fields the first row has.

### Switching from another actor?

Most merge and dedupe actors take the same ideas under other names, so moving takes minutes:

| Elsewhere | Here |
|---|---|
| A list of dataset ids | **Apify datasets to merge** (`datasetIds`); the same ids work through the API |
| `fields` (what must be unique) | **Dedupe by** (`dedupeBy`); no fields means identical rows are removed, not everything kept |
| A dataset of earlier items to filter against | **Only rows not returned by earlier runs** (`onlyNew`), or **Master dataset** |
| An output dataset name | **Master dataset** (`masterDataset`) |
| Add the dataset id to each row | **Add a field saying where each row came from** (`addSourceField`) |
| JavaScript transform functions | Not here: chain [Dataset Transformer](https://apify.com/humble-echidna/dataset-transform) to filter, rename and reshape, or [AI for Every Row](https://apify.com/humble-echidna/ai-rows) |

**Loading every run of an actor by its name isn't here, on purpose:** that requires an actor with full access to your
account, since Apify doesn't let a limited-permissions run list your runs. Use an integration with **Master dataset**
from now on; for your past runs, pick their datasets once.

### Limits

- Up to 200 datasets and files per run, and 2,000,000 rows read per run.
- `first` holds only keys in memory. `last`, `mostComplete` and `merge` also hold one row per key until the end. Both are
  limited to 60% of the run's memory; the status tells you to give the run more memory if it's not enough (about 300,000
  rows of 2 KB at the default 1 GB with these three modes).
- Files: up to 200 MB (after decompression), public web addresses on ports 80 and 443; Excel's first sheet only.
- A single row can be up to 5 MB; larger ones are skipped and counted.
- It finds exact matches after the differences you ignore, not similar-looking text (`Acme Inc` and `Acme Incorporated`
  are two rows).
- Fingerprints are 64 bits: a false match between two different keys is about a 1-in-a-million chance at 6 million
  distinct keys, so don't use it where a single lost row among millions matters.

### FAQ

#### Why is a dataset reported as unreadable?

Pick it in the form's dataset picker, or pass its id in `datasetIds`: those are what give the run access. A dataset
named anywhere else in the input can't be read by a limited-permissions run. It must also be in your account or shared
with you, and not expired (unnamed datasets are deleted after your retention period).

#### Why didn't my duplicates go away?

Check that **Dedupe by** names the fields as written (case-sensitive; dots for nested fields), and whether the values
differ in something you aren't ignoring: compare two rows, then add `lowercase` or `url`. The `SUMMARY` counts rows with
no key, which are always kept.

#### Which order are rows in?

The order they were first read, datasets first and then files, each in its listed order. With `last`, `mostComplete` or
`merge`, the kept row stays where its key first appeared.

#### Can I start the memory again?

Yes: delete the `seen-…` records in your `dataset-merge-memory` key-value store, or use a new **Memory name**.

#### Is it legal to merge datasets and files with it?

It only processes data you choose: your own Apify datasets, and files at addresses you supply, fetched politely
(robots.txt honoured, honest User-Agent, public addresses on ports 80 and 443). Make sure you're allowed to use what you
point it at. The example data is the US Treasury's Fiscal Data API, which is "offered free, without restriction" for
commercial and non-commercial use.

#### Something that used to work now fails. Why?

The run log and the status say what went wrong. Please open an issue with the input you used.

### Related actors

| Actor | Use it when |
|---|---|
| [Dataset Transformer](https://apify.com/humble-echidna/dataset-transform) | One source to filter, sort, rename or flatten, as well as dedupe. |
| [Dataset Diff](https://apify.com/humble-echidna/dataset-diff) | You want what changed or was removed between two runs, not only what's new. |
| [Join Datasets](https://apify.com/humble-echidna/dataset-join) | You want to add columns from another dataset by a key, not stack rows. |
| [Google Sheets Import & Export](https://apify.com/humble-echidna/google-sheets) | The merged rows should end up in a sheet. |

### Feedback and support

Found a bug, or need a comparison that isn't here? Open an issue on the **Issues** tab with the input you used.

### Versions

Current version: **0.1**. See the Changelog tab for what changed in each version.

# Changelog

This Actor's version history is a separate document: https://apify.com/humble-echidna/dataset-merge/changelog.md

# Actor input Schema

## `datasetIds` (type: `array`):

The datasets to combine, read in this order: ids or names of your own Apify datasets, e.g. the runs of a scraper (or pick them in the form). Up to 200 sources. Read-only access to exactly these, nothing else in your account. In an integration use `{{resource.defaultDatasetId}}`: Apify lets the run read only datasets named here. Reading runs by actor isn't possible under limited permissions, so list the datasets, or use masterDataset.

## `fileUrls` (type: `array`):

Direct links to CSV/TSV, JSON (a list of objects), JSON Lines or Excel (.xlsx, first sheet) files on public web addresses (http/https, ports 80 and 443), up to 200 MB each; read after the datasets. The default is the US Treasury's exchange rates on 2025-12-31 and 2024-12-31, the example's data; it's ignored as soon as a dataset is given.

## `dedupeBy` (type: `array`):

Fields whose combined values identify a row, one per line or comma-separated, e.g. `email` or `name, city`. Dots reach nested fields (`address.postalCode`, `offers.0.url`). Names are case-sensitive. Empty, or `*`: only rows identical in every field are duplicates. A row with all these fields empty is always kept. The prefill is the example's key.

## `normalize` (type: `array`):

How key values are compared, any of: `trim` (the default: spaces at the ends and repeated spaces), `lowercase` (case), `accents` (é = e), `url` (web addresses compared without scheme, www, default port, fragment, trailing slash and tracking parameters such as utm\_\* and gclid), `digits` (only digits count, for phone numbers). The output keeps the original values.

## `keep` (type: `string`):

`first` (the default): the first row with the key, written as it's read, so memory holds only keys. `last`: the last one. `mostComplete`: the one with the most filled-in fields (a tie keeps the earlier). `merge`: the first row with its empty fields filled in from its duplicates. The last three hold one row per key in memory and write at the end.

## `onlyNew` (type: `boolean`):

Default false. When true, keys this actor returned in earlier runs are remembered in a key-value store in your account and skipped, free. For a scraper's integration this means each run gives only rows never seen before. The memory belongs to memoryName, else the saved task, else the scraper that wrote the first dataset, and starts afresh if dedupeBy or normalize change.

## `memoryName` (type: `string`):

Optional name for what onlyNew remembers, so several tasks or scrapers can share one memory, e.g. `all-leads`. Letters, digits and hyphens, at most 63 characters. Default: none, the memory follows the task or scraper.

## `masterDataset` (type: `string`):

Optional name of a dataset to keep a running list in, e.g. `all-leads`: created on first use, new unique rows are added on every run, and rows it already holds are skipped (this turns on onlyNew). Letters, digits and hyphens, at most 63 characters. Rows also go to the run's own dataset. If your account has that name from elsewhere, `humble-echidna-<name>` is used.

## `addSourceField` (type: `boolean`):

Default false. When true, each row gets `mergeSource`: `dataset:<id>` or the file's address it was read from (for `merge`, the first row's source).

## `exportFormat` (type: `string`):

`none` (the default), `csv` or `jsonl`: also save the output as `OUTPUT.csv` or `OUTPUT.jsonl` in the run's key-value store, up to 100 MB, free of charge. In the CSV, text a spreadsheet would run as a formula gets an apostrophe in front.

## `maxResults` (type: `integer`):

Stop after writing this many rows, e.g. 100. Minimum 1; leave empty (the default) for no limit. The run also stops cleanly at the maximum cost per run you set in the run options, whichever comes first. With onlyNew, rows not written are returned by the next run.

## Actor input object example

```json
{
  "fileUrls": [
    "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300",
    "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2024-12-31&page[size]=300"
  ],
  "dedupeBy": [
    "country_currency_desc"
  ],
  "normalize": [
    "trim"
  ],
  "keep": "first",
  "onlyNew": false,
  "addSourceField": false,
  "exportFormat": "none"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `runStats` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "fileUrls": [
        "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300",
        "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2024-12-31&page[size]=300"
    ],
    "dedupeBy": [
        "country_currency_desc"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("humble-echidna/dataset-merge").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "fileUrls": [
        "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300",
        "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2024-12-31&page[size]=300",
    ],
    "dedupeBy": ["country_currency_desc"],
}

# Run the Actor and wait for it to finish
run = client.actor("humble-echidna/dataset-merge").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "fileUrls": [
    "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300",
    "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2024-12-31&page[size]=300"
  ],
  "dedupeBy": [
    "country_currency_desc"
  ]
}' |
apify call humble-echidna/dataset-merge --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,humble-echidna/dataset-merge"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TuhZWSa5GIp55bOaS/builds/1NgI4LotkqXsh83Gk/openapi.json
