# Merge Datasets & Remove Duplicates: Combine Runs, CSVs, Sheets (`nerolabs/dataset-merge-dedupe`) Actor

Returns one de-duplicated list from many Apify datasets, CSV, Excel or JSON files and Google Sheets, with blanks filled from the copies and each row's source labelled. Inputs: datasetIds, fileUrls, dedupKeys. Charged per merged row written ($0.001). Agent-ready via x402 and MCP.

- **URL**: https://apify.com/nerolabs/dataset-merge-dedupe.md
- **Developed by:** [Adam Pearce](https://apify.com/nerolabs) (community)
- **Categories:** Developer tools, Automation, Lead generation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 merged rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Merge datasets into one list and remove the duplicates

Ran the same scraper once per city, keyword or week and now have ten separate datasets full of overlap? **Merge Datasets & Remove Duplicates** takes **any number of Apify datasets, CSV or Excel files and Google Sheets**, stacks them into one list, removes the records that appear in more than one of them, fills in the blanks from the copies it removed, and tells you which list every row came from. No code, no spreadsheet formulas.

It will:

- **Merge** many datasets, files and Google Sheets in one run (up to 100 sources and 300,000 rows read per run)
- **Remove duplicates across them** by the fields you choose (`phone`, `website`, `placeId`, `url`...) or by the whole row
- **Match sensibly**: `0117 496 0012` matches `01174960012`, and `https://www.acme.com/` matches `acme.com`
- **Fill blanks from the duplicates**: an email found in one run and a rating found in another end up on the same row
- **Label every row** with the list it came from, every list it appeared in, and how many copies were merged
- **Export** a ready-to-open CSV and Excel file, append to a named dataset, or POST the result to a webhook

### Why use this?

Splitting a big scrape into many smaller runs is normal (one run per city, per search term, per day), but it leaves you with many datasets and a lot of overlap: the same business shows up in two neighbouring towns, the same product in two categories, the same job in two searches. Merging those by hand means exporting each one, pasting them together and removing duplicates in a spreadsheet.

- **Lead generation**: combine Google Maps or directory runs from many areas into one de-duplicated lead list, keeping the most complete record for each business
- **Ecommerce and price tracking**: merge category or keyword runs into one product list without double counting
- **Job and listing data**: combine searches into one feed with each listing once
- **Combining sources**: merge an Apify dataset with a CSV export from your CRM and a shared Google Sheet in one step
- **AI agents and pipelines**: one deterministic call that turns several result sets into one clean list

### How to use it

1. Run your scraper as many times as you need. Each run's results are a dataset.
2. Open this Actor and pick those datasets in **Datasets to merge**, and/or paste links in **Files or Google Sheets to merge**.
3. In **Match duplicates on these fields**, enter the field that identifies the same record, for example `phone`, `website`, `placeId` or `url`. Leave it empty to compare whole rows.
4. Run it. Download the CSV or Excel file from the **Output** tab, or read the merged dataset through the API.

Try it with the example input: two small sample lead lists with three businesses in both, written in different phone formats. The result is 9 unique businesses from 12 rows.

### Input

| Field | What it does |
|---|---|
| Datasets to merge | Any number of Apify datasets, picked with the dataset picker |
| Files or Google Sheets to merge | CSV, TSV, Excel, JSON or JSON Lines links, or Google Sheet links, one per line |
| Inline data | Optional JSON array, merged in as one more list |
| Match duplicates on these fields | Field names that identify the same record; empty means compare whole rows |
| How strictly to match | Normalized (default), Exact, Fuzzy (near matches, up to 5,000 rows) or None (just stack the lists) |
| Which duplicate to keep | The most complete row (default), the first one seen or the last one seen |
| Fill blanks from the duplicates | On by default; never overwrites a value that is already there |
| Add source columns | On by default: `_source`, `_foundInSources`, `_duplicatesMerged` |
| Max output rows, named dataset, download files, webhook | Optional extras |

Example input:

```json
{
    "datasetIds": ["aBcD1234efGh5678i", "jKlM9012noPq3456r"],
    "fileUrls": ["https://docs.google.com/spreadsheets/d/your-sheet-id/edit"],
    "dedupKeys": ["phone"],
    "dedupMode": "normalized"
}
```

Lists are read in the order given: datasets first, then files, then inline data. Every column from every list is kept; a row simply has no value in a column its list did not have.

### Output

One row per unique record, with all of its original fields plus three source columns:

```json
{
    "name": "Harbour Plumbing Ltd",
    "phone": "01174960012",
    "website": "harbour-plumbing.example.com",
    "email": "jobs@harbour-plumbing.example.com",
    "city": "Bristol",
    "rating": "4.8",
    "_source": "sample-leads-run-2.csv",
    "_foundInSources": "sample-leads-run-1.csv | sample-leads-run-2.csv",
    "_duplicatesMerged": 1
}
```

| Column | Meaning |
|---|---|
| `_source` | The list the kept row came from (a dataset's name or ID, or a file's name) |
| `_foundInSources` | Every list the record appeared in, separated by `\|` |
| `_duplicatesMerged` | How many copies were merged into this row (0 means it was unique) |

The run also saves:

- `merged-data.csv` and `merged-data.xlsx`, ready-to-open files of the merged list
- `MERGE_REPORT`: rows read from each source, any source that could not be read and why, duplicates removed, blanks filled, the full column list and any warnings

You can also download the dataset in JSON, CSV, Excel, XML or HTML from the Output tab.

### How much does it cost to merge datasets?

You pay per merged row written, plus a small fee per download file:

| Event | Price (Free plan) | Bronze | Silver | Gold and above |
|---|---|---|---|---|
| Merged row written | $0.001 | $0.0009 | $0.0008 | $0.0007 |
| Download file (CSV or Excel) | $0.01 | $0.009 | $0.008 | $0.007 |
| Webhook delivered (2xx only) | $0.02 | $0.018 | $0.016 | $0.014 |

Rows read, duplicates removed and file downloads are never charged. **Example:** merging 10 city runs of 2,000 rows each, where 20% of rows are duplicates, writes 16,000 unique rows: $16.00 on the Free plan or $11.20 on Gold, plus $0.02 for the CSV and Excel files. Set **Max output rows** or a spending limit on the run to cap the cost; the Actor stops writing at your limit and says so.

### Tips

- **Pick the field that really identifies a record.** For Google Maps use `placeId` (or `phone`), for web pages `url`, for products an ID or SKU. Whole-row matching misses duplicates whose ratings or review counts changed between runs.
- **Use several fields together** when one is not unique, for example `name` and `city`.
- **Normalized matching** ignores case, spaces, punctuation, accents, `http(s)://` and `www.`. It does not convert country codes, so `+44 117...` and `0117...` are different values; use Fuzzy matching for lists under 5,000 rows if that matters.
- **Only new records?** If you want just the records that were not in last week's run, use [Dataset Diff & Change Detector](https://apify.com/nerolabs/dataset-diff-detector). To merge two lists side by side on a shared key (like a VLOOKUP), use [Dataset Join & Merge](https://apify.com/nerolabs/dataset-join-merge). To clean up one list (emails, phones, nested fields), use [Dataset Cleaner & Exporter](https://apify.com/nerolabs/dataset-cleaner-exporter).
- **Pipelines:** call it from a webhook or the API after your last scraper run finishes, with `outputDatasetName` to keep one growing master list.

### For AI agents

Pass `datasetIds` (array of dataset IDs) and/or `fileUrls` (array of links), plus `dedupKeys` (array of field names). The merged rows are the run's default dataset; `MERGE_REPORT` in the key-value store has the counts. Charged per merged row written ($0.001), per download file ($0.01) and per delivered webhook ($0.02). Works with limited permissions, through the Apify MCP server and with x402 payments.

### FAQ

**Does it change my original datasets or files?** No. It only reads them and writes a new dataset.

**What if one of the datasets or links cannot be read?** It is skipped, the reason is written to `MERGE_REPORT` and the run log, and the rest are still merged. The run fails only if no source can be read.

**Can it merge every run of a scraper automatically?** Pick the datasets you want, or send their IDs through the API. Reading all of an Actor's runs on its own would need full access to your account, which this Actor deliberately does not ask for.

**Is there a size limit?** Up to 300,000 rows read per run across all sources, and 100 MB per file. Both the CSV and the Excel file cover the whole merged list.

**Something not right?** Open an issue on the Issues tab and I will look at it the same day. If this saved you stitching runs together by hand, a review on the Store page helps a lot.

# Actor input Schema

## `datasetIds` (type: `array`):

Pick any number of Apify datasets to combine into one list, for example the results of the same scraper run once per city, keyword or date. Rows from all of them are stacked in the order you list them, then duplicates between them are removed. Declared this way so the Actor runs with limited permissions: it may read the datasets you pick and nothing else on your account.

## `fileUrls` (type: `array`):

Links to CSV, TSV, Excel (.xlsx, first sheet), JSON or JSON Lines files, or Google Sheets (normal sheet link, shared as 'Anyone with the link can view'). One link per line. Merged together with any datasets above. Downloads are never charged; up to 100 MB per file.

## `data` (type: `array`):

Optional. A JSON array of rows, merged in as one more list (labelled 'inline data').

## `fileFormat` (type: `string`):

Only needed if automatic detection reads a file link in the wrong format. Applies to every file link.

## `dedupKeys` (type: `array`):

Field (column) names that identify the same record across lists, for example 'phone', 'website', 'placeId' or 'url'. Rows with the same values in all of these fields count as duplicates. Leave empty to compare whole rows. Names must match exactly, including capital letters. Rows with no value in these fields are always kept.

## `dedupMode` (type: `string`):

Normalized treats '0117 496 0012' and '01174960012', or 'https://www.acme.com/' and 'acme.com', as the same. Fuzzy also matches small spelling differences, using the threshold below. None merges the lists without removing anything.

## `similarityThreshold` (type: `number`):

Only used with Fuzzy matching. 1.0 means identical; 0.9 allows about one character in ten to differ.

## `keepStrategy` (type: `string`):

When the same record appears more than once, which copy becomes the merged row. Lists are read in the order given, datasets first, then files, then inline data.

## `fillBlanksFromDuplicates` (type: `boolean`):

When on, any field that is empty in the kept row is filled in from the duplicates it replaced, so an email found in one run and a rating found in another end up on the same row. Values that are already filled are never overwritten.

## `addSourceInfo` (type: `boolean`):

Adds three columns to every row: \_source (the list the kept row came from: a dataset's name or ID, or a file's name), \_foundInSources (every list the record appeared in, separated by ' | ') and \_duplicatesMerged (how many copies were merged into it).

## `maxItems` (type: `integer`):

Optional cap on merged rows written (and charged). Up to 300,000 rows are read across all sources per run.

## `outputDatasetName` (type: `string`):

Optional. A dataset name (3 to 63 letters, digits or hyphens) that keeps growing across runs, for a scheduled pipeline. Not charged again.

## `exportFormats` (type: `array`):

Ready-to-open files of the merged list, saved in the run's key-value store. Charged per file.

## `webhookUrl` (type: `string`):

Optional. The merged rows (up to 5,000 inline) and download links are POSTed here as JSON when the run finishes, for Zapier, Make, n8n or your own API. Charged only on a 2xx response.

## Actor input object example

```json
{
  "fileUrls": [
    "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-1.csv",
    "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-2.csv"
  ],
  "fileFormat": "auto",
  "dedupKeys": [
    "phone"
  ],
  "dedupMode": "normalized",
  "similarityThreshold": 0.9,
  "keepStrategy": "most_complete",
  "fillBlanksFromDuplicates": true,
  "addSourceInfo": true,
  "exportFormats": [
    "csv",
    "xlsx"
  ]
}
```

# Actor output Schema

## `mergedRecords` (type: `string`):

One row per unique record across every list, with source columns.

## `csvFile` (type: `string`):

The merged list as a CSV file.

## `xlsxFile` (type: `string`):

The merged list as an Excel (.xlsx) file.

## `mergeReport` (type: `string`):

Rows read per source, duplicates removed, blanks filled, columns and any warnings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "fileUrls": [
        "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-1.csv",
        "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-2.csv"
    ],
    "dedupKeys": [
        "phone"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("nerolabs/dataset-merge-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "fileUrls": [
        "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-1.csv",
        "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-2.csv",
    ],
    "dedupKeys": ["phone"],
}

# Run the Actor and wait for it to finish
run = client.actor("nerolabs/dataset-merge-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "fileUrls": [
    "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-1.csv",
    "https://nerolabs-samples.nerolabs.workers.dev/sample-leads-run-2.csv"
  ],
  "dedupKeys": [
    "phone"
  ]
}' |
apify call nerolabs/dataset-merge-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nerolabs/dataset-merge-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YusjhqfDOZpE4dEsS/builds/Yvha0FI3mldDNsqd5/openapi.json