# Merge & Dedup Datasets, CSV, JSON & Excel Files (`wrentools/dataset-merge-dedup`) Actor

Merge Apify datasets, CSV, JSON, JSON Lines and Excel files into one clean dataset. Remove duplicates on any fields, ignoring case, spaces and URL variants; keep the first, last or most complete row; filter, pick and rename fields. No code. Duplicates and filtered rows are free.

- **URL**: https://apify.com/wrentools/dataset-merge-dedup.md
- **Developed by:** [WrenTools](https://apify.com/wrentools) (community)
- **Categories:** Developer tools, Integrations
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 row writtens

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Merge & Dedup Datasets, CSV, JSON & Excel Files

Combine the results of several scraper runs, lead lists or exports into one clean dataset. Give it your Apify
datasets, links to CSV, JSON, JSON Lines or Excel files, or rows pasted as JSON; it merges them, removes the
duplicates on the fields you choose, filters, keeps only the fields you want under the names you want, and writes the
result to the run's dataset, from which Apify exports CSV, Excel, JSON or XML.

No code to write: every step is an input field. You pay for the rows written; duplicates, filtered rows and failed
sources are not charged as written rows. Rows read from Apify datasets you pick cost $0.001 per started 1,000 (see Pricing).

### What you get

- **Merge** any mix of sources in one run: up to 100 Apify datasets, up to 100 file URLs, and inline rows.
  Duplicates are found across all of them.
- **Deduplicate** on one or more fields (dotted paths such as `address.city` work), or on whole rows (`["*"]`):
  - **forgiving matching**, each switchable: surrounding spaces (on by default), upper and lower case (on by
    default), runs of spaces, and **URL variants** (`http`/`https`, `www.`, trailing slash, `#fragment`, `utm_*`,
    `gclid`, `fbclid` and other tracking parameters, query order). `5`, `"5"` and `5.0` are the same value;
  - **keep the first, the last, or the most complete** duplicate (the one with the most non-empty fields);
  - rows whose dedup fields are all empty are kept, treated as one duplicate group, or dropped: your choice.
- **Filter** with 16 operators (equals, contains, starts/ends with, in a list, greater/less than, regular
  expression, empty/not empty, and their negatives), combined with AND or OR. Numbers are read from text such as
  `"$1,234.50"`.
- **Shape** the output: write only some fields (in your order), drop fields, rename fields, and add a field saying
  which source each row came from.
- **A SUMMARY** for every run: per source, the rows read, filtered out, removed as duplicates and written, and why a
  source failed.
- **Flat memory on large inputs:** rows are streamed and written in batches of 1,000; deduplication keeps only a
  16-byte fingerprint per distinct key, not the rows. Datasets are read 5,000 rows at a time, and when you pick
  output fields only those fields are downloaded.

Failed sources are reported in the SUMMARY with a reason and write nothing: `dataset-not-found`,
`dataset-not-readable`, `dataset-read-failed`, `fetch-failed`, `robots-disallowed`, `file-too-large`,
`unsupported-file`, `invalid-json`, `invalid-csv`, `invalid-xlsx`. A file is parsed completely before any of its
rows is used, so a damaged file costs nothing.

### Respectful by design

- **Your datasets are read only if you pick them.** The Actor runs with Apify's limited permissions: it can read
  exactly the datasets you choose in the input, for reading, and nothing else in your account.
- **Only the files you link are downloaded**, one request each. The site's robots.txt is read first and obeyed, and
  downloads from one site are spaced (500 ms by default). No proxy, no browser, no login.
- Only public internet addresses are fetched: a URL or redirect into a private, loopback or link-local network is
  refused.
- Use files and data you have the right to process.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `datasetIds` | dataset\[] | | your Apify datasets (picker) |
| `fileUrls` | string\[] | | links to `.csv`/`.tsv`, JSON array, JSON Lines, `.xlsx` (first sheet) files |
| `items` | array | | rows as JSON objects; other values become `{"value": ...}` |
| `dedupFields` | string\[] | | fields that together identify a duplicate; `["*"]` = whole row; empty = merge only |
| `keep` | `first` | `last` | `mostComplete` | `first` | which duplicate survives; kept rows stay in reading order |
| `trimValues` | boolean | `true` | ignore leading and trailing spaces |
| `ignoreCase` | boolean | `true` | ignore upper and lower case |
| `collapseWhitespace` | boolean | `false` | treat runs of spaces, tabs and line breaks as one space |
| `normalizeUrls` | boolean | `false` | compare URL values loosely (above) |
| `emptyKeyRows` | `keep` | `dedup` | `drop` | `keep` | rows whose dedup fields are all empty |
| `filters` | array | | `[{"field": "country", "op": "in", "value": ["US", "CA"]}, {"field": "score", "op": "gte", "value": 50}]` |
| `filterMode` | `all` | `any` | `all` | AND or OR |
| `filtersCaseSensitive` | boolean | `false` | |
| `fields` | string\[] | | write only these fields, in this order; a missing one is written as `null` |
| `omitFields` | string\[] | | top-level fields to leave out |
| `renameFields` | object | | `{"old name": "new name"}`, applied last |
| `addSourceField` | string | | add a field with each row's source (`dataset:<id>`, the file URL, or `items`) |
| `maxOutputRows` | integer | `0` | stop after this many rows (0 = no limit) |
| `fileFormat` | `auto` | `csv` | `json` | `jsonl` | `xlsx` | `auto` | detected from the bytes, file name and content type |
| `csvDelimiter` | `auto` | `comma` | `semicolon` | `tab` | `pipe` | `auto` | detected from the header row |
| `maxFileSizeMb` | integer | `50` | 1-200 |
| `requestDelayMs` | integer | `500` | pause between downloads from one site |

The order of work is: read (datasets, then files, then items, each in input order), filter, deduplicate, then pick,
drop and rename. Filters and dedup fields use the **original** field names. CSV values stay text; text in UTF-8,
UTF-16 (with a BOM) and Windows-1252 is read. Excel cells give their saved values, dates as ISO 8601 text.

### Output

Each row written is one dataset item, exactly as read (plus your shaping); nothing is wrapped around it. The run's
key-value store holds `SUMMARY`:

```json
{
  "rowsRead": 9, "rowsFiltered": 0, "duplicatesRemoved": 3, "emptyKeyRowsDropped": 0,
  "rowsWritten": 6, "rowsNotWritten": 0, "distinctKeys": 6, "sourcesRequested": 2, "sourcesFailed": 0,
  "truncated": false, "stoppedReason": null, "dedupFields": ["email"], "keep": "mostComplete",
  "sources": [
    {"source": "https://.../leads-crm.csv", "type": "file", "status": "ok", "format": "csv", "encoding": "utf-8",
     "rowsRead": 5, "rowsFiltered": 0, "duplicatesRemoved": 1, "emptyKeyRowsDropped": 0, "rowsWritten": 4,
     "rowsNotWritten": 0, "error": null}
  ]
}
```

A source's `status` is `ok`, `failed` (nothing read), `partial` (a dataset whose reading failed part way; the rows
before it were used), `stopped` (the run stopped while reading it) or `skipped` (the run stopped before it).

### Pricing

Pay per event, no start fee:

| Event | Price | Per 1,000 |
|---|---|---|
| `row-written` (one row written to the output dataset) | $0.0002 | $0.20 |
| `dataset-rows-read` (each started 1,000 rows read from Apify datasets you picked, counted over the run) | $0.001 | $0.001 ($1 per million) |

Rows filtered out, removed as duplicates, or from a source that failed are **not charged as written**. Rows read from
your Apify datasets carry the small read charge whether or not they are kept, because the platform bills every dataset
row read; rows from file URLs and pasted items carry none. Merging 50,000 rows from file URLs into 35,000 unique ones
costs $7.00; reading the same 50,000 rows from datasets adds $0.05 (50 blocks of 1,000). When you set a maximum charge for a run, the Actor
reads and writes only what it covers and stops; the SUMMARY says `"truncated": true` and why.

### Limits

- With `keep` set to `last` or `mostComplete`, every source is read before the first row is written (the rows wait in
  a temporary file on the run's disk), so very large inputs need a longer run timeout. The run stops 30 seconds before
  its timeout (90 s while still reading in those modes) and saves the SUMMARY; rows it could not write are counted in
  `rowsNotWritten` and not charged.
- A file is held in memory while it is read: allow about 4 times its size (a 50 MB CSV needs about 200 MB on top of
  the default 1 GB run's other needs). Datasets have no such limit.
- Memory grows with the number of distinct keys: about 80 bytes each when keeping the first, 120 when keeping the
  last or most complete (measured on 1 million keys), so 1 million distinct keys need about 80-120 MB.
- Legacy `.xls`, Google Sheets links and compressed (`.gz`, `.zip`) files are not read.
- Numbers in text are read with `,` as a thousands separator.

### Licences

Python's standard library, the Apify SDK (Apache-2.0), and openpyxl with et-xmlfile (both MIT) for Excel files. No
third-party API: nothing leaves the Actor except the downloads you asked for and the reads of the datasets you
picked.

# Actor input Schema

## `datasetIds` (type: `array`):

Your Apify datasets to merge, for example the results of several scraper runs. Only the datasets you pick here are read, and only for reading.

## `fileUrls` (type: `array`):

Links to files you are allowed to use: CSV or TSV with a header row, a JSON array, JSON Lines, or an .xlsx workbook (first sheet, first row as header). Each file is downloaded once; the site's robots.txt is respected.

## `items` (type: `array`):

Rows pasted in directly: a JSON array of objects. Read after the datasets and files.

## `dedupFields` (type: `array`):

Rows whose values in all these fields match are duplicates; only one is kept. Use dotted paths for nested fields (address.city). \["\*"] compares whole rows. Leave empty to merge without deduplicating.

## `keep` (type: `string`):

Kept rows stay in the order they were read. 'last' and 'most complete' read every source before writing.

## `trimValues` (type: `boolean`):

' ann@example.com ' matches 'ann@example.com'.

## `ignoreCase` (type: `boolean`):

'Ann@Example.com' matches 'ann@example.com'. Turn off for case-sensitive codes or IDs.

## `collapseWhitespace` (type: `boolean`):

'Harbor  Supply' matches 'Harbor Supply'; tabs and line breaks count as spaces.

## `normalizeUrls` (type: `boolean`):

In values that are URLs, ignore http/https, 'www.', a default port, a trailing slash, the #fragment, tracking parameters (utm\_\*, gclid, fbclid ...) and the order of query parameters.

## `emptyKeyRows` (type: `string`):

What to do with rows in which every dedup field is empty or missing.

## `filters` (type: `array`):

Keep only rows that match: a list of {"field": "country", "op": "equals", "value": "US"}. Operators: equals, notEquals, contains, notContains, startsWith, endsWith, in, notIn (value is a list), gt, gte, lt, lte (numbers), matches, notMatches (regular expression), isEmpty, isNotEmpty. Filters run before deduplication, on the original field names.

## `filterMode` (type: `string`):

Combine the filters with AND (all) or OR (any).

## `filtersCaseSensitive` (type: `boolean`):

Off: 'us' matches 'US' in text filters and regular expressions.

## `fields` (type: `array`):

Write only these fields, in this order (dotted paths allowed; a missing field is written as null). Empty writes every field. For datasets, only the fields needed are downloaded.

## `omitFields` (type: `array`):

Top-level fields to leave out of the output.

## `renameFields` (type: `object`):

Old name to new name, e.g. {"e-mail": "email", "Company Name": "company"}. Applied last, to the written rows.

## `addSourceField` (type: `string`):

If set, each written row gets a field of this name saying where it came from (dataset:<id>, the file URL, or 'items').

## `maxOutputRows` (type: `integer`):

Stop after writing this many rows (0 = no limit). Rows are charged as written.

## `fileFormat` (type: `string`):

Applies to every file URL. Detection reads the file's first bytes, its name and its content type.

## `csvDelimiter` (type: `string`):

The character between CSV columns.

## `maxFileSizeMb` (type: `integer`):

Larger downloads are stopped and reported as file-too-large, without charge. A file needs about 4 times its size in memory.

## `requestDelayMs` (type: `integer`):

Minimum pause between two downloads from the same site.

## Actor input object example

```json
{
  "fileUrls": [
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv",
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-signups.jsonl"
  ],
  "dedupFields": [
    "email"
  ],
  "keep": "mostComplete",
  "trimValues": true,
  "ignoreCase": true,
  "collapseWhitespace": false,
  "normalizeUrls": false,
  "emptyKeyRows": "keep",
  "filterMode": "all",
  "filtersCaseSensitive": false,
  "addSourceField": "_source",
  "maxOutputRows": 0,
  "fileFormat": "auto",
  "csvDelimiter": "auto",
  "maxFileSizeMb": 50,
  "requestDelayMs": 500
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "fileUrls": [
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv",
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-signups.jsonl"
    ],
    "dedupFields": [
        "email"
    ],
    "keep": "mostComplete",
    "addSourceField": "_source"
};

// Run the Actor and wait for it to finish
const run = await client.actor("wrentools/dataset-merge-dedup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "fileUrls": [
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv",
        "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-signups.jsonl",
    ],
    "dedupFields": ["email"],
    "keep": "mostComplete",
    "addSourceField": "_source",
}

# Run the Actor and wait for it to finish
run = client.actor("wrentools/dataset-merge-dedup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "fileUrls": [
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-crm.csv",
    "https://api.apify.com/v2/key-value-stores/uM2zEXblvgf0s1cES/records/leads-signups.jsonl"
  ],
  "dedupFields": [
    "email"
  ],
  "keep": "mostComplete",
  "addSourceField": "_source"
}' |
apify call wrentools/dataset-merge-dedup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,wrentools/dataset-merge-dedup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/OgxSEScI8fQM36Gpb/builds/T9Gn5fJt2sb8NQC1n/openapi.json