# Merge Datasets & Remove Duplicates: CSV, JSON, Apify (`phaux/dataset-merge-dedup`) Actor

Merge Apify datasets and CSV/JSON/JSONL files into one dataset and remove duplicates. Use to combine scraper outputs or exports. Input: datasetIds and/or fileUrls, dedupFields, optional renameFields, filters, keepFields. Output: merged deduplicated rows (optional \_source field) plus a SUMMARY.

- **URL**: https://apify.com/phaux/dataset-merge-dedup.md
- **Developed by:** [Phaux](https://apify.com/phaux) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Merge Datasets & Remove Duplicates: CSV, JSON, Apify

**Merge datasets** and **deduplicate** rows in one step: combine the output of several scraper runs or files into one clean table.

Merge **Apify datasets** and **CSV, JSON or JSON Lines files** into one clean dataset: remove duplicates by the fields you choose, rename columns, filter rows, and keep or drop fields. It streams large datasets page by page and keeps going when one source is broken, so a single bad input doesn't sink the whole run.

### Quick start for AI agents and API users

- **What it does:** Merge Apify datasets and CSV/JSON/JSONL files into one dataset and remove duplicates.
- **When to use it:** To combine scraper outputs or exports.
- **Example input** (the form's prefill; it finishes in under a minute):

```json
{
  "fileUrls": [
    "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
    "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
  ],
  "dedupFields": [
    "code"
  ],
  "renameFields": {
    "Code": "code",
    "Name": "name",
    "ISO3166-1-Alpha-2": "code",
    "official_name_en": "name"
  },
  "filters": [
    {
      "field": "code",
      "op": "startsWith",
      "value": "A"
    }
  ],
  "keepFields": [
    "code",
    "name"
  ],
  "addSourceField": true,
  "maxItems": 50
}
```

- **Example output:** one real dataset row from run `h384xdQMlabUo2zJf` (2026-10-23), with long fields shortened and empty fields left out. Fields not shown are listed in the output table below.

```json
{
  "code": "AF",
  "name": "Afghanistan",
  "_source": "https://raw.githubusercontent.com/datasets/country-list/main/data.csv"
}
```

- **Cost (pay per event, Free-plan prices):** Row $0.001. Failed items are not charged.
- **From an AI agent:** with the Apify MCP server, call `call-actor` with actor `phaux/dataset-merge-dedup`, or load it as its own tool with `https://mcp.apify.com?tools=phaux/dataset-merge-dedup`. Read results with `get-dataset-items`.

### What does Dataset Merge & Dedup do?

- **Merges** any number of datasets (picked in Console, or by ID via API) and file URLs, in the order you give them.
- **Removes duplicates** by one or more fields (`dedupFields`, nested dot paths like `address.city` work), or drops rows that are identical in every field. The first row wins; `1`, `1.0` and `"1"` count as the same value, and case can be ignored.
- **Renames fields first**, so differently named columns from different sources line up (`{"Company Name": "company", "company_name": "company"}`).
- **Filters rows** with simple conditions: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`, `contains`, `notContains`, `startsWith`, `endsWith`, `regex`, `in`, `notIn`, `exists`, `notExists`, `empty`, `notEmpty`.
- **Keeps or drops fields** (`keepFields` in a fixed column order, or `omitFields`), and can add a `_source` column.

### Why use it?

- **Reliable:** clear input checks before anything runs, a per-source error report (`SOURCE_ERRORS`) instead of a crashed run, and a time guard that saves results before the timeout.
- **Fast:** reads 5,000 rows per request and writes in batches of 1,000; merging 9,600 rows from three datasets took about 4 seconds in testing.
- **Limited permissions:** the run can read only the datasets you pick, read-only.
- **Pay only for rows kept:** duplicates and filtered-out rows are free.

Use cases: combining runs of the same scraper (daily, per region or per keyword), de-duplicating leads or products across runs, aligning CSV exports from different tools, and cleaning data before sending it to a spreadsheet, CRM or database.

### How to use it

1. Pick one or more datasets under **Apify datasets** and/or paste file URLs under **CSV, JSON or JSONL file URLs**. Empty input runs a small example with two public country-code CSV files.
2. Optionally map column names in **Rename fields**, set **Deduplicate by fields**, and add **Filters**.
3. Choose **Keep only these fields** for a tidy table.
4. Run it and download the merged dataset as CSV, Excel or JSON.

### Input

```json
{
  "fileUrls": [
    "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
    "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
  ],
  "renameFields": {"Code": "code", "Name": "name", "ISO3166-1-Alpha-2": "code", "official_name_en": "name"},
  "dedupFields": ["code"],
  "filters": [{"field": "code", "op": "startsWith", "value": "A"}],
  "keepFields": ["code", "name"],
  "addSourceField": true,
  "maxItems": 50
}
```

| Field | Type | Default | Description |
|---|---|---|---|
| `datasetIds` | array | — | Datasets to merge, in order. In Console pick them with the dataset picker; via API pass dataset IDs (or 'username~dataset-name'). Only these datasets can be read by the run (read-only access). |
| `fileUrls` | array | — | Public URLs of CSV (any delimiter), JSON (array, or object with an items/data list) or JSON Lines files. Each is downloaded once, up to maxFileSizeMb. |
| `dedupFields` | array | — | Rows with the same values in all these fields count as duplicates; the first one is kept. Dot paths work for nested fields (e.g. 'address.city'). Rows where all of these fields are empty are always kept. Leave empty for no field-based dedup. |
| `dedupWholeItem` | boolean | `false` | When no dedup fields are set: drop rows that are identical in every field. |
| `caseInsensitive` | boolean | `false` | Ignore letter case (and surrounding spaces) when comparing dedup keys and text filters. |
| `renameFields` | object | — | Map of old top-level field name to new name, e.g. {"Company Name": "company"}. Applied first. Map several source names to one new name to align differently named columns. |
| `filters` | array | — | Keep only rows matching these conditions, e.g. \[{"field": "price", "op": "gt", "value": 10}, {"field": "email", "op": "notEmpty"}]. Ops: eq, neq, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, regex, in, notIn, exists, notExists, empty, notEmpty. Numbers in text are compared as numbers. |
| `filterMode` | string | `"all"` | all = a row must match every filter; any = at least one. Allowed: `all`, `any`. |
| `keepFields` | array | — | Output only these fields, in this order (dot paths allowed; missing values become null). Leave empty to keep all fields. |
| `omitFields` | array | — | Top-level fields to remove (ignored when keepFields is set). |
| `addSourceField` | boolean | `false` | Add the dataset ID or file URL each row came from. |
| `sourceFieldName` | string | `"_source"` | Name of the source field. |
| `maxItems` | integer | `0` | Stop after this many rows are saved (0 = no limit). Also caps the cost of a run. |
| `skipHiddenFields` | boolean | `false` | Drop fields starting with # from dataset rows (Apify's hidden-field convention). |
| `csvDelimiter` | string | `"auto"` | auto detects , ; tab or |. Or give one character (use 'tab' for tab). |
| `maxFileSizeMb` | integer | `100` | Per file URL (1–500). Large JSON arrays need about 10x their size in memory; JSON Lines and CSV need less. Raise the run memory for big files. |
| `failOnSourceError` | boolean | `false` | Off: a missing dataset or broken file is reported in SOURCE_ERRORS and the other sources are still merged. On: the run fails at the first broken source. |
| `timeBudgetSecs` | integer | `0` | Stop reading after this many seconds and keep the rows saved so far (0 = only the run timeout guard). |

### Output

Each saved row keeps your own fields (after renaming, `keepFields` and `omitFields`). With `addSourceField` on, `_source` holds the dataset ID or file URL of the row. Example with the prefilled input:

```json
{"code": "AF", "name": "Afghanistan", "_source": "https://raw.githubusercontent.com/datasets/country-list/main/data.csv"}
```

The key-value store also gets `RUN_STATS` (rows read, filtered out, duplicates and saved, per source) and `SOURCE_ERRORS` (datasets or files that could not be read, with the reason).

### How much does it cost?

Pay per event. Apify platform usage is included.

| Apify plan | Price per output row | Per 1,000 rows |
|---|---|---|
| Free | $0.001 | $1.00 |
| Bronze | $0.0009 | $0.90 |
| Silver | $0.0008 | $0.80 |
| Gold, Platinum, Diamond | $0.0007 | $0.70 |

Plus an Actor start fee of $0.00005 per GB of memory (default 512 MB = one event). Duplicates and filtered-out rows are not charged. Use **Max output rows** to cap the cost of a run.

### Use with AI agents, MCP and the API

- **MCP:** add `https://mcp.apify.com?tools=phaux/dataset-merge-dedup` to an MCP client and ask, for example, "merge these two datasets and remove duplicate URLs".
- **API:** `POST https://api.apify.com/v2/acts/phaux~dataset-merge-dedup/run-sync-get-dataset-items` with `{"datasetIds": ["..."], "dedupFields": ["url"]}`.
- **After another Actor:** call it from an integration or webhook with the other run's `defaultDatasetId`.

### Data source and compliance

- Reads only **your own data**: the datasets you pick (read-only access granted per run) and file URLs you provide.
- File URLs are fetched once each, with robots.txt honoured for the user agent `PhauxDatasetMerge`, private and internal addresses refused, a size cap (`maxFileSizeMb`), and no further requests to a host after HTTP 429.
- No data is kept outside your run's storage.

### Limits

- `dedupFields` keeps the first occurrence; it doesn't merge values from duplicate rows.
- JSON arrays are loaded whole, so a big JSON file needs about 10× its size in memory; CSV and JSON Lines need much less. Raise the run memory (up to 4 GB) for large files.
- Dedup keys are held in memory (about 100 bytes per unique row), so tens of millions of rows need more memory.
- Rename works on top-level field names; `keepFields`, `dedupFields` and filters accept dot paths for nested values.

### FAQ

- **"Dataset doesn't exist or isn't shared with this run"?** Pick the dataset with the picker in Console, or pass its exact ID (or `username~dataset-name`) in `datasetIds`.
- **Why weren't two similar rows deduplicated?** The values differ, for example in letter case: switch on **Case-insensitive matching**. Leading and trailing spaces in `dedupFields` values are always ignored.
- **Can I export CSV or Excel?** Yes: export the default dataset from the run's Storage tab, or use the "Merged rows as CSV" output link.
- **Something isn't working?** Open an issue on the Issues tab with the run ID.

# Actor input Schema

## `datasetIds` (type: `array`):

Datasets to merge, in order. In Console pick them with the dataset picker; via API pass dataset IDs (or 'username~dataset-name'). Only these datasets can be read by the run (read-only access). Example: \["<dataset ID from a previous run>"].

## `fileUrls` (type: `array`):

Public URLs of CSV (any delimiter), JSON (array, or object with an items/data list) or JSON Lines files. Each is downloaded once, up to maxFileSizeMb. Example: \["https://raw.githubusercontent.com/datasets/country-list/main/data.csv", "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"].

## `dedupFields` (type: `array`):

Rows with the same values in all these fields count as duplicates; the first one is kept. Dot paths work for nested fields (e.g. 'address.city'). Rows where all of these fields are empty are always kept. Leave empty for no field-based dedup.

## `dedupWholeItem` (type: `boolean`):

When no dedup fields are set: drop rows that are identical in every field. Default: false.

## `caseInsensitive` (type: `boolean`):

Ignore letter case (and surrounding spaces) when comparing dedup keys and text filters. Default: false.

## `renameFields` (type: `object`):

Map of old top-level field name to new name, e.g. {"Company Name": "company"}. Applied first. Map several source names to one new name to align differently named columns.

## `filters` (type: `array`):

Keep only rows matching these conditions, e.g. \[{"field": "price", "op": "gt", "value": 10}, {"field": "email", "op": "notEmpty"}]. Ops: eq, neq, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, regex, in, notIn, exists, notExists, empty, notEmpty. Numbers in text are compared as numbers.

## `filterMode` (type: `string`):

all = a row must match every filter; any = at least one. Default: "all".

## `keepFields` (type: `array`):

Output only these fields, in this order (dot paths allowed; missing values become null). Leave empty to keep all fields. Example: \["url", "title", "price"].

## `omitFields` (type: `array`):

Top-level fields to remove (ignored when keepFields is set). Example: \["html", "debug"].

## `addSourceField` (type: `boolean`):

Add the dataset ID or file URL each row came from. Default: false.

## `sourceFieldName` (type: `string`):

Name of the source field. Example: "\_source". Default: "\_source".

## `maxItems` (type: `integer`):

Stop after this many rows are saved (0 = no limit). Also caps the cost of a run. Default: 0.

## `skipHiddenFields` (type: `boolean`):

Drop fields starting with # from dataset rows (Apify's hidden-field convention). Default: false.

## `csvDelimiter` (type: `string`):

auto detects , ; tab or |. Or give one character (use 'tab' for tab). Example: "," or ";". Default: "auto".

## `maxFileSizeMb` (type: `integer`):

Per file URL (1–500). Large JSON arrays need about 10x their size in memory; JSON Lines and CSV need less. Raise the run memory for big files. Default: 100.

## `failOnSourceError` (type: `boolean`):

Off: a missing dataset or broken file is reported in SOURCE_ERRORS and the other sources are still merged. On: the run fails at the first broken source. Default: false.

## `timeBudgetSecs` (type: `integer`):

Stop reading after this many seconds and keep the rows saved so far (0 = only the run timeout guard). Default: 0.

## Actor input object example

```json
{
  "fileUrls": [
    "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
    "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
  ],
  "dedupFields": [
    "code"
  ],
  "dedupWholeItem": false,
  "caseInsensitive": false,
  "renameFields": {
    "Code": "code",
    "Name": "name",
    "ISO3166-1-Alpha-2": "code",
    "official_name_en": "name"
  },
  "filters": [
    {
      "field": "code",
      "op": "startsWith",
      "value": "A"
    }
  ],
  "filterMode": "all",
  "keepFields": [
    "code",
    "name"
  ],
  "addSourceField": true,
  "sourceFieldName": "_source",
  "maxItems": 50,
  "skipHiddenFields": false,
  "csvDelimiter": "auto",
  "maxFileSizeMb": 100,
  "failOnSourceError": false,
  "timeBudgetSecs": 0
}
```

# Actor output Schema

## `rows` (type: `string`):

All saved rows (default dataset). Export as JSON, CSV or Excel from the dataset.

## `rowsCsv` (type: `string`):

Same rows as CSV.

## `runStats` (type: `string`):

RUN_STATS: rows read, filtered out, duplicates and saved, per source.

## `sourceErrors` (type: `string`):

Datasets or files that could not be read, with the reason.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "fileUrls": [
        "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
        "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
    ],
    "dedupFields": [
        "code"
    ],
    "renameFields": {
        "Code": "code",
        "Name": "name",
        "ISO3166-1-Alpha-2": "code",
        "official_name_en": "name"
    },
    "filters": [
        {
            "field": "code",
            "op": "startsWith",
            "value": "A"
        }
    ],
    "keepFields": [
        "code",
        "name"
    ],
    "addSourceField": true,
    "maxItems": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("phaux/dataset-merge-dedup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "fileUrls": [
        "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
        "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv",
    ],
    "dedupFields": ["code"],
    "renameFields": {
        "Code": "code",
        "Name": "name",
        "ISO3166-1-Alpha-2": "code",
        "official_name_en": "name",
    },
    "filters": [{
            "field": "code",
            "op": "startsWith",
            "value": "A",
        }],
    "keepFields": [
        "code",
        "name",
    ],
    "addSourceField": True,
    "maxItems": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("phaux/dataset-merge-dedup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "fileUrls": [
    "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
    "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
  ],
  "dedupFields": [
    "code"
  ],
  "renameFields": {
    "Code": "code",
    "Name": "name",
    "ISO3166-1-Alpha-2": "code",
    "official_name_en": "name"
  },
  "filters": [
    {
      "field": "code",
      "op": "startsWith",
      "value": "A"
    }
  ],
  "keepFields": [
    "code",
    "name"
  ],
  "addSourceField": true,
  "maxItems": 50
}' |
apify call phaux/dataset-merge-dedup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,phaux/dataset-merge-dedup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FLggYc484VnBsofqi/builds/PSOmegFQesOQiJIhS/openapi.json