# Dedup & Merge Datasets — Apify, JSON, CSV, $1.50/1K (`kadro/dedup-merge-datasets`) Actor

Merge Apify datasets, JSON and CSV files, then remove duplicate rows by the fields you choose.

- **URL**: https://apify.com/kadro/dedup-merge-datasets.md
- **Developed by:** [Kadro Tools](https://apify.com/kadro) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.50 / 1,000 unique rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dedup & Merge Datasets: combine Apify datasets, JSON and CSV and drop duplicates

Scraped the same thing in several runs, or exported lists from different places? This Actor merges Apify datasets, JSON, JSON Lines and CSV files (by link) and pasted rows into one clean list, and removes duplicates using the fields you choose. It works with limited permissions, so you do not have to hand over full account access.

### What you get

| Field | What it tells you |
|---|---|
| `(all your original fields)` | Every column of the surviving row is kept exactly as it came. |
| `_source` | Where the row came from: `inline`, `dataset:<id>` or the file URL. |

Example row:

```json
{"id": 1, "name": "Ana ", "city": "Madrid", "_source": "inline"}
```

### How to use it in 3 steps

1. Add your data: pick one or more Apify datasets, paste file links (JSON, JSON Lines, CSV, up to 20 files of 5 MB) or paste rows as a JSON list.
2. Choose the fields that define a duplicate (for example `url` or `email`), keep the `first` or `last` row, and decide whether to ignore case and extra spaces.
3. Run it and download the unique rows from the dataset as JSON, CSV or Excel. The run status shows how many rows came in and how many duplicates were removed.

### Pricing

**$1.50 per 1,000 unique rows** (1,000 unique rows = $1.50). You pay only for rows that are kept; removed duplicates are free, and a run with no rows costs nothing. Default input: 4 rows in, 3 out.

### Integrations

- **Apify API**: call the Actor from any language and read the dataset as JSON, CSV or Excel.
- **Make, n8n and Zapier**: use the official Apify modules, run this Actor and map the output rows.
- **Google Sheets**: export the dataset to Sheets from the run page, or schedule runs and append the rows.
- **MCP (Claude, Cursor)**: add the Apify MCP server `https://mcp.apify.com` with `actors=kadro/dedup-merge-datasets`. Example prompt: "Merge these two Apify datasets and remove duplicates by the `url` field."

### FAQ

**How is a duplicate defined?**
Two rows are duplicates when all the fields you list match. If you list none, the whole row is compared (fields starting with `_` are ignored).

**Does it need full account permissions?**
No. It runs with limited permissions and only reads the datasets you select in the picker.

**How big can the input be?**
Up to 200,000 rows per run, held in memory. Files: up to 20 links, 5 MB each.

**Which files can it read?**
JSON (list of objects), JSON Lines and CSV/TSV with automatic delimiter detection. Excel is not read here; convert it first with the CSV, Excel & JSON Converter.

**Is the data safe?**
Only public http(s) addresses are fetched (private and local addresses are blocked) and files are parsed as data, never executed.

### Limitations

- Everything is processed in memory, so the limit is 200,000 rows per run.
- Matching is exact after trimming spaces and, optionally, lower-casing; there is no fuzzy matching.
- Nested objects are compared as JSON text.
- Files over 5 MB or on private networks are skipped with a warning in the log.

### Other Kadro tools

- [CSV, Excel & JSON Converter](https://apify.com/kadro/csv-excel-json-converter): get your files into CSV or JSON first.
- [Bulk Email Validator](https://apify.com/kadro/email-syntax-mx-validator): clean the emails you just merged.
- [Bulk Phone Number Validator](https://apify.com/kadro/bulk-phone-number-validator): validate phone columns in bulk.
- [Website Tech Stack Detector](https://apify.com/kadro/website-tech-stack-detector): enrich merged site lists with their tech stack.

Found it useful? Please leave a review ⭐ — it helps a lot. Bugs or feature ideas: open an issue, I answer within 24 h.

# Actor input Schema

## `inlineRows` (type: `array`):

Rows pasted as a JSON list of objects. Handy for quick tests.

## `datasets` (type: `array`):

Datasets from your account to merge (pick one or more). Optional.

## `fileUrls` (type: `array`):

Public http(s) links to files (max 5 MB each, 20 files). Optional.

## `dedupFields` (type: `array`):

Two rows are duplicates when these fields match. Leave empty to compare the whole row.

## `keep` (type: `string`):

Keep the first or the last row found.

## `ignoreCase` (type: `boolean`):

Treat 'Ana ' and 'ana' as equal.

## Actor input object example

```json
{
  "inlineRows": [
    {
      "id": 1,
      "name": "Ana ",
      "city": "Madrid"
    },
    {
      "id": 2,
      "name": "Luis",
      "city": "Sevilla"
    },
    {
      "id": 1,
      "name": "ana",
      "city": "Madrid"
    },
    {
      "id": 3,
      "name": "Marta",
      "city": "Bilbao"
    }
  ],
  "dedupFields": [
    "id"
  ],
  "keep": "first",
  "ignoreCase": true
}
```

# Actor output Schema

## `results` (type: `string`):

One row per result.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "inlineRows": [
        {
            "id": 1,
            "name": "Ana ",
            "city": "Madrid"
        },
        {
            "id": 2,
            "name": "Luis",
            "city": "Sevilla"
        },
        {
            "id": 1,
            "name": "ana",
            "city": "Madrid"
        },
        {
            "id": 3,
            "name": "Marta",
            "city": "Bilbao"
        }
    ],
    "dedupFields": [
        "id"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("kadro/dedup-merge-datasets").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "inlineRows": [
        {
            "id": 1,
            "name": "Ana ",
            "city": "Madrid",
        },
        {
            "id": 2,
            "name": "Luis",
            "city": "Sevilla",
        },
        {
            "id": 1,
            "name": "ana",
            "city": "Madrid",
        },
        {
            "id": 3,
            "name": "Marta",
            "city": "Bilbao",
        },
    ],
    "dedupFields": ["id"],
}

# Run the Actor and wait for it to finish
run = client.actor("kadro/dedup-merge-datasets").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "inlineRows": [
    {
      "id": 1,
      "name": "Ana ",
      "city": "Madrid"
    },
    {
      "id": 2,
      "name": "Luis",
      "city": "Sevilla"
    },
    {
      "id": 1,
      "name": "ana",
      "city": "Madrid"
    },
    {
      "id": 3,
      "name": "Marta",
      "city": "Bilbao"
    }
  ],
  "dedupFields": [
    "id"
  ]
}' |
apify call kadro/dedup-merge-datasets --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kadro/dedup-merge-datasets"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/km7AV1S4IAawflmcA/builds/VofGP1cxJBrNQ288k/openapi.json