# Remove Duplicates & Merge CSV, JSON and Datasets (`rowfeed/dataset-merge-dedup`) Actor

Merge Apify datasets, CSV and JSON files into one table and remove duplicates by any fields (url, email, website ...), keeping the first or newest row. Loose matching of case, spaces and URLs. $0.10 per 1,000 rows out.

- **URL**: https://apify.com/rowfeed/dataset-merge-dedup.md
- **Developed by:** [Rowfeed](https://apify.com/rowfeed) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 row outs

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Remove Duplicates & Merge CSV, JSON and Datasets

Merge the results of several Actor runs, CSV files and JSON files into one clean table and remove duplicate rows by the fields you choose: url, email, website, phone, title + city, or anything nested. Keep the first row or the newest one. Download the result as CSV, Excel or JSON from the dataset, or send it on with any integration.

Typical uses:

- **Combine daily runs** of a scraper and keep one row per product, place or lead, with the newest data.
- **Merge lead lists** from different tools and drop the people or companies you already have (match on email or website).
- **Clean a CSV** with duplicate rows, or turn a JSON file into a table (JSON to CSV).

### What it does

- **Sources:** Apify datasets (pick them in the form or paste dataset IDs), public CSV/JSON/JSON Lines file links (CSV with comma, semicolon or tab, UTF-8 or Windows/Excel encoding, up to 20 MB each) and rows pasted as JSON. Mix as many as you like. Pasted rows are read first, then the datasets and then the files, each in the order you list them.
- **Remove duplicates by fields:** rows with the same values in the chosen fields are duplicates. Use dots for nested fields (`address.city`). With no fields given, only rows identical in every field are removed.
- **Loose matching (on by default):** `ACME ` = `acme`, `1` = `"1"` (CSV text and dataset numbers match), and `https://www.acme.com/` = `http://acme.com` = `www.acme.com`. In URLs only the domain ignores letter case, so `bit.ly/3AbC` and `bit.ly/3abc` stay different. Turn it off for exact matching.
- **Keep first or last:** "last" keeps the newest version of each row, at the position where it first appeared.
- **Rows without a value** in the dedup fields are always kept, never merged with each other.
- **Output only some fields**, in your order, and optionally add a `_source` column that says which dataset or file each row came from.

The run's `STATS` record shows rows read, duplicates removed, rows written and any source that could not be read. A source that fails (a wrong dataset ID, a broken link) is skipped and named in the log. The others are still merged.

### Example

Input (the pre-filled example):

```json
{
  "rows": [
    {"company": "Acme", "website": "https://acme.com/"},
    {"company": "ACME ", "website": "http://www.acme.com"},
    {"company": "Globex", "website": "https://globex.com"}
  ],
  "dedupFields": ["website"]
}
```

Output: 2 rows, Acme and Globex. The second Acme is the same website written differently.

To merge two runs of any Actor, give their dataset IDs, for example:

```json
{
  "datasetIds": ["aBcD1234efGH5678i", "JkLm9012noPQ3456r"],
  "dedupFields": ["url"],
  "keep": "last"
}
```

### Pricing

Pay per event, no subscription:

- **$0.0001 per row written ($0.10 per 1,000 rows).** Removed duplicates are free.
- $0.001 per run start.

Example: merge 3 runs with 20,000 rows in total, 12,000 unique: $1.20.
Other paid tools charge $0.50 to $2.30 per 1,000 rows. You can cap the cost with the maximum charge setting. The Actor stops writing at the cap and never charges more.

### Good to know

- Each dataset needs to be one this run can read: your own datasets, or ones you selected in the form.
- Excel files: save as CSV first. Files over 20 MB: load them into an Apify dataset first.
- If none of the rows has a value in your dedup fields, the run says so in the log and status (field names are case-sensitive).
- Large merges with "keep last" hold all unique rows in memory until the end. Give the run more memory (Run options) for millions of rows.

### More tools from Rowfeed

These are Actors whose results are worth merging:

- [Website Email Extractor & Contact Finder](https://apify.com/rowfeed/website-contact-finder) – company emails and phones for the websites in your merged list.
- [Google Sheets Export](https://apify.com/rowfeed/google-sheets-export) – send the merged table straight to a Google Sheet, without OAuth.
- [Google Maps Scraper](https://apify.com/rowfeed/google-maps-scraper) – places with phone, website, rating and address; merge searches across cities by place ID.
- [Keyword Generator](https://apify.com/rowfeed/keyword-generator) – long-tail keyword ideas; merge many seed runs and keep each keyword once.
- [SERP API & Rank Checker](https://apify.com/rowfeed/google-serp-api) – Google results for any keyword; merge daily runs into one rank history.
- [Website Technology Checker](https://apify.com/rowfeed/website-technology-checker) – CMS, ecommerce and analytics tools any website uses.
- [WHOIS Lookup & DNS Checker](https://apify.com/rowfeed/domain-rdap-checker) – domain age, expiry, DNS, SSL and an SPF/DMARC grade for lead lists.

# Actor input Schema

## `datasetIds` (type: `array`):

Datasets to merge, e.g. the results of several runs of any Actor. Pick them or paste dataset IDs (Storage tab, or the run's Output).

## `fileUrls` (type: `array`):

Public links to CSV (comma, semicolon or tab), JSON array or JSON Lines files, up to 50 MB each. Excel: save as CSV first.

## `rows` (type: `array`):

Or paste rows as a JSON array of objects.

## `dedupFields` (type: `array`):

Rows with the same values in these fields count as duplicates, e.g. url, email, website or title + city. Dots reach nested fields (address.city). Empty = only rows that are identical in every field.

## `keep` (type: `string`):

"Last" keeps the newest version of each row, e.g. when merging today's run into yesterday's.

## `looseMatch` (type: `boolean`):

Ignore upper/lower case and extra spaces, and treat http/https, www. and a trailing slash in URLs as the same.

## `outputFields` (type: `array`):

Optional: keep only these columns, in this order (dots flatten nested fields). Empty = all fields.

## `addSource` (type: `boolean`):

Add \_source to each row: which dataset, file or input the row came from.

## Actor input object example

```json
{
  "datasetIds": [
    "aBcD1234efGH5678i"
  ],
  "fileUrls": [
    "https://example.com/leads.csv"
  ],
  "rows": [
    {
      "company": "Acme",
      "website": "https://acme.com/"
    },
    {
      "company": "ACME ",
      "website": "http://www.acme.com"
    },
    {
      "company": "Globex",
      "website": "https://globex.com"
    }
  ],
  "dedupFields": [
    "website"
  ],
  "keep": "first",
  "looseMatch": true,
  "addSource": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "rows": [
        {
            "company": "Acme",
            "website": "https://acme.com/"
        },
        {
            "company": "ACME ",
            "website": "http://www.acme.com"
        },
        {
            "company": "Globex",
            "website": "https://globex.com"
        }
    ],
    "dedupFields": [
        "website"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("rowfeed/dataset-merge-dedup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "rows": [
        {
            "company": "Acme",
            "website": "https://acme.com/",
        },
        {
            "company": "ACME ",
            "website": "http://www.acme.com",
        },
        {
            "company": "Globex",
            "website": "https://globex.com",
        },
    ],
    "dedupFields": ["website"],
}

# Run the Actor and wait for it to finish
run = client.actor("rowfeed/dataset-merge-dedup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "rows": [
    {
      "company": "Acme",
      "website": "https://acme.com/"
    },
    {
      "company": "ACME ",
      "website": "http://www.acme.com"
    },
    {
      "company": "Globex",
      "website": "https://globex.com"
    }
  ],
  "dedupFields": [
    "website"
  ]
}' |
apify call rowfeed/dataset-merge-dedup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rowfeed/dataset-merge-dedup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/doY6kQbVTeBd4ahOy/builds/0NP8k8dF3ZF0GoONc/openapi.json