# Remove Duplicates & Merge Datasets: Dedup, Only New (`brasspage/dataset-dedup-merge`) Actor

Dedup and merge Apify datasets: removes duplicate items by any fields, or by the whole item, and returns the unique items (or only the duplicates) with their original fields. Can skip items seen in earlier runs, for daily new results. Works with limited permissions. $0.02 per 1,000 items.

- **URL**: https://apify.com/brasspage/dataset-dedup-merge.md
- **Developed by:** [Brasspage](https://apify.com/brasspage) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.02 / 1,000 item processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Remove Duplicates & Merge Datasets: Dedup, Only New

Dedup and merge Apify datasets by any field. Pick one or more Apify datasets. The Actor reads them in order, and returns:

- **Unique items:** the first item of each key, with all its original fields. Or choose **only the duplicates**.
- **Dedup by any fields:** for example `email`, `url`, or `name` + `address.city`. Leave it empty to compare whole items.
- **Only new items:** remember every key across runs and skip items an earlier run already saw. Put it after a daily scraper to get only today's new results.
- **A summary** in the `OUTPUT` record: items read, unique, duplicates, items written, and items read per dataset.

**$0.02 per 1,000 items.** It works with **limited permissions**: the Actor can read only the datasets you pick.

### How to use

1. Open the Actor in the Apify Console and go to the **Input** tab.
2. Pick your datasets in **Datasets to merge** (Storage > Datasets), or paste items into **JSON items**.
3. Type the fields that make an item unique in **Dedup by fields**, for example `email`.
4. Click **Start**.
5. Open the **Output** tab and download the unique items as CSV, Excel or JSON, or read them through the API.

The prefilled input dedups 5 sample items by email and returns 3, so a one-click run shows how it works.

### What can you do with it?

- **Merge runs:** join the results of 10 scraper runs into one clean list.
- **Clean leads:** keep one row per email or phone, and ignore case and extra spaces with **Ignore case and extra spaces**.
- **Get only new results each day:** run a scraper on a schedule, add this Actor as an integration with `{{resource.defaultDatasetId}}` and **Only items not seen in earlier runs**, and get only the new jobs, listings or posts.
- **Find duplicates:** set **Output** to **Only the duplicates** to see what repeats.
- **AI agents:** call the Actor as a tool (Apify API or Apify MCP server) to dedup a dataset before it reads it.

### Input example

```json
{
  "datasetIds": ["<first dataset ID>", "<second dataset ID>"],
  "dedupFields": ["email"],
  "normalize": true,
  "output": "unique",
  "onlyNew": true,
  "memoryKey": "daily-leads",
  "maxItems": 100000
}
```

### Output example

The dataset holds your items exactly as they were, minus the duplicates. The `OUTPUT` record holds the summary:

```json
{
  "itemsRead": 5,
  "uniqueItems": 3,
  "duplicateItems": 2,
  "itemsOutput": 3,
  "output": "unique",
  "dedupFields": ["email"],
  "onlyNew": false,
  "sources": [{ "datasetId": "input items", "itemsRead": 5 }],
  "notes": []
}
```

### Run it after another Actor (integration)

1. Open your scraper, go to the **Integrations** tab, and add this Actor.
2. Set **Datasets to merge** to `{{resource.defaultDatasetId}}`.
3. Turn on **Only items not seen in earlier runs** and give a **Memory name**, for example the scraper's name.

Each run now writes only the items that are new since the last run.

### How duplicates are found

- Two items are duplicates when all fields in **Dedup by fields** have the same values. Dots read nested fields: `address.city`.
- Without fields, whole items are compared. The order of keys does not matter.
- With **Ignore case and extra spaces**, `"HI@x.com "` equals `"hi@x.com"`. The output keeps the original text.
- An item where every dedup field is empty is always kept, so items with no email are not merged into one.
- The first item of each key wins, in the order of **Datasets to merge**, then **JSON items**.
- Keys are stored as short hashes, so a run uses little memory. Only new items mode keeps up to 5,000,000 keys in the key-value store `dedup-memory-<name>` in your account. Delete that store to start fresh.

### Pricing

- **$0.02 per 1,000 items processed** ($0.00002 per item read and checked).
- A tiny Actor start event (Apify's standard start charge) applies once per run.
- Set **Max items to read** or a **maximum total charge** on the run, and the Actor stops before it goes over.

**The prefilled input** reads 5 items, so it costs about $0.00015: $0.0001 for the 5 items plus $0.00005 for the start event (one event at the default 1 GB of memory).

**Apify free plan:** each run reads the first 10,000 items. Upgrade to a paid Apify plan to process any number in one run.

### FAQ

**Why do other dedup Actors fail with a permission error?** Under limited permissions, Apify gives an Actor access only to the datasets picked in a dataset picker field. A plain text field with a dataset ID is not enough. This Actor uses the picker, so it can read your datasets without full account access.

**Can I dedup a dataset from another account?** Only if your account can read it.

**Does it change my datasets?** No. It only reads them and writes the result to the run's own dataset.

**How does it read big datasets?** In pages of 5,000 items, so memory stays low. Only the short hash of each key is kept.

### Limits

- **Apify free plan:** up to 10,000 items per run. Paid plans have no fixed cap.
- **Max items to read** and the run's **maximum total charge** stop the run early. The `OUTPUT` record then has a note.
- **Only new items** remembers up to 5,000,000 keys per memory name. Older keys are forgotten first.
- The Actor can read only the datasets picked in **Datasets to merge**. A dataset it cannot read stops the run with an error.
- **JSON items** must be valid JSON: an array of objects, or one object.

### Related Actors

- [Google Sheets Export & Sync](https://apify.com/brasspage/google-sheets-sync): send the unique items to a Google Sheet.
- [ATS Jobs API: Greenhouse, Lever, Ashby, Workable](https://apify.com/brasspage/careers-site-jobs-api): pair it with only new items for daily new job alerts.

### Issues and support

Found a bug or need a feature? Open an issue in the Actor's **Issues** tab. We check it every day and reply there.

# Changelog

This Actor's version history is a separate document: https://apify.com/brasspage/dataset-dedup-merge/changelog.md

# Actor input Schema

## `datasetIds` (type: `array`):

One or more Apify datasets, read in this order. Pick them from Storage > Datasets. In an integration, use {{resource.defaultDatasetId}}. The Actor gets read access only to the datasets you pick here.

## `items` (type: `array`):

Optional items to add after the datasets, as a JSON array of objects. Useful for a quick test or a small list.

## `dedupFields` (type: `array`):

Items with the same values in all of these fields count as duplicates. Use dots for nested fields, for example "address.city". Leave empty to compare whole items. Items where every field is empty are kept.

## `normalize` (type: `boolean`):

Compare text without case and extra spaces, so "HI@x.com " equals "hi@x.com". The output keeps the original values.

## `output` (type: `string`):

What goes into this run's dataset.

## `onlyNew` (type: `boolean`):

Remember the keys of every item across runs and skip items an earlier run already saw. Use it on a scheduled scraper to get only the new results each day.

## `memoryKey` (type: `string`):

Runs with the same memory name share one memory. Use a different name for each scraper or task. Stored in your account as the key-value store dedup-memory-<name>; delete it to start fresh.

## `maxItems` (type: `integer`):

Stop after this many items. Leave empty for no limit (the Apify free plan reads up to 10,000 items per run).

## Actor input object example

```json
{
  "datasetIds": [
    "WkzbQMuFYuamGv3YF"
  ],
  "items": [
    {
      "name": "Spokewell Bikes",
      "email": "hi@spokewell.example",
      "city": "Boise"
    },
    {
      "name": "Spokewell Bikes",
      "email": "HI@spokewell.example ",
      "city": "Boise"
    },
    {
      "name": "Copper Kettle Cafe",
      "email": "hello@copperkettle.example",
      "city": "Austin"
    },
    {
      "name": "Copper Kettle",
      "email": "hello@copperkettle.example",
      "city": "Austin"
    },
    {
      "name": "North Fork Plumbing",
      "email": "office@northfork.example",
      "city": "Denver"
    }
  ],
  "dedupFields": [
    "email"
  ],
  "normalize": true,
  "output": "unique",
  "onlyNew": false,
  "memoryKey": "default",
  "maxItems": 100000
}
```

# Actor output Schema

## `items` (type: `string`):

The unique items (or only the duplicates), with their original fields.

## `summary` (type: `string`):

Items read, unique, duplicates, items written, items read per dataset, and notes.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "name": "Spokewell Bikes",
            "email": "hi@spokewell.example",
            "city": "Boise"
        },
        {
            "name": "Spokewell Bikes",
            "email": "HI@spokewell.example ",
            "city": "Boise"
        },
        {
            "name": "Copper Kettle Cafe",
            "email": "hello@copperkettle.example",
            "city": "Austin"
        },
        {
            "name": "Copper Kettle",
            "email": "hello@copperkettle.example",
            "city": "Austin"
        },
        {
            "name": "North Fork Plumbing",
            "email": "office@northfork.example",
            "city": "Denver"
        }
    ],
    "dedupFields": [
        "email"
    ],
    "normalize": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("brasspage/dataset-dedup-merge").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "items": [
        {
            "name": "Spokewell Bikes",
            "email": "hi@spokewell.example",
            "city": "Boise",
        },
        {
            "name": "Spokewell Bikes",
            "email": "HI@spokewell.example ",
            "city": "Boise",
        },
        {
            "name": "Copper Kettle Cafe",
            "email": "hello@copperkettle.example",
            "city": "Austin",
        },
        {
            "name": "Copper Kettle",
            "email": "hello@copperkettle.example",
            "city": "Austin",
        },
        {
            "name": "North Fork Plumbing",
            "email": "office@northfork.example",
            "city": "Denver",
        },
    ],
    "dedupFields": ["email"],
    "normalize": True,
}

# Run the Actor and wait for it to finish
run = client.actor("brasspage/dataset-dedup-merge").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    {
      "name": "Spokewell Bikes",
      "email": "hi@spokewell.example",
      "city": "Boise"
    },
    {
      "name": "Spokewell Bikes",
      "email": "HI@spokewell.example ",
      "city": "Boise"
    },
    {
      "name": "Copper Kettle Cafe",
      "email": "hello@copperkettle.example",
      "city": "Austin"
    },
    {
      "name": "Copper Kettle",
      "email": "hello@copperkettle.example",
      "city": "Austin"
    },
    {
      "name": "North Fork Plumbing",
      "email": "office@northfork.example",
      "city": "Denver"
    }
  ],
  "dedupFields": [
    "email"
  ],
  "normalize": true
}' |
apify call brasspage/dataset-dedup-merge --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,brasspage/dataset-dedup-merge"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0IcTXP7CX1DIKVDBh/builds/afje5Cta43u0AA8iz/openapi.json