# Merge & Dedup Datasets: Remove Duplicates, Only New Items (`charlesmarkette/merge-deduplicate-datasets-only-new-items`) Actor

Merge any number of Apify datasets and remove duplicates by the fields you choose (or the whole item). Keep the first or newest copy, or output only items not seen in earlier runs: ideal after a scheduled scraper. Streams millions of items. Duplicates are free; $0.0019 per 1,000 items output.

- **URL**: https://apify.com/charlesmarkette/merge-deduplicate-datasets-only-new-items.md
- **Developed by:** [Charles Markette](https://apify.com/charlesmarkette) (community)
- **Stats:** 5 total users, 4 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.90 / 1,000 batch of 1,000 output items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Merge & Deduplicate Datasets (only new items, fast)

Combine any number of Apify datasets into one clean dataset:

- **Remove duplicates** by the fields you choose (`url`, `id`, `seller.id` …), or by the whole item when you choose none.
- **Keep the first or the last** copy. With "last", the newer dataset wins, which is handy when re-scraping.
- **Only new items:** remember what earlier runs returned, and output just the new ones. This is the classic setup for a scheduled scraper that should only alert you about new listings, leads or prices.
- **Clean up:** drop items missing required fields, keep only some fields, remove others, ignore letter case or extra spaces in keys, and tag each item with the dataset it came from.

With "keep first" it streams items and holds only an 8-byte fingerprint per unique key, so a run at the default 1 GB memory handles several million unique items. "Keep last" has to hold one copy of each unique item until the end, so give it more memory for big datasets. It runs with **limited permissions**: it can only read the datasets you select and write its own output and memory.

### Typical setups

**After a scraper (Integrations):** in your scraper's task, go to **Integrations → Connect Actor or task**, pick this Actor, and use:

```json
{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "mode": "onlyNew",
  "dedupFields": ["url"],
  "stateName": "my-listing-monitor"
}
```

Each scheduled run then outputs only listings that weren't in any earlier run. The fixed `stateName` matters because every scraper run creates a new dataset. The memory is saved as the run goes (at least every 15 seconds, and right away when the cost limit is hit or the run is migrated or aborted), so after a timeout or crash the next run repeats at most the last few seconds of items.

**Merge several runs:** pick the datasets, set `dedupFields` to `["id"]` and `keep` to `last`.

### Input

| Field | Notes |
|---|---|
| Datasets | One or more datasets, processed in the order given. |
| Items | A JSON array to deduplicate instead of datasets (used when no dataset is picked). |
| Mode | Remove duplicates, only new since earlier runs, or just merge. |
| Deduplicate by fields | Dots for nested fields. Empty = compare whole items. |
| Which duplicate to keep | First (streams) or last (keeps one buffered item per key). |
| Ignore letter case / extra spaces | Applied to the key fields only. |
| Drop items missing these fields | Missing, empty string, empty list or empty object all count as missing. Works in every mode. |
| Output only these fields / Remove these fields | Shape the output. |
| Add `_sourceDataset` | Records where each item came from. |
| Max output items | Optional cap. |
| Monitoring name | For "only new": runs with the same name (and the same dedup settings) share one memory. Without a name, a saved task keeps its own memory; otherwise the memory belongs to that exact set of datasets. |

### Output

- **Dataset:** the unique (or merged) items.
- **Key-value store `OUTPUT`:** counts of items read, duplicates removed, items missing required fields, and items output.

### Pricing

You pay per 1,000 items written, and each started block of 1,000 counts as one. Items that are read but removed as duplicates are free. If you set a maximum cost per run, the Actor stops cleanly at that limit, and in "only new" mode it remembers only the items it actually returned.

### Notes

- "Only new" remembers about 5 million keys per memory, then forgets the oldest (in blocks of 100,000).
- Don't run two "only new" runs with the same memory at the same time; the one that finishes last wins.
- Keys are compared by type: `1` and `"1"` are different keys. Changing the dedup fields or the case/space options starts a fresh memory.
- Keys are compared as compact 64-bit fingerprints. Two different keys colliding is astronomically unlikely, but not impossible.

# Actor input Schema

## `datasetIds` (type: `array`):

One or more datasets to merge, in order. In an integration, use \["{{resource.defaultDatasetId}}"]. Leave empty to use 'Items' below instead.

## `items` (type: `array`):

Optional. A JSON array of objects to deduplicate, used when no dataset is picked. Handy for quick tests and for calling this Actor from code.

## `mode` (type: `string`):

'Only new' remembers what earlier runs of this saved task (or this exact setup) returned and outputs just the new items.

## `dedupFields` (type: `array`):

Items with the same values in these fields are duplicates, e.g. 'url' or 'id'. Use dots for nested fields ('seller.id'). Empty = compare whole items.

## `keep` (type: `string`):

With several datasets, 'last' keeps the version from the later dataset.

## `caseInsensitive` (type: `boolean`):

'ACME' and 'acme' count as the same.

## `trimWhitespace` (type: `boolean`):

Trims and collapses whitespace before comparing.

## `requiredFields` (type: `array`):

Items where any of these fields is missing or empty are skipped.

## `outputFields` (type: `array`):

Optional field list (dots for nested). Empty = keep all fields.

## `omitFields` (type: `array`):

Top-level fields to drop from the output.

## `addSourceDataset` (type: `boolean`):

Adds the ID of the dataset each item came from.

## `maxItems` (type: `integer`):

Optional cap on items written.

## `stateName` (type: `string`):

Optional. Runs with the same name share their memory of seen items, even when the input dataset changes every run (typical for scheduled scrapers). Leave empty to tie memory to the saved task or to these exact datasets.

## Actor input object example

```json
{
  "items": [
    {
      "url": "https://example.com/a",
      "price": 10
    },
    {
      "url": "https://example.com/b",
      "price": 12
    },
    {
      "url": "https://example.com/a",
      "price": 10
    }
  ],
  "mode": "dedupe",
  "dedupFields": [
    "url"
  ],
  "keep": "first",
  "caseInsensitive": false,
  "trimWhitespace": false,
  "addSourceDataset": false
}
```

# Actor output Schema

## `items` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "url": "https://example.com/a",
            "price": 10
        },
        {
            "url": "https://example.com/b",
            "price": 12
        },
        {
            "url": "https://example.com/a",
            "price": 10
        }
    ],
    "dedupFields": [
        "url"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("charlesmarkette/merge-deduplicate-datasets-only-new-items").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "items": [
        {
            "url": "https://example.com/a",
            "price": 10,
        },
        {
            "url": "https://example.com/b",
            "price": 12,
        },
        {
            "url": "https://example.com/a",
            "price": 10,
        },
    ],
    "dedupFields": ["url"],
}

# Run the Actor and wait for it to finish
run = client.actor("charlesmarkette/merge-deduplicate-datasets-only-new-items").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    {
      "url": "https://example.com/a",
      "price": 10
    },
    {
      "url": "https://example.com/b",
      "price": 12
    },
    {
      "url": "https://example.com/a",
      "price": 10
    }
  ],
  "dedupFields": [
    "url"
  ]
}' |
apify call charlesmarkette/merge-deduplicate-datasets-only-new-items --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,charlesmarkette/merge-deduplicate-datasets-only-new-items"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LuSTAphtTWbzYEtK3/builds/pmxpm2SSDrBdZMTr9/openapi.json