# Merge & Deduplicate Datasets: Remove Duplicates (`grace_hartwell/dataset-merge-dedup`) Actor

Merge multiple Apify datasets into one and remove duplicates by any field. Only-new-items mode for scheduled scrapers. Streams large datasets.

- **URL**: https://apify.com/grace_hartwell/dataset-merge-dedup.md
- **Developed by:** [Grace M. Hartwell](https://apify.com/grace_hartwell) (community)
- **Categories:** Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 output items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

<img src="https://apify-image-uploads-prod.s3.us-east-1.amazonaws.com/dpsKHwXKHdbwBgwOT-profile-dxtJoi0ame-avatar-hartwell-800.png" alt="Hartwell Toolworks" width="96" height="96" />

**Merge & Deduplicate Datasets** combines any number of **Apify datasets into one and removes duplicate items** by one or more fields, such as `url`, `id` or `product.id` + `shop`. It reads datasets page by page, so it can handle large datasets. It runs with **limited permissions**: it reads only the datasets you pick. An optional **only-new-items mode** remembers what earlier runs already output, so **scheduled scrapers deliver only new results**. Pricing: **$0.25 per 1,000 unique output items**. Duplicates and skipped items are free.

### What does Merge & Deduplicate Datasets do?

1. **Merge datasets**: reads all selected datasets in the order you list them.
2. **Remove duplicates**: keeps the **first** item for each unique combination of your `dedupFields`. Dot paths like `product.id` work. Comparison can be case-insensitive and can ignore leading or trailing whitespace. Leave `dedupFields` empty to remove only fully identical items.
3. **Output only new items** (optional): skips items that earlier runs with the same `stateKey` already output. The remembered keys live in a key-value store the Actor creates in your account (`dedup-state-<stateKey>`).
4. **Filter by reference datasets** (optional): also skips items whose key appears in other datasets, for example "already imported into the CRM".
5. **Select fields** (optional): outputs only the fields you choose and can add `_sourceDatasetId` to each item.
6. **Saves a run summary**: counts of read, output and duplicate items, plus the stop reason, are saved to the `OUTPUT` record of the run's key-value store.

### Key features

- ✅ **Pre-flight check**: before anything is written or charged, the Actor checks that every dataset exists and is readable. If one isn't, the run fails with a clear message, e.g. `Dataset "x" was not found…` or `No permission to read dataset…`.
- 🔐 **Limited permissions**: datasets are selected with Apify's dataset picker (`READ` only). The Actor writes only to its own default dataset and key-value store and, in only-new mode, to a key-value store it created itself. It does not list your runs, Actors or other storages.
- 🌊 **Streaming**: reads in pages (default 5,000 items) and writes in batches (default 1,000). Memory is used mainly for one compact 80-bit hash per unique key.
- 🔁 **Retry-safe**: every read uses exponential backoff with jitter for 429, 5xx and network errors. Permanent errors (401/403/404) are not retried.
- 🧭 **Migration-safe**: the Actor saves checkpoints at page boundaries, and a restarted run continues where it stopped. Checkpoints and the only-new state are stored as compact binary records of at most 4 MB each (10 bytes per key), tested with 5 million keys.
- 🧮 **Collision-safe keys**: field values are JSON-encoded and joined with a separator, so `"ab"+"c"` never equals `"a"+"bc"`. Objects are compared regardless of key order.
- 💸 **Budget-aware**: stops cleanly at your *Max total charge* or `maxOutputItems`. Items that were not output are **not** remembered as seen, so a later run still treats them as new.

### Use cases

1. **Combine parallel scraper runs**: merge the datasets of many scraper runs into one clean, deduplicated dataset.
2. **Incremental scraping**: schedule a scraper, then output **only new** listings, jobs, products or posts each day.
3. **Skip already-processed records**: use reference datasets to exclude items you have already imported or handled.
4. **Shrink exports**: remove duplicates and keep only the fields you need before exporting to Google Sheets, Airtable, a CRM or a vector database.
5. **Clean up datasets**: remove fully identical items (empty `dedupFields`) or normalize keys with case-insensitive and trimmed comparison.

### How much does it cost to merge and deduplicate datasets?

The Actor uses **pay-per-event** pricing:

| Event | Price | When it is charged |
| --- | --- | --- |
| `item-output` | **$0.00025** per unique item written ($0.25 per 1,000) | When a unique item is written to the output dataset |
| `apify-actor-start` | **$0.005** per start event | Automatically by Apify at run start |

**Free:** items read, in-run duplicates, items seen in previous runs, items filtered by reference datasets and dropped items.

**Example calculation: merge 100,000 items, 60,000 of them unique (default 1 GB memory):**

```
60,000 unique items × $0.00025 = $15.00
+ 1 Actor start × $0.005       = $0.005
------------------------------------------
Total                           = $15.005   (= $0.25 per 1,000 output items + $0.005 per run)
```

The 40,000 duplicates cost nothing. Apify charges the start event once for runs with up to 1 GB of memory and once more for each additional GB, so a run with 4 GB of memory costs 4 × $0.005 = $0.02 in start events.

### How to merge and deduplicate Apify datasets

1. Open **Merge & Deduplicate Datasets** in Apify Store and click **Try for free**.
2. In **Datasets to merge**, select one or more datasets with the dataset picker.
3. In **Deduplicate by fields**, enter the field(s) that define a duplicate, e.g. `url` or `product.id` and `shop`.
4. Optional: turn on **Only output items not seen in previous runs** and choose a **State key** (one per pipeline).
5. Optional: add **Reference datasets**, choose **Output only these fields**, and set **Max output items** or a *Max total charge*.
6. Click **Start**. The unique items appear in the run's dataset, and the summary is in the `OUTPUT` record of the key-value store.
7. For scheduled scrapers, add this Actor as a follow-up step (for example with a schedule or an integration) and keep the same `stateKey` on every run.

### Input example

```json
{
    "datasetIds": ["aBcD1234EfGh5678", "IjKl9012MnOp3456"],
    "dedupFields": ["url"],
    "caseInsensitive": false,
    "trimStrings": true,
    "missingFieldMode": "keep-all",
    "onlyNewItems": true,
    "stateKey": "daily-products",
    "updateState": true,
    "referenceDatasetIds": [],
    "outputFields": ["url", "title", "price.value"],
    "addSourceDatasetId": true,
    "maxOutputItems": 0,
    "pageSize": 5000,
    "pushBatchSize": 1000
}
```

| Input field | Type | Default | Description |
| --- | --- | --- | --- |
| `datasetIds` | array | – | Datasets to merge (dataset picker, READ access). Needed for real use: without it the Actor runs the small demo below. |
| `demoItems` | array | 5 demo items | Used **only** when no dataset is selected: a tiny built-in example (deduplicated by `url`) so a run with empty input still shows the result. Ignored as soon as you select a dataset. Max 100 items; output items are charged like normal output items. |
| `dedupFields` | array | prefill `["url"]` | Fields whose combined value must be unique. Dot paths allowed. Empty = whole-item comparison. |
| `caseInsensitive` | boolean | `false` | Treat `"ABC"` and `"abc"` as equal. |
| `trimStrings` | boolean | `true` | Ignore leading/trailing whitespace. |
| `missingFieldMode` | string | `keep-all` | Items missing **all** dedup fields: `keep-all` (always output), `as-empty` (collapse into one), `drop`. |
| `onlyNewItems` | boolean | `false` | Skip items output by earlier runs with the same `stateKey`. |
| `stateKey` | string | `default` | Name of the remembered state (store `dedup-state-<key>`). |
| `updateState` | boolean | `true` | `false` = dry run: compare with the state without changing it. |
| `referenceDatasetIds` | array | – | Items whose key appears here are not output. |
| `outputFields` | array | – | Output only these fields (dot paths are re-nested). |
| `addSourceDatasetId` | boolean | `false` | Add `_sourceDatasetId` to each item. |
| `maxOutputItems` | integer | `0` | `0` = unlimited; also capped by *Max total charge*. |
| `pageSize` | integer | `5000` | Items read per API call (100–50,000). |
| `pushBatchSize` | integer | `1000` | Items written per API call (1–5,000). |

### Output example

**Dataset:** your original items (or only the selected fields), unique by your dedup key. With `outputFields: ["url", "title", "price.value"]` and `addSourceDatasetId: true`:

```json
[
    { "url": "https://shop.example/p/0", "title": "Product 0", "price": { "value": 0 }, "_sourceDatasetId": "aBcD1234EfGh5678" },
    { "url": "https://shop.example/p/1", "title": "Product 1", "price": { "value": 1 }, "_sourceDatasetId": "aBcD1234EfGh5678" }
]
```

**Run summary** (`OUTPUT` record in the default key-value store; numbers from a local test run):

```json
{
    "read": 20003,
    "output": 13002,
    "duplicates": 7001,
    "seenInPreviousRuns": 0,
    "droppedMissingKey": 0,
    "keptWithoutKey": 2,
    "referenceItems": 0,
    "filteredByReference": 0,
    "perDataset": {
        "src-a": { "itemCount": 12000, "read": 12000, "output": 10000 },
        "src-b": { "itemCount": 8003, "read": 8003, "output": 3002 }
    },
    "stoppedReason": null,
    "dedupFields": ["url"],
    "onlyNewItems": true,
    "stateStore": "dedup-state-e2e"
}
```

#### Output fields

| Field | Where | Description |
| --- | --- | --- |
| *(your item fields)* | Dataset | Original item, or only `outputFields` (dot paths re-nested, e.g. `price.value` → `{ "price": { "value": … } }`). |
| `_sourceDatasetId` | Dataset | ID of the dataset the item came from (only with `addSourceDatasetId`). |
| `read` | `OUTPUT` | Items read from the source datasets. |
| `output` | `OUTPUT` | Unique items written (= charged `item-output` events). |
| `duplicates` | `OUTPUT` | Duplicates found within this run. |
| `seenInPreviousRuns` | `OUTPUT` | Items skipped by only-new mode. |
| `filteredByReference` / `referenceItems` | `OUTPUT` | Items skipped because of reference datasets / items read from reference datasets. |
| `keptWithoutKey` / `droppedMissingKey` | `OUTPUT` | Items missing all dedup fields that were kept / dropped. |
| `perDataset` | `OUTPUT` | `itemCount`, `read`, `output` per source dataset. |
| `stoppedReason` | `OUTPUT` | `null`, `maxOutputItems reached`, or a *max total charge* message. |
| `dedupFields`, `onlyNewItems`, `stateStore` | `OUTPUT` | Settings used for the run. |

### Integrations and API

The unique items are in the run's default dataset, and the summary is in the `OUTPUT` record. Both are available through the Apify API and the API clients, and you can trigger the Actor with Apify schedules, webhooks or integrations after your scraper finishes.

```bash
curl -X POST "https://api.apify.com/v2/acts/grace_hartwell~dataset-merge-dedup/runs?token=[YOUR_APIFY_TOKEN]" \
  -H "Content-Type: application/json" \
  -d '{"datasetIds":["aBcD1234EfGh5678","IjKl9012MnOp3456"],"dedupFields":["url"],"onlyNewItems":true,"stateKey":"daily-products"}'
```

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('grace_hartwell/dataset-merge-dedup').call({
    datasetIds: ['aBcD1234EfGh5678', 'IjKl9012MnOp3456'],
    dedupFields: ['url'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const summary = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');
```

### FAQ

#### Which item is kept when there are duplicates?

The first one found, in the order of the datasets you selected and the order of items within each dataset. Items are written in the order they are read.

#### How do I remove duplicates by more than one field?

List several fields in `dedupFields`, e.g. `["product.id", "shop"]`. Two items count as duplicates only if **all** listed values match.

#### How does the only-new-items mode work?

The Actor stores a hash of each output item's key in the key-value store `dedup-state-<stateKey>`. The next run with the same `stateKey` skips those items and reports them as `seenInPreviousRuns`. Use different state keys for unrelated pipelines.

#### Can I do a dry run?

Yes. Set `updateState: false` to compare with the remembered state without changing it. Note that output items are still written and charged.

#### How large can the datasets be?

The Actor streams items, so the size of the source datasets is not the problem. What uses memory is the set of unique-key hashes. For very large numbers of unique keys, give the run more memory (up to 4 GB). The extra memory also increases the number of start events charged (see pricing).

#### Can it write into one of my existing datasets?

No. In this limited-permissions version, output always goes to the run's default dataset. Use the API or an integration to forward it.

#### Why did my run fail right at the start?

The pre-flight check found a dataset that doesn't exist or can't be read. Select the datasets with the dataset picker, and use the dataset **ID**, not its name. Nothing is written or charged in that case.

### Limitations: what this Actor does not do

- **Apify datasets only**: it doesn't read CSV or JSON files, URLs, or other storage types.
- **Exact matching only**: no fuzzy or similarity matching. Only string values used directly as dedup fields are trimmed or lower-cased; nested values are compared exactly.
- **Keeps the first item; does not merge fields** from duplicates into one record.
- **Items missing all dedup fields** are never deduplicated in `keep-all` mode. They are also not remembered by only-new mode, so they are output (and charged) again on every run. Use `as-empty` or `drop` if that is not what you want.
- **Reference keys are remembered**: in only-new mode with `updateState: true`, keys loaded from reference datasets are saved to the state too, so later runs keep skipping them.
- **Hidden fields and empty items** are skipped: datasets are read with `clean: true`, which drops empty items and fields whose names start with `#`.
- **Output only to the run's default dataset**; no direct export to external systems.
- **No custom sorting**: items are output in read order.
- **Each run is capped** by `maxOutputItems` and your *Max total charge*.

***

Author: Grace M. Hartwell

# Actor input Schema

## `datasetIds` (type: `array`):

Pick one or more datasets. The picker grants this run READ access to exactly these datasets (limited permissions) – nothing else in your account is touched. Required for real use; if you leave it empty, the Actor runs a small demo on "Demo items" instead.

## `dedupFields` (type: `array`):

Fields whose combined value must be unique, e.g. \["url"] or \["product.id", "shop"]. Dot paths for nested fields. Leave empty to remove only fully identical items.

## `caseInsensitive` (type: `boolean`):

Treat "ABC" and "abc" as equal.

## `trimStrings` (type: `boolean`):

Ignore leading/trailing whitespace when comparing strings.

## `missingFieldMode` (type: `string`):

keep-all: output them (never treated as duplicates). as-empty: treat missing as an empty value (all such items collapse into one). drop: skip them.

## `onlyNewItems` (type: `boolean`):

Remembers item keys in a key-value store created by this Actor (named dedup-state-<state key>) and skips items already output in earlier runs. Great for scheduled scrapers.

## `stateKey` (type: `string`):

Name of the remembered state. Use different keys for unrelated pipelines.

## `updateState` (type: `boolean`):

Turn off for a dry run that compares against the state without changing it.

## `referenceDatasetIds` (type: `array`):

Optional: items whose key appears in these datasets are treated as already seen and not output. READ access only.

## `outputFields` (type: `array`):

Optional projection, e.g. \["url", "title", "price.value"]. Empty = full items.

## `addSourceDatasetId` (type: `boolean`):

Add the ID of the dataset each item came from.

## `maxOutputItems` (type: `integer`):

0 = unlimited. Output is also capped by your 'Max total charge' budget.

## `pageSize` (type: `integer`):

Items read per API call (streaming). Lower it for very large items.

## `pushBatchSize` (type: `integer`):

Items written per API call.

## `demoItems` (type: `array`):

A tiny built-in example so a run without datasets still shows how deduplication works (e.g. Apify's default-input test). Ignored as soon as you select at least one dataset above. Max 100 items. Output items are charged like normal output items.

## Actor input object example

```json
{
  "dedupFields": [
    "url"
  ],
  "caseInsensitive": false,
  "trimStrings": true,
  "missingFieldMode": "keep-all",
  "onlyNewItems": false,
  "stateKey": "default",
  "updateState": true,
  "addSourceDatasetId": false,
  "maxOutputItems": 0,
  "pageSize": 5000,
  "pushBatchSize": 1000,
  "demoItems": [
    {
      "url": "https://example.com/product/1",
      "title": "Blue mug",
      "price": 12.9
    },
    {
      "url": "https://example.com/product/2",
      "title": "Red mug",
      "price": 11.5
    },
    {
      "url": "https://example.com/product/1",
      "title": "Blue mug (duplicate)",
      "price": 12.9
    },
    {
      "url": " https://example.com/product/3 ",
      "title": "Green mug",
      "price": 13
    },
    {
      "url": "https://example.com/product/2",
      "title": "Red mug (duplicate)",
      "price": 11.5
    }
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupFields": [
        "url"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("grace_hartwell/dataset-merge-dedup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "dedupFields": ["url"] }

# Run the Actor and wait for it to finish
run = client.actor("grace_hartwell/dataset-merge-dedup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupFields": [
    "url"
  ]
}' |
apify call grace_hartwell/dataset-merge-dedup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,grace_hartwell/dataset-merge-dedup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bNElWhoPzzAYNZSNG/builds/OfoKn04n4kSvY2woR/openapi.json