# Merge & Deduplicate Datasets (`ctriolab/merge-dedupe-datasets`) Actor

Merge many Apify datasets into one and remove duplicates by any fields, including nested dot paths. Keep the first or last copy, pick and rename fields. Streams page by page so memory stays low on large datasets. Limited permissions, clear errors before any work.

- **URL**: https://apify.com/ctriolab/merge-dedupe-datasets.md
- **Developed by:** [Ctrio Lab](https://apify.com/ctriolab) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.03 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Merge & Deduplicate Datasets

Merge any number of Apify datasets into one clean dataset and remove duplicates by one or more fields. Built for large runs: items are streamed page by page, so memory stays flat even when you merge millions of items. Runs with **limited permissions**: it can only read the datasets you pick.

### What it does

- **Merge** datasets from many runs (for example 30 daily runs of the same scraper) into one dataset, in the order you list them.
- **Deduplicate** by one field (`url`) or a combination (`name` + `address.city`). Nested values use dot paths such as `address.city` or `images.0.url`.
- **Keep the first or the last** duplicate. "Last" is handy when datasets are listed oldest first and you want the freshest copy of each item.
- **Ignore case and extra spaces** when comparing (on by default), so `ACME  Inc` and `acme inc` are the same.
- **Shape the output**: keep only some fields and rename them (`address.city` to `city`).
- **Clear errors before any work**: unknown dataset, dataset you did not share with the run, or a dedupe field that does not exist (you get the list of fields that do exist).
- **Run summary** in the key-value store record `SUMMARY`: items read, unique items, duplicates removed, items without a dedupe key, items read per dataset.

### How to use it

1. In **Datasets to merge**, pick your datasets with the picker or paste dataset IDs. A run's results live in its default dataset (run detail page, Storage tab).
2. In **Deduplicate by fields**, enter the field(s) that identify an item, e.g. `url`. Leave it empty to only merge.
   Want to try first? Use our public demo dataset `CotvbQSQ340wBAwq3` (19 coffee shops with 9 duplicates) and dedupe by `url`.
3. Start the run. Download the merged dataset as JSON, CSV or Excel, or read it through the API.

Example input:

```json
{
  "datasetIds": ["<datasetId1>", "<datasetId2>"],
  "dedupeFields": ["url"],
  "keep": "last",
  "fields": ["url", "title", "address.city", "rating"],
  "rename": { "address.city": "city" }
}
```

### Input

| Field | What it does |
|---|---|
| `datasetIds` | Datasets to read, in order. Dataset IDs or `username~dataset-name`. Required. |
| `dedupeFields` | Fields whose combined values must be unique. Dot paths for nested values. Empty = merge only. |
| `keep` | `first` (default) or `last` occurrence of each duplicate. `last` reads the input twice. |
| `normalizeValues` | Trim, collapse spaces and lowercase before comparing (default on). Objects and arrays are compared by content. |
| `fields` | Optional list of fields to output, in order. Missing values become `null`. |
| `rename` | Optional map from a field or dot path to a new name. |
| `pageSize` | Items per API call (default 1,000). Lower it for very large items. |

Items that have **none** of the dedupe fields are never treated as duplicates. They are kept and counted in `itemsWithoutDedupeKey`.

### Why it does not run out of memory

The Actor never loads a whole dataset. It reads one page (1,000 items by default) while the next page is being fetched, writes the unique items, and keeps only a 16-character hash of each unique key. That is about 55 MB per million unique keys (measured), so the default 512 MB handles several million unique items. In our tests with 256 MB of memory, a 105,000-item merge (keep first) peaked at 125 MB and finished in 48 seconds; the same data with keep last (two passes, 210,000 item reads) peaked at 88 MB.

If the platform migrates the run to another server, the Actor resumes without writing the same item twice.

### Pricing

Pay per event: **$0.05** per 1,000 items written to the output dataset, plus a tiny Actor start fee. Items read and duplicates removed are not charged. Platform usage is included. You can cap the cost of a run with the maximum charge setting; the Actor stops cleanly when it is reached.

### Limits

- Datasets must be ones your account can read. Because the Actor runs with limited permissions, it cannot list your runs by itself: pass dataset IDs (every run has one).
- Transform functions (custom JavaScript) are not supported. Use `fields` and `rename` for shaping.
- The order of the output follows the input: dataset order, then item order (for `keep: last`, the position of the last occurrence).

### FAQ

**Can I merge the results of all runs of my scraper?**
Yes. Copy the default dataset ID of each run into `datasetIds`. You can also call this Actor from an integration or webhook after each run.

**How are nested objects compared?**
By content with sorted keys, so `{"a":1,"b":2}` equals `{"b":2,"a":1}`.

**Does it change my source datasets?**
No. It only reads them (read-only permission) and writes to the run's own dataset.

Made by [Ctrio Lab](https://ctriolab.com). Questions or bugs: open an issue on the Actor's Issues tab.

# Actor input Schema

## `datasetIds` (type: `array`):

Pick one or more datasets (or paste dataset IDs). Items are read in this order. Tip: a run's results are its default dataset, shown on the run's Storage tab.

## `dedupeFields` (type: `array`):

Items with the same combination of these values are duplicates. Use dots for nested values, e.g. "address.city" or "images.0.url". Leave empty to only merge.

## `keep` (type: `string`):

"First" keeps the earliest item (dataset order, then item order). "Last" keeps the latest one, for example the newest scrape when datasets are listed oldest first. "Last" reads the input twice.

## `normalizeValues` (type: `boolean`):

Compare values after trimming, collapsing spaces and lowercasing, so "ACME  Inc" equals "acme inc" and 1 equals "1". Turn off for exact comparison. Objects and arrays are compared by content.

## `fields` (type: `array`):

Optional. Keep only these fields (dot paths allowed) in the output, in this order. Missing values become null. Leave empty to keep whole items.

## `rename` (type: `object`):

Optional. Map a field or dot path to a new output name, e.g. {"address.city": "city", "title": "name"}.

## `pageSize` (type: `integer`):

How many items are read per API call. Lower it for very large items (for example full HTML) to keep memory low.

## Actor input object example

```json
{
  "dedupeFields": [
    "url"
  ],
  "keep": "first",
  "normalizeValues": true,
  "pageSize": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

Merged and deduplicated items.

## `summary` (type: `string`):

Items read, unique items, duplicates removed and items read per dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupeFields": [
        "url"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ctriolab/merge-dedupe-datasets").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "dedupeFields": ["url"] }

# Run the Actor and wait for it to finish
run = client.actor("ctriolab/merge-dedupe-datasets").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupeFields": [
    "url"
  ]
}' |
apify call ctriolab/merge-dedupe-datasets --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ctriolab/merge-dedupe-datasets"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bBguVqCd5Yqf3gBIb/builds/1jW2h1pB0rmu1M2T2/openapi.json