# Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX (`actor-forge/dataset-merge-dedupe`) Actor

Merge Apify datasets, CSV, JSON, JSONL or Excel files and pasted rows into one clean dataset: remove duplicates by any fields (case and whitespace normalised), keep only items new since your last run, left/inner join on a key, flatten nested JSON. $0.05 per 1,000 rows out.

- **URL**: https://apify.com/actor-forge/dataset-merge-dedupe.md
- **Developed by:** [ActorForge](https://apify.com/actor-forge) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.05 / 1,000 row outputs

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX

Merge Apify datasets, CSV, JSON, JSONL or Excel files and pasted rows into one clean dataset: remove duplicates by any fields (case and whitespace normalised), keep only items new since your last run, left/inner join on a key, flatten nested JSON. $0.05 per 1,000 rows out.

**Combine scraper outputs into one clean, duplicate-free dataset, reliably.** Pick any number of **Apify datasets**, add **CSV, TSV, JSON, JSON Lines or Excel (XLSX) file URLs** or paste rows as JSON, and get one merged dataset with **duplicates removed by the fields you choose** (email, URL, product ID, company name + city, or the whole row). Values are **normalised before comparing**, so `" ACME  Ltd"` and `"acme ltd"`, or `42` and `"42"`, count as the same, while your rows stay exactly as they were. Turn on **only new items** and the actor remembers what it delivered (in a store on your own account), so a **scheduled scrape outputs only the listings, jobs, leads or prices that are new since the last run**, perfect for alerts, CRMs and Slack/Make/n8n/Zapier pipelines. Add a **left or inner join** on a key (e.g. enrich leads with a company lookup table by domain), **flatten nested JSON** into spreadsheet columns, keep or drop fields, and tag each row with its source. It streams datasets page by page, so millions of rows work, and it runs with **limited permissions**: it can read only the datasets you pick. Export the result as CSV, Excel, JSON, JSONL or XML from the dataset. **$0.05 per 1,000 rows delivered**; duplicates and skipped rows are free.

### What data does it return?

Every run outputs clean, schema-validated rows like these:

| id | name | \_source | url | title | price |
| --- | --- | --- | --- | --- | --- |
| 1 | a | input:rows |  |  |  |
| 2 | b | input:rows |  |  |  |
|  |  | dataset:Xk3...R9 | https://shop.example.com/p/42 | Trail Shoe X2 | 129 |

### How much does it cost?

Transparent pay-per-event pricing — you only pay for what you get, and the
actor **always respects your run budget**: set a maximum total charge and it
stops cleanly at the cap instead of overrunning.

| Event | Price | Charged | Worked example |
| --- | --- | --- | --- |
| `row-output` | $0.00005 | per row delivered (duplicates removed, rows already seen and failed sources are free) | Merge 3 datasets of 40,000 rows into 25,000 unique rows = $1.25 |
| `apify-actor-start` | $0.00005 | per run per GB of memory (platform start fee) | One run at 1 GB = $0.00005 |

### Input options

| Field | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `datasetIds` | array | no | — | Apify datasets to merge (pick them, or pass dataset IDs via the API). Picking a dataset here gives this run read access to it; nothing else on your account is readable. Read in the order given, after any pasted rows. |
| `fileUrls` | array | no | — | Public URLs of CSV/TSV (delimiter auto-detected, first row = column names), JSON (an array, or an object with an items/data/results array), JSON Lines or XLSX (first visible sheet) files, up to 100 MB each. Read after the datasets. |
| `rows` | array | no | — | Rows pasted as a JSON array of objects. Read first. |
| `dedupeFields` | array | no | — | Rows with the same values in all of these fields are duplicates; the first one is kept. Dot paths reach nested values (e.g. address.postcode). Leave empty to compare whole rows. Rows where all these fields are empty are always kept. |
| `normalizeValues` | boolean | no | `true` | Compare text case-insensitively with leading/trailing and repeated whitespace ignored (" ACME  Ltd" = "acme ltd"), and a number equal to its text form (42 = "42"). The rows themselves are not changed. |
| `onlyNewItems` | boolean | no | `false` | Remember the dedupe keys of every delivered row (in a key-value store on your own account) and skip rows already delivered by an earlier run with the same history name. Ideal for scheduled scrapes: each run outputs only what is new. |
| `historyKey` | string | no | `"default"` | Separate 'only new items' memories for different pipelines, e.g. jobs-london or competitor-prices. Letters, digits, - and *. |
| `joinDatasetId` | string | no | — | Optional. A dataset whose rows are joined onto the merged rows (e.g. company details by domain). Up to 500,000 lookup rows; the first row per key is used. |
| `joinFileUrl` | string | no | — | Optional, instead of a lookup dataset: a CSV, JSON, JSONL or XLSX file URL with the lookup rows. |
| `joinOn` | string | no | — | Field of the merged rows to match on (dot paths allowed). Required for a join. |
| `joinLookupField` | string | no | — | Field of the lookup rows to match on. Defaults to the same name as 'Join on field'. |
| `joinPrefix` | string | no | `""` | Optional prefix for the fields copied from the lookup row, e.g. company*. Lookup fields only fill fields that are missing or empty in the row, so use a prefix to keep both versions. |
| `joinType` | string | no | `"left"` | Left: keep every row (unmatched rows unchanged). Inner: keep only rows that matched a lookup row. |
| `flatten` | boolean | no | `false` | Turn nested objects and arrays into flat columns (address.city, tags.0), handy for spreadsheets and CRMs. |
| `fields` | array | no | — | Optional. Output only these fields, in this order (dot paths allowed; the path becomes the column name). |
| `omitFields` | array | no | — | Optional. Drop these fields from every row (after flattening, use the flat names). |
| `addSourceField` | boolean | no | `false` | Tag each row with where it came from (input:rows, dataset:\<id> or file:\<url>). |
| `maxItems` | integer | no | `1000000` | Stop after delivering this many rows. |

### Limits & honest notes

- Sources are read in order (pasted rows, then datasets, then files) and the first occurrence of each key is kept. Rows whose dedupe fields are all empty are never treated as duplicates.
- Dedupe keys are kept in memory as 53-bit hashes: about 5–10 million unique keys per run at 1 GB (raise the memory for more). The chance of two different keys colliding is negligible (about 1 in 10,000 at 1 million keys).
- Only new items: the remembered keys live in the key-value store "dataset-merge-dedupe-history" on your own account (one record per history name, newest 5 million keys). Delete the record to start over. Rows not delivered (for example after hitting the spend limit) are not remembered.
- Datasets: with limited permissions the run can read only the datasets selected in the Datasets / lookup dataset fields (or passed by ID in the API input). A dataset that cannot be read is reported in the run summary and the other sources are still merged.
- Files are downloaded once (up to 100 MB each) and parsed in memory. CSV values stay text (no type guessing, so IDs, postcodes and phone numbers keep their leading zeros); XLSX reads the first visible sheet with typed values and ISO dates. Private and internal network addresses are refused at every redirect hop.
- Joins hold the lookup side in memory (up to 500,000 rows; the first row per key wins). Lookup fields fill only fields that are missing or empty in the row; use a prefix to keep both.
- When a run delivers nothing (no sources, empty sources, all duplicates or nothing new), it adds one free row with \_status and \_message explaining why and still succeeds.

### Related ActorForge actors

- [Word, Excel & PowerPoint to Markdown](https://apify.com/actor-forge/office-to-markdown) — turn spreadsheets and documents into JSON or Markdown
- [Website Change Monitor](https://apify.com/actor-forge/website-change-monitor) — alerts when a page changes

### Changelog

- **2026-10-23** — Initial release: merge Apify datasets, CSV/TSV/JSON/JSONL/XLSX URLs and pasted rows; normalised multi-field dedupe; only new items across runs; left/inner join; flatten; field selection; source tag.

***

*Built and maintained by **ActorForge**. Reliability first: every actor in the
portfolio is monitored by an automated nightly self-test, and breakages are
typically fixed the same day.*

# Actor input Schema

## `datasetIds` (type: `array`):

Apify datasets to merge (pick them, or pass dataset IDs via the API). Picking a dataset here gives this run read access to it; nothing else on your account is readable. Read in the order given, after any pasted rows.

## `fileUrls` (type: `array`):

Public URLs of CSV/TSV (delimiter auto-detected, first row = column names), JSON (an array, or an object with an items/data/results array), JSON Lines or XLSX (first visible sheet) files, up to 100 MB each. Read after the datasets.

## `rows` (type: `array`):

Rows pasted as a JSON array of objects. Read first.

## `dedupeFields` (type: `array`):

Rows with the same values in all of these fields are duplicates; the first one is kept. Dot paths reach nested values (e.g. address.postcode). Leave empty to compare whole rows. Rows where all these fields are empty are always kept.

## `normalizeValues` (type: `boolean`):

Compare text case-insensitively with leading/trailing and repeated whitespace ignored (" ACME  Ltd" = "acme ltd"), and a number equal to its text form (42 = "42"). The rows themselves are not changed.

## `onlyNewItems` (type: `boolean`):

Remember the dedupe keys of every delivered row (in a key-value store on your own account) and skip rows already delivered by an earlier run with the same history name. Ideal for scheduled scrapes: each run outputs only what is new.

## `historyKey` (type: `string`):

Separate 'only new items' memories for different pipelines, e.g. jobs-london or competitor-prices. Letters, digits, - and \_.

## `joinDatasetId` (type: `string`):

Optional. A dataset whose rows are joined onto the merged rows (e.g. company details by domain). Up to 500,000 lookup rows; the first row per key is used.

## `joinFileUrl` (type: `string`):

Optional, instead of a lookup dataset: a CSV, JSON, JSONL or XLSX file URL with the lookup rows.

## `joinOn` (type: `string`):

Field of the merged rows to match on (dot paths allowed). Required for a join.

## `joinLookupField` (type: `string`):

Field of the lookup rows to match on. Defaults to the same name as 'Join on field'.

## `joinPrefix` (type: `string`):

Optional prefix for the fields copied from the lookup row, e.g. company\_. Lookup fields only fill fields that are missing or empty in the row, so use a prefix to keep both versions.

## `joinType` (type: `string`):

Left: keep every row (unmatched rows unchanged). Inner: keep only rows that matched a lookup row.

## `flatten` (type: `boolean`):

Turn nested objects and arrays into flat columns (address.city, tags.0), handy for spreadsheets and CRMs.

## `fields` (type: `array`):

Optional. Output only these fields, in this order (dot paths allowed; the path becomes the column name).

## `omitFields` (type: `array`):

Optional. Drop these fields from every row (after flattening, use the flat names).

## `addSourceField` (type: `boolean`):

Tag each row with where it came from (input:rows, dataset:<id> or file:<url>).

## `maxItems` (type: `integer`):

Stop after delivering this many rows.

## Actor input object example

```json
{
  "rows": [
    {
      "id": 1,
      "name": "a"
    },
    {
      "id": 1,
      "name": "a"
    },
    {
      "id": 2,
      "name": "b"
    }
  ],
  "dedupeFields": [
    "id"
  ],
  "normalizeValues": true,
  "onlyNewItems": false,
  "historyKey": "default",
  "joinPrefix": "",
  "joinType": "left",
  "flatten": false,
  "addSourceField": false,
  "maxItems": 1000000
}
```

# Actor output Schema

## `results` (type: `string`):

The merged, deduplicated rows (export as CSV, Excel, JSON, JSONL or XML from the dataset).

## `summary` (type: `string`):

Rows read and delivered, duplicates removed, rows skipped as already seen, join matches, per-source counts and errors, billing.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "rows": [
        {
            "id": 1,
            "name": "a"
        },
        {
            "id": 1,
            "name": "a"
        },
        {
            "id": 2,
            "name": "b"
        }
    ],
    "dedupeFields": [
        "id"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("actor-forge/dataset-merge-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "rows": [
        {
            "id": 1,
            "name": "a",
        },
        {
            "id": 1,
            "name": "a",
        },
        {
            "id": 2,
            "name": "b",
        },
    ],
    "dedupeFields": ["id"],
}

# Run the Actor and wait for it to finish
run = client.actor("actor-forge/dataset-merge-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "rows": [
    {
      "id": 1,
      "name": "a"
    },
    {
      "id": 1,
      "name": "a"
    },
    {
      "id": 2,
      "name": "b"
    }
  ],
  "dedupeFields": [
    "id"
  ]
}' |
apify call actor-forge/dataset-merge-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,actor-forge/dataset-merge-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qatyCuLQ6pgFRnPDn/builds/m6V5yTnJ7EbBRLedP/openapi.json