# Dataset Dedup, Merge & Compare: Only New Items, Reconcile (`neorhapsody/dataset-merge-dedup`) Actor

Merge Apify datasets and remove or merge duplicates (with normalized URLs, phones, domains), keep only items new since earlier runs using persistent memory, or compare two datasets or CSV files and get field-level differences. Limited permissions.

- **URL**: https://apify.com/neorhapsody/dataset-merge-dedup.md
- **Developed by:** [fikri marhan](https://apify.com/neorhapsody) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 output rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Dedup, Merge & Compare: Only New Items, Reconcile

**Remove or merge duplicates across Apify datasets, get only items that are new since earlier runs, or compare two datasets (or CSV files) and see exactly what changed.** It runs with **limited permissions** and works on your private datasets: pick them in the input, or pass `{{resource.defaultDatasetId}}` from an integration.

Use it after any scraper: dedupe Google Maps leads by phone or website domain, merge overlapping runs into one complete row, send only new job or real-estate listings to Slack every day, or reconcile a price list or bank export against last month's.

### What it does

- **Merge datasets**: combine any number of datasets, in the order you list them.
- **Deduplicate**: by one or more fields (`url`, `title` + `company`, nested `location.city`), or by the whole item. Choose which copy survives:
  - **First** (fastest, single streaming pass)
  - **Last** (newest data wins)
  - **Merge duplicates**: all copies become one row with the most complete value per field. Empty fields are filled from other copies, lists such as emails are combined, and the row records its sources (`_mergedCount`, `_mergedFrom`, `_filledFrom`, `_conflictingFields`).
- **Normalized matching** (optional):
  - ignore case, extra whitespace and accents
  - **URLs**: ignores http/https, `www.`, trailing slash, tracking parameters and parameter order
  - **phone numbers**: digits only, so `+60 12-345 6789` = `012-345 6789`
  - **domain only**: `https://www.acme.com/contact` = `sales@acme.com` = `acme.com`
- **Only new items, across runs**: give a **memory name** (e.g. `daily-jobs`). The Actor remembers every item it has output under that name, in a key-value store in your own account, and later runs output only items it has never seen. You can also exclude items found in specific datasets.
- **Compare & reconcile two datasets or CSV/JSON files**: match rows by key fields (`invoiceNo=reference` when names differ) and get one row per record with status:

  - `matched`
  - `different`, with field-level differences and the numeric or date gap
  - `only_in_a`
  - `only_in_b`

  Number tolerance (absolute or %) and date tolerance (days) let rounding and posting delays count as equal.
- **Filter and transform**: equals, contains, greater/less than, regex, exists, or in a list; keep, drop or rename fields; tag rows with their source dataset.
- **Reports**:
  - `REPORT` is an HTML report: duplicate clusters and field fill rates in dedup mode, a reconciliation summary in compare mode.
  - `RUN_SUMMARY` (JSON) has all counts, data-quality stats, memory size, errors by code and the events charged.

### Built for large datasets and limited permissions

- Streams datasets page by page, prefetching the next page. Keys are stored as compact hashes.
- In a local test with 1,000,000 rows (25% duplicates):

  - Keep first: 6 s, about 230 MB of memory
  - Keep last: 11 s, about 280 MB
  - Merge duplicates: 15 s, about 600 MB
  - Excluding against a 750,000-key memory: about 235 MB

  On the platform, reading and writing through the Apify API takes longer than this local test. The default memory is 1 GB.
- Datasets you **pick** in the input are readable under limited permissions, so no full-permission approval is needed. This matters for schedules, integrations and AI agents.
- If a dedup run of another tool failed with *"Insufficient permissions for the dataset"*, it was most likely given a private dataset ID as plain text under limited permissions. Here the dataset fields are dataset pickers, which grant read access to exactly the datasets you choose.
- Dataset IDs, run IDs and Console URLs pasted into **Sources as text** are recognised. Under limited permissions, text gives no access to private data, so for those you get a clear error record that says which dataset ID to pick instead. Public datasets work directly.

### Try it free

- **Dedup demo:** run with no datasets selected. It processes a public sample dataset (40 fictional products, 10 duplicate URLs) and outputs 30 unique rows, free.
- **Compare demo:** set *What to do* to *Compare & reconcile* and leave both sides empty. It compares last week's and this week's sample products (changed prices, stock and titles, plus removed and new items), free.

### Input examples

Only new listings after every scraper run (add as an integration on your scraper's task):

```json
{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "dedupFields": ["url"],
  "normalizeUrls": true,
  "memoryName": "daily-listings"
}
```

Merge several runs of the same scraper (e.g. overlapping Google Maps searches) into one complete row per place:

```json
{
  "datasetIds": ["<run1DatasetId>", "<run2DatasetId>", "<run3DatasetId>"],
  "dedupFields": ["placeId"],
  "keep": "merge"
}
```

Reconcile invoices against payments (amounts within 0.01, dates within 3 days):

```json
{
  "mode": "compare",
  "fileUrlA": "https://example.com/invoices.csv",
  "fileUrlB": "https://example.com/bank-export.csv",
  "compareKeyFields": ["invoiceNo=reference"],
  "compareFields": ["amount=paidAmount", "dueDate=paidOn"],
  "numericTolerance": 0.01,
  "dateToleranceDays": 3,
  "includeMatched": false
}
```

### Output

Rows go to the run's default dataset, or to a dataset you pick in *Write to existing dataset*. Export them as JSON, CSV, Excel or HTML, or fetch them via the API. Dataset views:

- **Results**: everything
- **Compare: summary per row**
- **Compare: details**
- **Merge: source tracking**
- **Errors**

A compare row looks like this:

```json
{ "status": "different", "key": { "url": "https://shop.example.com/p/2" }, "differenceCount": 1, "differentFields": "price",
  "differences": [{ "field": "price", "a": 42.99, "b": 38.69, "kind": "number", "difference": -4.3, "percent": -10.0023 }],
  "a": { "...": "row from A" }, "b": { "...": "row from B" } }
```

### Pricing

Pay per event, with no separate platform-usage charges:

- **$0.20 per 1,000 output rows** (`output-row`, $0.0002 per row written: a unique or merged item, or a comparison row)
- **$0.001 per run** (`run`, charged once, after the input is validated)

These are **free**: removed duplicates, filtered rows, already-seen rows, error records and demo runs. In compare mode, turning off *Output identical rows* means you pay only for differences and unmatched rows.

Examples:

- Merging 50,000 rows into 30,000 unique rows costs $6.00 + $0.001.
- A daily "only new" run that finds 40 new items costs $0.009.

If you set a **maximum cost per run**, the Actor writes only as many rows as fit and then stops cleanly. `RUN_SUMMARY` shows when that happened.

### How to remove duplicates from an Apify dataset

1. Pick one or more datasets under **Datasets to merge**.
2. Under **Dedup by fields**, enter the fields that identify a duplicate, e.g. `url`, or `phone` with *Normalize phone numbers* on.
3. Run it. Unique rows go to the dataset. `REPORT` shows the duplicate clusters and how complete each field is.

### How to get only new items since the last run

Set a **memory name** and run the Actor after each scraper run, with `{{resource.defaultDatasetId}}` in an integration. The first run outputs everything and remembers it; later runs output only new items. To start over, tick *Reset memory* or use a new name. The memory is the key-value store `dedup-memory-<name>` under Storage in your account.

### How to compare two datasets or CSV files

Choose *Compare & reconcile*, pick dataset A and B (or paste public CSV/JSON URLs), set the key fields, and optionally the fields to compare and the tolerances. Filter the output by `status`, or open `REPORT` for totals, differences per field and the first 100 examples of each kind. Rows are matched one-to-one. When a key appears more than once, the closest pairs are matched first and extra rows are listed as only in A or B (many-to-one matching is not supported yet).

### FAQ

**Does it need full permissions?** No. It reads only the datasets you pick, writes to its own storage (or to a dataset you pick with write access), and keeps memory in a key-value store it creates in your account.

**Is matching fuzzy?** It's normalized, not similarity-based. Values match when they're equal after the normalization options you choose. That avoids merging rows that only look alike.

**How is "most complete" decided when merging?** Non-empty values beat empty ones, lists are combined without repeats, and nested objects are merged. When two copies have different non-empty values, the first one (in dataset order) is kept and the field is listed in `_conflictingFields`.

**What about rows where the dedup field is empty?** They can't be matched reliably, so by default they're kept as separate rows and counted as `emptyKeyRows` in `RUN_SUMMARY` (and they're not stored in memory). To treat them as duplicates of each other, turn on *Treat empty keys as duplicates*.

**How big can datasets be?** Dedup modes stream, so millions of rows work on the default 1 GB. Merge mode keeps only open duplicate groups in memory. Compare mode keeps the smaller side in memory: a few hundred thousand rows per side fit in 1 GB, and for more you can raise memory.

### Troubleshooting

A source that can't be read doesn't stop the run. Each problem is written as an **error record** to the run's default dataset, with `isError: true`, an `errorCode`, a plain-English `errorMessage` and the offending `input`. The *Errors* view shows them. Error records are free, and counts per code are in `RUN_SUMMARY`.

| Code | Meaning | What to do |
|---|---|---|
| `INVALID_INPUT` | Malformed input: a bad filter, a bad ID, a non-URL in a file field. Fatal problems stop the run with this code in the status message. | Fix the field named in the message. |
| `DATASET_NOT_ACCESSIBLE` | A dataset doesn't exist, or wasn't picked in a dataset field (text gives no access under limited permissions). | Pick the dataset with the dataset picker, or check the ID. |
| `RUN_ID_NOT_SUPPORTED` | A run ID or run URL was given. Limited-permission Actors can't look up your runs. | Open the run → Storage tab → copy the Dataset ID and pick it. In integrations use `{{resource.defaultDatasetId}}`. |
| `FILE_READ_FAILED` | A CSV/JSON URL couldn't be downloaded or parsed. | Make sure the URL is public (no login) and the file is CSV, JSON or JSON Lines. |
| `DATASET_READ_FAILED` | Reading stopped partway through (API error after retries). | Re-run. If it keeps happening, report it. |
| `OUTPUT_NOT_WRITABLE` | The output dataset can't be opened for writing. | Pick it in *Write to existing dataset*, or leave it empty. |
| `MEMORY_NOT_AVAILABLE` / `MEMORY_NOT_SAVED` | The memory store couldn't be opened or saved. | Try another memory name. If it persists, report it. |
| `FIELD_NOT_FOUND` (warning) | A dedup, filter or key field never appears. It's usually a typo or a field the scraper renamed. | Check the field path, e.g. `location.city`. |
| `MEMORY_CONFIG_CHANGED` (warning) | Dedup fields or matching options changed since the memory was created. | Use a new memory name to start clean. |
| `DUPLICATE_KEYS` (warning) | In compare mode, some keys appear more than once. | Add key fields to make keys unique. |
| `NO_KEY_FIELDS` (warning) | Compare without key fields only matches identical rows. | Set key fields. |
| `DEMO_MODE` (warning) | Nothing was selected, so the free demo ran. | Select your data. |
| `DATASET_EMPTY` (warning) | A selected dataset or file has no rows. | None. |
| `MAX_CHARGE_REACHED` (warning) | Your maximum cost per run was reached and the run stopped early. | Raise the limit. |
| `INTERNAL_ERROR` | An unexpected bug. | Please report it (see below). |

**Found a bug or a wrong result?** Open an **Issue** on this Actor's page in Apify Store (the *Issues* tab) and describe what you expected. Please **paste the run URL into the issue**: Apify then shares that run (log, input, dataset and RUN_SUMMARY) with the developer automatically, and only that run. Alternatively, enable **Settings → Login & Privacy → Share run data with developers** for this Actor. The Actor sends no telemetry anywhere; everything stays in your own run storage.

# Actor input Schema

## `mode` (type: `string`):

Dedup & merge: combine datasets and remove or merge duplicates (default). Compare & reconcile: match two datasets (or CSV/JSON files) by key and report matched rows, field-level differences, and rows only in A or only in B.

## `datasetIds` (type: `array`):

Pick one or more datasets (dataset ID or name; in integrations use {{resource.defaultDatasetId}}). Items are read in the order listed. Picking grants this Actor read access, so private datasets work under limited permissions. Leave empty (and Sources empty) for a free demo on a public sample dataset.

## `sources` (type: `array`):

Optional. Paste dataset IDs, run IDs, or Apify Console/API URLs of datasets or runs. Note: text gives no access to private data under limited permissions, so private datasets must be picked above; public datasets work here. For anything unreadable you get a clear error record explaining what to pick instead.

## `dedupEnabled` (type: `boolean`):

Remove duplicate items across all selected datasets.

## `dedupFields` (type: `array`):

Fields that identify a duplicate, e.g. \["url"] or \["title","company"]. Dot paths for nested fields work ("location.city"). Leave empty to compare whole items.

## `keep` (type: `string`):

First occurrence: fastest, single streaming pass. Last occurrence: newest data wins (two passes, low memory). Merge duplicates: combine all copies into one item, keeping the most complete value per field (empty fields filled from other copies, lists combined) with \_mergedCount, \_mergedFrom, \_filledFrom and \_conflictingFields for source tracking.

## `mergeArrays` (type: `boolean`):

In merge mode, combine list fields (e.g. emails, phones) from all duplicates without repeats. Off: keep the first non-empty list.

## `dedupEmptyKeys` (type: `boolean`):

Off (default): rows where all dedup fields are empty (e.g. no website) are kept as separate rows, because they can't be matched reliably. On: they count as duplicates of each other, so only one is kept.

## `caseInsensitive` (type: `boolean`):

Treat "Apify" and "apify" as the same value when comparing dedup fields.

## `trimWhitespace` (type: `boolean`):

Trim and collapse whitespace before comparing dedup fields.

## `normalizeDiacritics` (type: `boolean`):

Treat "Café Zürich" and "Cafe Zurich" as the same value.

## `normalizeUrls` (type: `boolean`):

For values that look like URLs: ignore http/https, www., trailing slash, #fragment, tracking parameters (utm\_\*, fbclid, gclid…) and query parameter order.

## `normalizePhones` (type: `boolean`):

For values that look like phone numbers: compare digits only, ignoring formatting, leading zeros and country codes (last 9 digits), so "+60 12-345 6789" matches "012-345 6789".

## `normalizeDomains` (type: `boolean`):

For values that look like URLs, emails or domains: compare only the domain, so "https://www.acme.com/contact", "acme.com" and "sales@acme.com" match. Use it to dedupe companies by website.

## `excludeDatasetIds` (type: `array`):

Optional. Items whose dedup key already exists in these datasets (e.g. the previous run's output) are dropped, so you get only NEW items.

## `memoryName` (type: `string`):

Optional. Give a name (e.g. "daily-jobs") and the Actor remembers every item it has output under that name, in a key-value store named dedup-memory-<name> in your own account. Later runs with the same name output only items never seen before. Ideal for scheduled scrapers and integrations. Works under limited permissions.

## `resetMemory` (type: `boolean`):

Forget everything remembered under the memory name before this run.

## `datasetA` (type: `string`):

Compare mode. First dataset. Leave both sides empty for a free demo (last week's vs this week's sample products).

## `datasetB` (type: `string`):

Compare mode. Second dataset.

## `fileUrlA` (type: `string`):

Compare mode. Public URL of a CSV, JSON or JSON Lines file to use as side A instead of a dataset (e.g. a bank export or price list). Max 200 MB.

## `fileUrlB` (type: `string`):

Compare mode. Public URL of a CSV, JSON or JSON Lines file to use as side B.

## `compareKeyFields` (type: `array`):

Fields that identify the same record on both sides, e.g. "id" or "sku". If names differ, write fieldInA=fieldInB, e.g. "invoiceNo=reference". Matching uses the case/whitespace/accent/URL/phone/domain options above. Rows are paired one-to-one; extra rows with a repeated key are reported as only in A/B.

## `compareFields` (type: `array`):

Optional. Fields to check for differences (fieldInA=fieldInB if names differ). Empty = every field present on both sides except the key fields.

## `ignoreFields` (type: `array`):

Optional. Fields never compared, e.g. scrapedAt or updatedAt.

## `numericTolerance` (type: `number`):

Numbers that differ by at most this much count as equal, e.g. 0.01. Numbers are recognised in text too ("$1,234.50", "RM 12", "(15.00)").

## `numericTolerancePercent` (type: `number`):

Numbers within this percentage of the larger value count as equal, e.g. 1 for 1%. If both tolerances are set, the larger allowance applies.

## `dateToleranceDays` (type: `number`):

Dates within this many days count as equal, e.g. 2 for posting delays. Use ISO dates (2026-10-23) where possible; US-style 10/23/2026 is read as month/day.

## `includeMatched` (type: `boolean`):

Also output rows that matched with no differences (each is a charged row). Turn off to get only differences and unmatched rows.

## `filters` (type: `array`):

Keep only items matching ALL filters. Each filter: {"field": "price", "operator": "greaterThan", "value": 10}. Operators: equals, notEquals, contains, notContains, greaterThan, lessThan, exists, notExists, regex, in (comma-separated list). String comparisons are case-insensitive.

## `pickFields` (type: `array`):

Optional. Output only these fields (dot paths allowed). Leave empty to keep all fields.

## `omitFields` (type: `array`):

Optional. Fields to drop from every item.

## `renameFields` (type: `array`):

Optional. Old field name -> new field name.

## `addSourceDatasetField` (type: `boolean`):

Add a \_sourceDatasetId field to each output item.

## `maxItemsPerDataset` (type: `integer`):

Read at most this many items from each dataset or side (0 = all). Useful for testing.

## `outputDatasetId` (type: `string`):

Optional. Append results to this dataset instead of the run's default dataset.

## Actor input object example

```json
{
  "mode": "dedup",
  "dedupEnabled": true,
  "dedupFields": [
    "url"
  ],
  "keep": "first",
  "mergeArrays": true,
  "dedupEmptyKeys": false,
  "caseInsensitive": true,
  "trimWhitespace": true,
  "normalizeDiacritics": false,
  "normalizeUrls": false,
  "normalizePhones": false,
  "normalizeDomains": false,
  "resetMemory": false,
  "numericTolerance": 0,
  "numericTolerancePercent": 0,
  "dateToleranceDays": 0,
  "includeMatched": true,
  "addSourceDatasetField": false,
  "maxItemsPerDataset": 0
}
```

# Actor output Schema

## `results` (type: `string`):

All output rows plus error records.

## `compare` (type: `string`):

Compare mode: one row per record with status and differing fields.

## `compareDetails` (type: `string`):

Compare mode: field-level differences with both rows.

## `merged` (type: `string`):

Merge mode: copies merged, conflicting fields and sources.

## `errors` (type: `string`):

Error records: code, message and input.

## `report` (type: `string`):

Data-quality report (dedup) or reconciliation report (compare).

## `runSummary` (type: `string`):

RUN_SUMMARY JSON: counts, data quality, memory, errors by code and charged events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupFields": [
        "url"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("neorhapsody/dataset-merge-dedup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "dedupFields": ["url"] }

# Run the Actor and wait for it to finish
run = client.actor("neorhapsody/dataset-merge-dedup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupFields": [
    "url"
  ]
}' |
apify call neorhapsody/dataset-merge-dedup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neorhapsody/dataset-merge-dedup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lU95PlB385oxx36aZ/builds/bqeakkF39iCYYCXKM/openapi.json