# Dataset Deduplication & Merge - Remove Duplicates (`ahmed.fouaad82/dataset-dedup-merge`) Actor

Merge Apify datasets and remove duplicate rows by selected fields. First/last matching. $0.02/1,000 source rows; first 50 rows/run free.

- **URL**: https://apify.com/ahmed.fouaad82/dataset-dedup-merge.md
- **Developed by:** [Ahmed Fouad](https://apify.com/ahmed.fouaad82) (community)
- **Categories:** Developer tools, Integrations, Automation
- **Stats:** 2 total users, 1 monthly users, 66.7% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.02 / 1,000 source row processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Dataset Deduplication & Merge - Remove Duplicates

**Merge Apify datasets and remove duplicate rows** by selected fields for developers, data analysts and scheduled-scraping workflows. Keep the first or last matching row and retain source provenance; source datasets are never modified.

**Pricing: $0.02 per 1,000 source rows processed, not output rows. The first 50 source rows in each run are free.** The Pricing tab is authoritative. Apify plan allowances and any applicable platform/storage charges are separate; no additional free allowance is promised.

**Example output** (illustrative abbreviated record, not a captured result):

```json
{"item":{"id":"example-1","product":"Example product"},"occurrences":2,"sourceDatasetId":"example-dataset","sourceIndex":0}
```

**Try it:** with no sources selected, the example input uses four built-in synthetic product rows with one duplicate. Replace Source datasets with your own to process real data. The default example produces three unique rows without a billable source-row event. `sourceDatasetId` is `built-in-demo`, not a real dataset. Set `demoMode=false` to require a real source.

### What does Apify Dataset Merge & Deduplicate do?

Merge one or more [Apify](https://apify.com) datasets and **remove duplicate rows by chosen fields** using the Apify API. Use it to combine scheduled scrape results, clean repeated leads, or compare product feeds. It does not scrape websites or use proxies.

### Why use this Actor?

- Disk-backed SQLite index instead of keeping millions of keys in JavaScript memory.
- Limited permissions: source datasets receive READ access only; outputs go to the run's own default dataset.
- First or last representative, case-insensitive comparison, whitespace trimming, composite keys and duplicate-group reporting.
- Batches, API retries and visible processing counts. No browser or proxy costs.

### How to merge and deduplicate datasets

1. Select one or more **Source datasets**. Input order determines first/last priority.
2. Set **keyFields**, for example `["email"]` or `["shopId", "product.sku"]`.
3. Choose first/last and run. Export the results or connect them to Make, Zapier or another Actor.

The input tab lists every option. Example (replace the illustrative dataset IDs with your own):

```json
{"datasetIds":["WkzbQMuFYuamGv3YF"],"keyFields":["email"],"keep":"last","caseInsensitive":true,"trim":true,"ignoreEmpty":true,"batchSize":1000}
```

### Output data and exports

| Field | Type | Meaning |
|---|---|---|
| item | object | Kept original row, after optional top-level field projection |
| occurrences | integer | Count of matching source rows |
| sourceDatasetId | string | Dataset containing the representative |
| sourceIndex | integer | Zero-based original row index |

```json
[{"item":{"id":"example-1","email":"example@example.com"},"occurrences":2,"sourceDatasetId":"WkzbQMuFYuamGv3YF","sourceIndex":0}]
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Use `unwind=item` or flatten `item` when a flat export is needed. `SUMMARY` in the default key-value store contains processed/output/duplicate counts and a completion flag.

### Pricing

**$0.02 per 1,000 source rows processed** (`processed-item`, $0.00002 per row); the first 50 source rows in every run are free. The exact current price is always shown in the Pricing tab. A batch may be read before a charge limit is detected, but only the permitted source prefix is indexed. Platform storage/transfer charges, if applicable to your plan, are separate.

### Matching semantics and advanced options

- All key fields together define equality. Nested paths use dots; literal field names containing dots cannot be addressed as keys.
- Trimming and lowercasing affect comparison only, not original output. Objects compare regardless of property order; arrays remain ordered. Numbers and strings remain distinct; null and missing remain distinct.
- `ignoreEmpty=true` means rows with any missing, null or empty-string key remain separate; they are not dropped. With trimming, whitespace-only keys count as empty. False and zero are valid values.
- `onlyDuplicates=true` outputs one selected representative per repeated key, not all discarded rows.
- `pickFields` and `omitFields` operate on top-level original fields after comparison. Omit wins. Empty pick means keep all.
- Output order follows the selected representative's position in the merged source sequence.
- Sources should be finished and immutable. The row count is snapshotted per dataset; new appends after that are excluded. Read batches are unfiltered so empty/hidden rows do not break offsets.
- Memory is bounded by API/output batches, a 16 MiB SQLite cache and runtime overhead, not total row count. Disk space grows with unique keys and payload sizes. Very large individual rows need smaller batches and possibly more memory. Millions of rows are a design goal, not a measured platform guarantee.
- All modes index before producing output. Spending/item limits yield valid results for the processed prefix, marked `complete=false`; last/duplicate decisions are not global if processing stops early.

### FAQ

#### Can I supply run IDs with limited permissions?

Apify only grants storage access to explicitly referenced dataset fields. A run ID does NOT grant access to its private metadata or default dataset. For private run IDs use the included local helper:

```bash
APIFY_TOKEN=your_token node tools/resolve-runs.mjs RUN_ID > input.json
```

The helper resolves using your local account token and writes input containing `datasetIds`. Then start this Actor with that input; the Actor never accepts an account token in its input. The optional `runIds` input works only if metadata and resulting storage are already accessible. Prefer datasetIds for reliable limited-permissions operation.

#### Does it resume after migration or abort?

No. The disk index is ephemeral. Abort writes diagnostic `STATE` and exits promptly. Migration fails explicitly rather than silently losing the index. Start a fresh run; do not resurrect. Tune timeout to allow the full merge.

#### Permissions and privacy

Only pass datasets you are authorized to process. The Actor preserves your original data, which may include personal information; you remain responsible for lawful processing. It never writes to or deletes source datasets. Report problems through the Issues tab and use the API tab for integrations.

# Actor input Schema

## `datasetIds` (type: `array`):

Select datasets to merge in this order. Grants READ access only to these datasets.

## `runIds` (type: `array`):

Accessible run IDs. Private arbitrary runs cannot be read with a limited token: resolve locally using tools/resolve-runs.mjs, then provide the resulting datasetIds.

## `keyFields` (type: `array`):

One or more dot-separated field paths. All chosen values form a composite key.

## `keep` (type: `string`):

Keep the earliest or latest occurrence in dataset order.

## `caseInsensitive` (type: `boolean`):

Lowercase string key values only; original output is unchanged.

## `trim` (type: `boolean`):

Trim string key values only.

## `ignoreEmpty` (type: `boolean`):

Keep each row with any null, missing or empty key separate instead of merging them.

## `onlyDuplicates` (type: `boolean`):

One representative for each key with at least two occurrences, not every discarded row.

## `pickFields` (type: `array`):

Top-level output fields only. Applied after matching; omit wins over pick. Empty means no restriction.

## `omitFields` (type: `array`):

Top-level output fields only. Applied after matching; omit wins over pick. Empty means no restriction.

## `batchSize` (type: `integer`):

Items fetched and output per batch. Lower for large rows.

## `maxItems` (type: `integer`):

Stop gracefully after this many source rows; zero means unlimited. Results then describe only the processed prefix. First 50 source rows processed in every run are free; pricing applies to source rows, not deduplicated output rows.

## `demoMode` (type: `boolean`):

With no source datasets/run IDs, deduplicate four synthetic product rows into three examples. Real sources always take precedence. Disable to require a source.

## Actor input object example

```json
{
  "runIds": [],
  "keyFields": [
    "id"
  ],
  "keep": "first",
  "caseInsensitive": false,
  "trim": true,
  "ignoreEmpty": true,
  "onlyDuplicates": false,
  "pickFields": [],
  "omitFields": [],
  "batchSize": 1000,
  "maxItems": 50,
  "demoMode": true
}
```

# Actor output Schema

## `dataset` (type: `string`):

One row per matching key

## `summary` (type: `string`):

Counts and completion flag

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keyFields": [
        "id"
    ],
    "maxItems": 50,
    "demoMode": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("ahmed.fouaad82/dataset-dedup-merge").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "keyFields": ["id"],
    "maxItems": 50,
    "demoMode": True,
}

# Run the Actor and wait for it to finish
run = client.actor("ahmed.fouaad82/dataset-dedup-merge").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keyFields": [
    "id"
  ],
  "maxItems": 50,
  "demoMode": true
}' |
apify call ahmed.fouaad82/dataset-dedup-merge --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ahmed.fouaad82/dataset-dedup-merge"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XC6p4tXQG2GHgisDw/builds/Fi5G0TDDrjXT9Zdre/openapi.json