# Merge & Deduplicate Datasets - Remove Duplicates (`luxe_tasks/dataset-dedupe`) Actor

Merge Apify datasets and JSON items, remove duplicates by key fields, then pick, rename or cast fields. Reads only the datasets you pick.

- **URL**: https://apify.com/luxe_tasks/dataset-dedupe.md
- **Developed by:** [Luxe Tasks](https://apify.com/luxe_tasks) (community)
- **Categories:** Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 output rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Merge & Dedupe

Merge one or more Apify datasets (and/or JSON items you paste in), remove duplicate rows by the key fields you choose, then optionally pick,
omit, rename or cast fields. The result is written to the run's default dataset, which you can export from Apify as JSON, CSV, Excel and more.

### Pricing

$0.30 per 1,000 output rows (event `output-row`, $0.0003 per row written) plus Apify's start fee of $0.005 per run per GB of memory (`apify-actor-start`), so $0.005 at the 1,024 MB default and up to $0.02 at the 4,096 MB maximum.

- The per-row charge applies only to rows written to your output dataset: duplicates removed, rows dropped by your settings and rows of the bundled sample have no per-row charge, and Apify's per-run start fee applies to every run, including sample runs and runs that stop on invalid input.
- An output row is one row in the final dataset after merging, dedupe and field changes; merging adds no rows, and rows kept by `missingKeyPolicy: keep` each count as one output row.
- If you set a **maximum total charge** on the run, the Actor writes only as many rows as that budget covers (after the start fee) and then stops with a clear message.

### Your data, your responsibility

You are the controller of the data you merge; the developer acts only as your processor. Do not submit special-category personal data (for example health, biometric, religious or political data) without a lawful basis to process it.

- **Reads only what you pick.** The Actor is designed for Apify's *limited permissions* mode: the dataset field uses Apify's dataset picker with read-only access, so the run can read only the datasets you select. It never creates, changes or deletes your other storages.
- **No outside calls.** The Actor talks only to the Apify API. It does not fetch URLs found in your data and sends nothing to any third party.
- **No row contents in logs.** The log and the `RUN_SUMMARY` record contain counts, dataset IDs and the field names you chose, never your field values.
- **Stays in your Apify account.** Results are written only to this run's default storage in your Apify account; the developer does not copy or keep your rows and does not use them to train AI models.
- **Formula escaping on by default.** Text values starting with `=`, `+`, `-`, `@`, tab or carriage return are prefixed with `'` so they are not run as formulas when you open a CSV/Excel export (turn off with `escapeFormulas: false`). Text such as `"-5"` is prefixed too; cast such fields to `number` with `castFields` if you want numbers.

### Input

| Field | Default | What it does |
|---|---|---|
| `datasetIds` | – | Datasets to merge, in order: pick them with the dataset picker or paste their IDs (read-only). Up to 20. |
| `items` | – | JSON array of objects to merge after the datasets. Up to 10,000 items. |
| `dedupeFields` | whole row | Field names (dot paths such as `product.sku` allowed) that make a row unique. Empty = compare the whole row. Up to 50 fields. |
| `keep` | `first` | Which copy of a duplicate to keep: `first` or `last` (in dataset order, then `items`). |
| `trimWhitespace` | `true` | Ignore leading/trailing spaces when comparing keys. Output values are not changed. |
| `caseInsensitive` | `false` | Compare text keys ignoring upper/lower case. |
| `missingKeyPolicy` | `keep` | Rows with all key fields missing: `keep` each one, `drop` them, or `group` them as one key. |
| `emptyCountsAsMissing` | `true` | Treat `""` and `null` key values as missing. |
| `pickFields` / `omitFields` | – | Keep only / remove these fields in the output. |
| `renameFields` | – | Object of `{"oldName": "newName"}`. |
| `castFields` | – | Object of `{"field": "string" \| "number" \| "integer" \| "boolean"}`. Values that cannot be cast are left unchanged and counted. |
| `escapeFormulas` | `true` | See "Formula escaping" above. |
| `addSourceFields` | `false` | Add `_sourceType`, `_sourceDatasetId`, `_sourceIndex` to each row. |
| `maxOutputRows` | no limit | Stop after writing this many rows. |

If you give neither `datasetIds` nor `items`, the Actor runs on a small bundled sample (6 rows, 1 duplicate) so you can see the output. Sample rows have no per-row charge; the per-run start fee still applies.

Rules worth knowing:

- Keys are compared on the **input** field names, before pick/omit/rename/cast. `pickFields`, `omitFields` and `renameFields` use input names; `castFields` uses output (renamed) names.
- Values are compared by type: `"1"` and `1` are different keys.
- The output keeps the order in which kept rows appeared.
- A run reads at most 2,000,000 rows in total (see Limits).

### Output

- **Default dataset**: the merged, de-duplicated rows.
- **Key-value store record `RUN_SUMMARY`**: counts for the run, e.g. `rowsRead`, `duplicatesRemoved`, `outputRows`, `chargedEvents`, `missingKey`, `castFailures`, `formulaEscaped`, `budgetReached`, `maxOutputReached`, `rowsReadCapReached`.

### Limits

- Up to 20 datasets, 10,000 inline items and 2,000,000 rows read per run (limits set by this Actor for memory safety, not Apify limits).
- If a run reaches the 2,000,000-row read limit, it stops reading, says so in the log and RUN_SUMMARY, and removes duplicates only among the rows it read.
- The number of rows mostly affects run time; memory depends mainly on how large each row is (see Known limitations).
- Tested at 1,024 MB of memory with rows of about 500 KB each (76 rows, about 38 MB); because the first read page is 50 rows, much wider rows can run out of memory at that size, so give runs with wider rows more memory.
- If the platform restarts the run, rows already written are not written or charged again.

### Known limitations

- **Very large rows may need more memory.** With small rows, 2,000,000 distinct keys peaked at about 491 MiB (515 MB) in a local test of the dedupe engine (development machine, not Apify; see CHANGELOG). In one Apify test, 76 rows of about 500 KB each peaked at about 174 MiB with 1,024 MB of memory; much wider rows may need 2,048 MB or more. After the first page of 50 rows, the Actor reads fewer rows per page when rows are large. If a run fails with an out-of-memory error, run it again with 2,048 MB; the start fee is per GB of memory (see Pricing). If a run runs out of memory, the rows it wrote are charged, except that the last batch (up to 500 rows) may be written without a charge. A new run starts over and writes and charges every row again.
- Text is compared as stored: no Unicode normalization (an accented letter typed two different ways counts as two values), and `caseInsensitive` uses simple lower-casing (`ß` and `SS` differ).
- Arrays inside key fields are order-sensitive (`[1,2]` and `[2,1]` differ).
- A field whose own name contains a dot (`"a.b"`) cannot be used as a key, because dots separate nested fields.
- This Actor can't take over a run from another Actor (metamorph). Such runs stop at the start, and this Actor reads, writes and charges no rows. Start this Actor directly instead.
- If a run is restarted by the platform, this Actor checks its saved progress, including the content of rows it already wrote, before continuing, and stops with a message saying why if anything does not line up. In a rare case, where the run's dataset already holds rows identical to this Actor's own output and the platform's row count is delayed for more than 15 seconds, this check cannot tell those rows apart from its own, and some rows could be skipped without an error. For runs where that matters, compare the output row count with the expected count.

### Support

Questions, bugs or feature requests: open an issue in the **Issues** tab on this Actor's page in Apify Store. Please do not paste personal or confidential row data into an issue.

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

# Changelog

This Actor's version history is a separate document: https://apify.com/luxe_tasks/dataset-dedupe/changelog.md

# Actor input Schema

## `datasetIds` (type: `array`):

Pick up to 20 datasets with the picker or paste their IDs. They are read in this order; with "Keep" = first, earlier datasets win. This Actor can read only the datasets you pick here (read-only).

## `items` (type: `array`):

Optional JSON array of objects (up to 10,000), read after the datasets. Example: \[{"url": "https://example.com/a", "name": "A"}]. Leave this and the datasets empty to process a built-in 6-row sample, which has no per-row charge; the per-run start fee still applies.

## `dedupeFields` (type: `array`):

Rows with equal values in ALL of these fields are duplicates. Dot paths reach nested fields (e.g. "seller.id"). Leave empty to compare whole rows.

## `keep` (type: `string`):

Which row of each duplicate group to keep: the first one read, or the last one (reads the sources twice).

## `trimWhitespace` (type: `boolean`):

" Lamp " and "Lamp" match. Output values are not changed.

## `caseInsensitive` (type: `boolean`):

"LAMP" and "lamp" match. Output values are not changed.

## `missingKeyPolicy` (type: `string`):

What to do with rows where every dedupe field is missing (or empty, see below): keep them all, drop them, or treat them as one group (keep one).

## `emptyCountsAsMissing` (type: `boolean`):

null and "" count as no value in the key fields.

## `pickFields` (type: `array`):

Top-level field names to keep in the output (input names). Leave empty to keep all.

## `omitFields` (type: `array`):

Top-level field names to remove (input names).

## `renameFields` (type: `object`):

Object of old name to new name, e.g. {"title": "name"}. A renamed field replaces an existing field with the new name.

## `castFields` (type: `object`):

Object of field name (after renaming) to type: string, number, integer or boolean, e.g. {"price": "number"}. Values that cannot be cast are kept unchanged and counted in RUN_SUMMARY.

## `escapeFormulas` (type: `boolean`):

Prefix a single quote (') to text values that start with = + - @, tab or carriage return, so CSV/Excel exports do not run them as formulas. Turn off to keep values byte-for-byte.

## `addSourceFields` (type: `boolean`):

Add \_sourceType, \_sourceDatasetId and \_sourceIndex to every output row.

## `maxOutputRows` (type: `integer`):

Stop after this many unique rows (0 or empty = no limit). Reading stops at 2,000,000 rows per run in any case.

## Actor input object example

```json
{
  "keep": "first",
  "trimWhitespace": true,
  "caseInsensitive": false,
  "missingKeyPolicy": "keep",
  "emptyCountsAsMissing": true,
  "escapeFormulas": true,
  "addSourceFields": false
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset: one row per kept input row, in input order (first or last occurrence). Export it as CSV, Excel or JSON from the Storage tab or the API.

## `runSummary` (type: `string`):

RUN_SUMMARY record: rows read per source, duplicates removed, rows written, rows with no key value, cast failures, formula-escaped values and charged events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("luxe_tasks/dataset-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("luxe_tasks/dataset-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call luxe_tasks/dataset-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,luxe_tasks/dataset-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8r09lroVlnerc3MZS/builds/YYAgPaWfjgpph1IIS/openapi.json