# Dedupe and Merge Datasets: Only New Items (`fernandosord/dataset-merge-dedupe`) Actor

Merge Apify datasets, runs and JSON or CSV files, remove duplicates by key fields with URL, email, phone and fuzzy matching, and output only items not seen in earlier runs. Runs after any scraper as an integration.

- **URL**: https://apify.com/fernandosord/dataset-merge-dedupe.md
- **Developed by:** [Plainsource](https://apify.com/fernandosord) (community)
- **Categories:** Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.02 / 1,000 item processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What this Actor does

Dataset Merge, Dedupe and Only New Items takes one or more Apify datasets (or JSON and CSV files), merges them into one list and removes duplicate records. You choose the fields that identify a record, such as a URL, an email, a phone number or a company name. Values are cleaned before they are compared, so `https://www.shop.com/item/?utm_source=ad` and `shop.com/item` count as the same page, and, with US as the default phone country, `(415) 555-2671` and `+1 415 555 2671` count as the same phone.

It can also remember what it has already returned. Put it after a scheduled scraper, turn on **Only output items not seen in earlier runs**, and each run outputs only the records that are new since the last run, plus, if you want, the ones whose details changed.

Typical uses:

- Combine the results of several scraper runs or tasks into one clean list
- Remove repeated leads, listings or products before sending them to a CRM, Google Sheets or an email tool
- Get only the new job posts, listings or products from a daily or hourly scrape
- Drop records you already have by comparing against an older dataset
- Clean up a JSON or CSV export from another tool

### How duplicates are matched

Records are duplicates when all **Dedupe key fields** (`dedupeFields`) match after cleaning. Leave the list empty to remove only records that are identical in every field.

| Key type | What is ignored when comparing |
|---|---|
| Text (default) | Spaces at the ends, repeated spaces, and upper or lower case (unless **Case-sensitive keys** is on) |
| **Treat as URLs** (`urlFields`) | `http`/`https`, `www.`, a trailing slash, `#fragment`, default ports, `index.html`, and tracking parameters such as `utm_*`, `gclid`, `fbclid`, `msclkid`, `mc_cid`, `_ga`. Other query parameters are kept and sorted. Paths stay case-sensitive. |
| **Treat as emails** (`emailFields`) | Case, a `mailto:` prefix, and a display name such as `Jane <jane@x.com>` |
| **Treat as phone numbers** (`phoneFields`) | Formatting. Numbers are converted to international format with libphonenumber, using **Default phone country** (`defaultPhoneCountry`) for numbers without a `+` code. Numbers it can't parse are compared by their digits. |
| **Fuzzy match field** (`fuzzyField`) | Case, accents, punctuation and word order, and small spelling differences up to **Fuzzy similarity threshold** (`fuzzyThreshold`, default 90%) |

Fuzzy similarity is 100 minus the edit distance as a percentage of the longer text. "Acme Corporation" and "Acme Corporaton" are 94% similar. To keep runs fast, a record is only compared with earlier records that share at least one word with it and have the same values in the other key fields. Fuzzy matching suits names and titles. It is a poor fit for codes that differ by one digit, such as "Product 101" and "Product 102", which are 91% similar.

**Which record to keep** (`keep`) decides what happens to each group of duplicates:

- `first`: the first record. Fastest, and the output keeps the input order.
- `last`: the latest record.
- `most-complete`: the record with the most filled-in fields. Ties go to the earlier one.
- `merge`: the first record, with its empty fields filled from the later duplicates.

Records where a key field is empty are handled by **Records with an empty key field** (`missingKeyAction`): `keep` outputs each of them, `drop` leaves them out, and `compare-as-empty` lets records with the same empty field match each other.

### Only new items, for scheduled scrapers

With **Only output items not seen in earlier runs** (`onlyNewItems`) on, the Actor saves a short hash of each output record's key in a key-value store named `plainsource-dataset-merge-dedupe-state` in your account. On the next run, records with a known key are left out.

- **State key** (`stateKey`) keeps separate lists apart. Use a different one for each scraper or task.
- **Also output changed items** (`outputChanged`) returns records seen before whose other fields changed since they were last output, for example a new price. Add fields that change on every scrape, such as `scrapedAt`, to **Fields to ignore when checking for changes** (`changeIgnoreFields`).
- **Add a \_status field** (`addStatusField`) marks each output record as `new` or `changed`.
- **Remember items from this run** (`rememberItems`) off lets you test settings without updating the list. **Forget remembered items first** (`forgetState`) starts the list over.
- If you change the key fields or their cleaning settings, the saved list no longer fits, so the next run treats every item as new and starts a fresh list.

Only records that were actually written are remembered. If a run stops at your maximum cost, the records it didn't read or write come back on the next run.

### Run it after another Actor

To dedupe a scraper's results every time it finishes:

1. Open the scraper (or its task) in Apify Console and go to the **Integrations** tab.
2. Choose **Connect Actor or task**, pick this Actor, and the event **Run succeeded**.
3. Set the input, keeping `datasetIds` as shown:

```json
{
    "datasetIds": ["{{resource.defaultDatasetId}}"],
    "dedupeFields": ["url"],
    "urlFields": ["url"],
    "onlyNewItems": true,
    "stateKey": "my-scraper"
}
```

Each run then reads the scraper's dataset and writes only the new items to its own default dataset. Add an integration on this Actor in turn (for example Google Sheets or a webhook) to send them on. To collect every run's new items in one place instead, add `"outputDataset": "my-scraper-new-items"`: they are then written to that named dataset, which grows with each run, and the run's default dataset stays empty.

Integrations and webhooks also pass a `payload` object. The Actor reads `payload.resource.defaultDatasetId` from it and adds that dataset to the input, so a plain webhook-style input works too. Prefer `datasetIds`: Apify runs this Actor with limited permissions, and depending on your account's storage access settings, it may only be allowed to read datasets passed in the **Datasets** field. If a dataset from `payload` is refused, the run fails with a message saying so; put the ID in `datasetIds` instead.

From code, call it after your scraper finishes:

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const scrape = await client.actor('<your-scraper>').call({ /* scraper input */ });
const dedupe = await client.actor('fernandosord/dataset-merge-dedupe').call({
    datasetIds: [scrape.defaultDatasetId],
    dedupeFields: ['url'],
    urlFields: ['url'],
    onlyNewItems: true,
    stateKey: 'my-scraper',
});
const { items } = await client.dataset(dedupe.defaultDatasetId).listItems();
```

### Input

Give at least one of **Datasets**, **Run IDs**, **JSON or CSV URLs** or **Inline JSON data**. Everything else is optional.

| Field | What it does |
|---|---|
| **Datasets** (`datasetIds`) | Apify datasets to merge, read in the order given. Pick them or paste IDs. |
| **Run IDs** (`runIds`) | Runs whose default dataset should be read. |
| **JSON or CSV URLs** (`dataUrls`) | Public files: a JSON array, JSON Lines, or CSV with a header row (comma, semicolon, tab or pipe). Up to 50 MB each and about 150 MB in total at the default 1 GB memory. |
| **Inline JSON data** (`rawData`) | Up to 10,000 items as a JSON array. |
| **Compare against datasets** (`compareDatasetIds`) | Records in these datasets are never output, and records that match them by key are removed. |
| **Dedupe key fields** (`dedupeFields`) | Fields that identify a record. Dot paths such as `address.city` work. |
| **Treat as URLs / emails / phone numbers** (`urlFields`, `emailFields`, `phoneFields`) | Key fields that get the extra cleaning described above. |
| **Default phone country** (`defaultPhoneCountry`) | Two-letter code such as `US` or `GB`. |
| **Case-sensitive keys** (`caseSensitive`) | Off by default. |
| **Fuzzy match field** and **Fuzzy similarity threshold** (`fuzzyField`, `fuzzyThreshold`) | One text field compared by similarity. |
| **Which record to keep** (`keep`) | `first`, `last`, `most-complete` or `merge`. |
| **Records with an empty key field** (`missingKeyAction`) | `keep`, `drop` or `compare-as-empty`. |
| **Only output items not seen in earlier runs** (`onlyNewItems`) and the fields below it | See the section above. **Maximum remembered items** (`rememberMaxKeys`) caps the list; the keys not seen for the longest time are forgotten first. |
| **What to output** (`outputItems`) | `unique` (default), `duplicates` (the removed records) or `nothing` (counts only). |
| **Output dataset name** (`outputDataset`) | Write to a named dataset instead of the run's default dataset. The Actor creates it on the first run and adds to it on later runs. |
| **Output fields**, **Fields to remove**, **Rename fields** (`outputFields`, `omitFields`, `renameFields`) | Shape the written records. `renameFields` is an object such as `{"title": "name"}`. |
| **Write cleaned key values** (`writeNormalizedValues`) | Write the cleaned URL, email or phone instead of the original. |
| **Add a \_status field** / **Add a \_source field** (`addStatusField`, `addSourceField`) | Adds `_status` (`new`, `changed`, `unique` or `duplicate`) or `_source` (the dataset ID or URL a record came from). |
| **Filters** and **Records must pass** (`filters`, `filterMatch`) | Keep only records that pass, for example `[{"field": "price", "op": "less-than", "value": 50}]`. Ops: `equals`, `not-equals`, `contains`, `not-contains`, `starts-with`, `ends-with`, `greater-than`, `greater-or-equal`, `less-than`, `less-or-equal`, `exists`, `not-exists`, `matches-regex`. |
| **Skip the first items** / **Maximum items to read** (`offset`, `limit`) | Counted across all inputs in order. |

### Output

The kept records go to the run's default dataset, unchanged apart from the output options you chose. This is the output of the example input (the form's prefilled values: seven company records, key `website` treated as a URL, keep `merge`), from a test run on Apify:

```json
[
    { "company": "Northwind Traders", "website": "https://www.northwind.example/?utm_source=directory", "email": "Sales@Northwind.example", "phone": "(415) 555-2671", "city": "San Francisco", "employees": 120 },
    { "company": "Contoso Ltd", "website": "https://contoso.example/about", "phone": "212-555-0187", "city": "New York", "email": "hello@contoso.example" },
    { "company": "Fabrikam", "website": "https://fabrikam.example", "email": "info@fabrikam.example", "city": "Austin" },
    { "company": "Tailspin Toys", "website": "https://tailspin.example/shop?gclid=abc123", "city": "Denver", "phone": "720 555 0144" }
]
```

The two Northwind records matched although one URL had `www.`, a tracking parameter and `https`, and the other had none of them. The merged record took `employees` from the second one. The two Contoso records matched on `/about` and `/about/#team`.

The run also saves a summary in the `OUTPUT` record of its key-value store:

```json
{
    "status": "succeeded",
    "inputItems": 7,
    "compareItems": 0,
    "filteredOut": 0,
    "missingKey": 0,
    "missingKeyDropped": 0,
    "duplicatesRemoved": 3,
    "fuzzyDuplicates": 0,
    "matchedCompareData": 0,
    "uniqueItems": 4,
    "newItems": null,
    "changedItems": null,
    "unchangedItems": null,
    "outputItems": 4,
    "stoppedAtCostLimit": false,
    "fuzzyIndexFull": false,
    "keep": "merge",
    "outputMode": "unique",
    "dedupeFields": ["website (url)"],
    "onlyNewItems": null,
    "outputDataset": { "id": "tSPt0JxjtRy1cft20", "name": null },
    "inputs": [{ "type": "rawData", "id": "rawData", "items": 7 }],
    "startedAt": "2026-10-26T02:53:05.367Z",
    "finishedAt": "2026-10-26T02:53:05.639Z",
    "durationSecs": 0.3
}
```

In only-new mode, `newItems`, `changedItems` (changed since last output) and `unchangedItems` (seen before) are numbers, and `onlyNewItems` shows the list. From a test run with a remembered list of 2,100 items, where one record was new and two had changed:

```json
"onlyNewItems": {
    "stateKey": "platform-test",
    "rememberedBefore": 2100,
    "lastUpdatedAt": "2026-10-26T02:51:10.805Z",
    "rememberedAfter": 2101,
    "forgotOldest": 0,
    "saved": true
}
```

`status` is `succeeded`, `stopped-at-cost-limit` (the run reached your maximum cost; see Pricing) or `failed` (with an `error` message, for example a dataset ID that doesn't exist or a URL that was refused).

### Pricing

You pay per event:

| Event | Price |
|---|---|
| Run start | $0.002 per run |
| Item processed | $0.00002 per input item read ($0.02 per 1,000) |
| Item output | $0.00001 per item written ($0.01 per 1,000) |

Every input item counts as processed, including duplicates, filtered-out items and items in **Compare against datasets**. Only written items count as output, so "only new" runs that find nothing new pay only for reading.

Examples:

- Dedupe 10,000 items down to 7,000: $0.002 + $0.20 + $0.07 = $0.272.
- A daily only-new run over 1,000 items that finds 50 new ones: about $0.0225 a day, or $0.68 a month.
- Dedupe 100,000 items down to 70,000: about $2.70.

If you set a maximum cost per run, the Actor reads only as many items as it can also afford to write, then stops and reports `stopped-at-cost-limit`. Nothing is charged beyond the limit.

### Limits

- Up to 100 datasets and runs, 20 URLs and 10,000 inline items per run. Downloaded files are held in memory, so each URL can be up to 50 MB and all of them together up to 15% of the run memory (about 150 MB at the default 1 GB). For more, give the run more memory or load the files into datasets first.
- Datasets are read 10,000 items at a time, so their size is limited by memory for the keys, not by the data. In a local benchmark of 1 million items (700,000 unique) with only-new mode on, memory peaked at 526 MB with **Which record to keep** set to `first`, 718 MB with `last` and 1.03 GB with `merge`, which holds partly merged records until each group's last duplicate is read. The default run memory is 1 GB. For `merge` or `last` on more than about 500,000 items, or only-new mode with more than 1 million remembered items, raise the run memory to 2 GB or more.
- `last`, `most-complete` and `merge` read the input twice. Inputs up to 50,000 items are kept in memory after the first read instead.
- Output order: `first` keeps the input order. The other modes write each record where its kept version appears (`merge`: where the group's last record appears).
- Fuzzy matching works within one run, holds up to 300,000 distinct texts, and compares each record with at most 2,000 earlier ones. Between runs, only-new mode compares exact cleaned keys.
- Remembered keys are 96-bit hashes, 20 bytes each in storage. Up to 10 million per state key.
- Dataset fields whose names start with `#` (hidden fields) are not read.
- Private and local network addresses (such as `localhost`, `10.x.x.x` and `169.254.169.254`, also when written as IPv6) are refused, including through redirects. Downloads time out after 120 seconds.
- **Run IDs:** Apify's limited permissions may block looking up a run by its ID. If that happens, the error says so; paste the run's dataset ID into **Datasets** instead.
- **Output dataset name:** use a new name. The Actor creates the dataset and can add to it on later runs, but limited permissions don't let it write to a dataset it didn't create.
- Custom JavaScript functions are not run.

### Moving from lukaskrivka/dedup-datasets

This Actor accepts the main input field names of `lukaskrivka/dedup-datasets` and `lukaskrivka/dedup-datasets-limited-permissions`, so an existing task or integration needs few changes. It is a separate project and is not affiliated with them.

| Their field | Here |
|---|---|
| `datasetIds`, `offset`, `limit` | Same meaning |
| `fields` | `dedupeFields` |
| `output` (`unique-items`, `duplicate-items`, `nothing`) | `outputItems` (`unique`, `duplicates`, `nothing`) |
| `outputDatasetId` | `outputDataset` (a name) |
| `fieldsToLoad` | `outputFields` |
| `datasetIdsOfFilterItems` | `compareDatasetIds` |
| `nullAsUnique` | `missingKeyAction`: `true` is `keep`, `false` is `compare-as-empty` |
| `appendDatasetIds` | Adds `__datasetId__` to each record, like `addSourceField` |
| `mode`, `parallelLoads`, `parallelPushes`, `uploadBatchSize`, `batchSizeLoad`, `verboseLog`, `customInputData` | Ignored; datasets are always read in pages |
| `preDedupTransformFunction`, `postDedupTransformFunction` | Not supported; a warning is logged. Use **Filters**, **Output fields** and **Rename fields**. |
| `outputTo: key-value-store` | Not supported; results go to a dataset |

### FAQ

**Why did two records that look the same not match?** Check that the field is in **Dedupe key fields** and has the right type (URL, email or phone). Text keys ignore case and extra spaces but not punctuation; use a fuzzy match field for names that differ in punctuation or spelling.

**Can it dedupe on several fields at once?** Yes, when all of them must match. "Same email or same phone" needs two runs: first with `email` as the key, then with `phone` as the key on the first run's output.

**How do I see what was removed?** Set **What to output** to **Duplicates**, or turn on **Add a \_source field** to see where each kept record came from.

**How do I start the only-new list over?** Turn on **Forget remembered items first** for one run, or use a new **State key**.

**Does it change my source datasets?** No. They are only read.

**Can I call it from an AI agent?** Yes, through the Apify API or the Apify MCP server, with the same input as above.

Found a bug or need an option? Open an issue on this Actor's Issues tab.

# Actor input Schema

## `datasetIds` (type: `array`):

Apify datasets to merge and dedupe, in the order given. Pick them, or paste dataset IDs. To run this Actor automatically after a scraper, add it as an integration and set this field to <code>\["{{resource.defaultDatasetId}}"]</code> (see the README).

## `runIds` (type: `array`):

Actor run IDs whose default dataset should be read. Apify's limited permissions may block looking up runs; if a run is refused, paste its dataset ID into "Datasets" instead.

## `dataUrls` (type: `array`):

Public http or https URLs of JSON arrays, JSON Lines or CSV files (comma, semicolon, tab or pipe separated, first row = header), up to 50 MB each and 15% of the run memory in total (about 150 MB at the default 1 GB). Private and local network addresses are refused.

## `rawData` (type: `array`):

Up to 10,000 items as a JSON array of objects. Useful for trying the Actor or for small lists.

## `compareDatasetIds` (type: `array`):

Optional. Records from these datasets are never output; records in "Datasets" that match one of them by key are removed. Use it to drop leads you already have, or items from an older export.

## `dedupeFields` (type: `array`):

Records whose values in all these fields match (after normalisation) are duplicates. Use dot paths for nested fields, for example <code>address.city</code>. Leave empty to remove only records that are identical in every field.

## `urlFields` (type: `array`):

Key fields that hold web addresses. They are compared without http/https, <code>www.</code>, trailing slash, <code>#fragment</code>, default port and tracking parameters (<code>utm\_\*</code>, <code>gclid</code>, <code>fbclid</code>, <code>msclkid</code> and similar). Other query parameters are kept and sorted.

## `emailFields` (type: `array`):

Key fields that hold email addresses. They are lowercased, and a <code>mailto:</code> prefix or a display name such as <code>Jane \<jane@x.com></code> is removed.

## `phoneFields` (type: `array`):

Key fields that hold phone numbers. They are converted to international format (for example <code>+14155552671</code>) with libphonenumber, so <code>(415) 555-2671</code> and <code>+1 415 555 2671</code> match.

## `defaultPhoneCountry` (type: `string`):

Two-letter country code (US, GB, DE, BR...) for phone numbers written without a + country code. Without it, such numbers are compared by their digits only.

## `caseSensitive` (type: `boolean`):

Off: "ACME" and "acme" match. On: they don't. Spaces at the ends and repeated spaces are ignored either way.

## `fuzzyField` (type: `string`):

Optional text field, such as a company or product name, compared by similarity instead of exact value. Case, accents, punctuation and word order are ignored. A record is compared with earlier records that share at least one word with it and have the same values in the other key fields. Leave empty for exact matching only.

## `fuzzyThreshold` (type: `integer`):

How similar two texts must be to count as the same, from 50 to 100. Similarity is 100 minus the edit distance as a percentage of the longer text: "Acme Corporation" and "Acme Corporaton" are 94% similar.

## `keep` (type: `string`):

<b>First</b> keeps the first record of each group and is the fastest. <b>Last</b> keeps the latest one. <b>Most complete</b> keeps the record with the most filled-in fields. <b>Merge</b> starts from the first record and fills its empty fields from the later duplicates. All but First read the input twice (inputs up to 50,000 items are kept in memory instead).

## `missingKeyAction` (type: `string`):

What to do with records where a key field is missing or empty. <b>Keep</b> outputs each of them (they never count as duplicates). <b>Drop</b> leaves them out. <b>Compare as empty</b> treats the empty value as a value, so two records missing the same field can match.

## `onlyNewItems` (type: `boolean`):

Remembers the key of every item this Actor outputs, under the state key below, and leaves out items with a key it has seen before. Made for scheduled scrapers: each run returns only what's new.

## `stateKey` (type: `string`):

Remembered items are kept per state key. Use a different key for each scraper or task, so separate lists don't mix. A new key starts empty.

## `outputChanged` (type: `boolean`):

Also output items seen before whose other fields changed since they were last output (for example a new price). Compared by a hash of every field except the key fields and the fields below.

## `changeIgnoreFields` (type: `array`):

Fields that change on every scrape and should not count as a change, such as <code>scrapedAt</code> or <code>position</code>.

## `rememberItems` (type: `boolean`):

Turn off to test settings: the run shows what is new but doesn't update the remembered items.

## `forgetState` (type: `boolean`):

Clears the items remembered under this state key before the run, so every item counts as new. Turn it off again after one run.

## `rememberMaxKeys` (type: `integer`):

When more keys than this are remembered, the ones not seen for the longest time are forgotten. Each remembered item takes 20 bytes of storage and roughly 200 bytes of memory during a run, so raise the run memory for more than 1 million.

## `outputItems` (type: `string`):

<b>Unique items</b>: one record per key. <b>Duplicates</b>: the records that were removed. <b>Nothing</b>: only the counts in the OUTPUT summary.

## `outputDataset` (type: `string`):

Optional. Write to a named dataset instead of the run's default dataset. It is created on the first run, and later runs add to it. Letters, digits and hyphens only.

## `outputFields` (type: `array`):

Optional. Only these fields are written (dot paths allowed). Leave empty to keep all fields. Choosing fields also makes datasets faster to read.

## `omitFields` (type: `array`):

Optional. Fields removed from every written record.

## `renameFields` (type: `object`):

Optional JSON object from old to new name, for example <code>{"title": "name", "address.city": "city"}</code>. Renaming happens after field selection.

## `writeNormalizedValues` (type: `boolean`):

Replace key field values in the output with their normalised form: URLs without tracking parameters, lowercased emails, phone numbers in +international format.

## `addStatusField` (type: `boolean`):

Adds <code>\_status</code> to each written record: <code>new</code> or <code>changed</code> in only-new mode, <code>unique</code> otherwise, <code>duplicate</code> when outputting duplicates.

## `addSourceField` (type: `boolean`):

Adds <code>\_source</code> with the dataset ID or URL each record came from.

## `filters` (type: `array`):

Optional. Only records that pass are deduped and output. JSON array such as <code>\[{"field": "price", "op": "less-than", "value": 50}, {"field": "email", "op": "exists"}]</code>. Ops: equals, not-equals, contains, not-contains, starts-with, ends-with, greater-than, greater-or-equal, less-than, less-or-equal, exists, not-exists, matches-regex. Text comparisons ignore case.

## `filterMatch` (type: `string`):

Whether a record must pass all filters or at least one.

## `offset` (type: `integer`):

Number of input items to skip, counted across all inputs in order.

## `limit` (type: `integer`):

Stop after reading this many input items. Leave empty for no limit.

## `payload` (type: `object`):

Filled in by Apify integrations and webhooks. Its <code>resource.defaultDatasetId</code> is added to "Datasets".

## Actor input object example

```json
{
  "rawData": [
    {
      "company": "Northwind Traders",
      "website": "https://www.northwind.example/?utm_source=directory",
      "email": "Sales@Northwind.example",
      "phone": "(415) 555-2671",
      "city": "San Francisco"
    },
    {
      "company": "Northwind Traders Inc.",
      "website": "http://northwind.example/",
      "email": "sales@northwind.example",
      "phone": "+1 415 555 2671",
      "employees": 120
    },
    {
      "company": "Contoso Ltd",
      "website": "https://contoso.example/about",
      "phone": "212-555-0187",
      "city": "New York"
    },
    {
      "company": "Contoso",
      "website": "https://www.contoso.example/about/#team",
      "email": "hello@contoso.example",
      "city": ""
    },
    {
      "company": "Fabrikam",
      "website": "https://fabrikam.example",
      "email": "info@fabrikam.example",
      "city": "Austin"
    },
    {
      "company": "Tailspin Toys",
      "website": "https://tailspin.example/shop?gclid=abc123",
      "city": "Denver"
    },
    {
      "company": "Tailspin Toys",
      "website": "https://tailspin.example/shop",
      "phone": "720 555 0144",
      "city": "Denver"
    }
  ],
  "dedupeFields": [
    "website"
  ],
  "urlFields": [
    "website"
  ],
  "caseSensitive": false,
  "fuzzyThreshold": 90,
  "keep": "merge",
  "missingKeyAction": "keep",
  "onlyNewItems": false,
  "stateKey": "default",
  "outputChanged": false,
  "rememberItems": true,
  "forgetState": false,
  "rememberMaxKeys": 1000000,
  "outputItems": "unique",
  "writeNormalizedValues": false,
  "addStatusField": false,
  "addSourceField": false,
  "filterMatch": "all",
  "offset": 0
}
```

# Actor output Schema

## `results` (type: `string`):

The kept records (or the removed duplicates, if chosen), in the run's default dataset. When "Output dataset name" is set, they are in that named dataset instead.

## `summary` (type: `string`):

Counts: items read, duplicates removed, new, changed and seen before, items written, and the inputs used.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "rawData": [
        {
            "company": "Northwind Traders",
            "website": "https://www.northwind.example/?utm_source=directory",
            "email": "Sales@Northwind.example",
            "phone": "(415) 555-2671",
            "city": "San Francisco"
        },
        {
            "company": "Northwind Traders Inc.",
            "website": "http://northwind.example/",
            "email": "sales@northwind.example",
            "phone": "+1 415 555 2671",
            "employees": 120
        },
        {
            "company": "Contoso Ltd",
            "website": "https://contoso.example/about",
            "phone": "212-555-0187",
            "city": "New York"
        },
        {
            "company": "Contoso",
            "website": "https://www.contoso.example/about/#team",
            "email": "hello@contoso.example",
            "city": ""
        },
        {
            "company": "Fabrikam",
            "website": "https://fabrikam.example",
            "email": "info@fabrikam.example",
            "city": "Austin"
        },
        {
            "company": "Tailspin Toys",
            "website": "https://tailspin.example/shop?gclid=abc123",
            "city": "Denver"
        },
        {
            "company": "Tailspin Toys",
            "website": "https://tailspin.example/shop",
            "phone": "720 555 0144",
            "city": "Denver"
        }
    ],
    "dedupeFields": [
        "website"
    ],
    "urlFields": [
        "website"
    ],
    "keep": "merge"
};

// Run the Actor and wait for it to finish
const run = await client.actor("fernandosord/dataset-merge-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "rawData": [
        {
            "company": "Northwind Traders",
            "website": "https://www.northwind.example/?utm_source=directory",
            "email": "Sales@Northwind.example",
            "phone": "(415) 555-2671",
            "city": "San Francisco",
        },
        {
            "company": "Northwind Traders Inc.",
            "website": "http://northwind.example/",
            "email": "sales@northwind.example",
            "phone": "+1 415 555 2671",
            "employees": 120,
        },
        {
            "company": "Contoso Ltd",
            "website": "https://contoso.example/about",
            "phone": "212-555-0187",
            "city": "New York",
        },
        {
            "company": "Contoso",
            "website": "https://www.contoso.example/about/#team",
            "email": "hello@contoso.example",
            "city": "",
        },
        {
            "company": "Fabrikam",
            "website": "https://fabrikam.example",
            "email": "info@fabrikam.example",
            "city": "Austin",
        },
        {
            "company": "Tailspin Toys",
            "website": "https://tailspin.example/shop?gclid=abc123",
            "city": "Denver",
        },
        {
            "company": "Tailspin Toys",
            "website": "https://tailspin.example/shop",
            "phone": "720 555 0144",
            "city": "Denver",
        },
    ],
    "dedupeFields": ["website"],
    "urlFields": ["website"],
    "keep": "merge",
}

# Run the Actor and wait for it to finish
run = client.actor("fernandosord/dataset-merge-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "rawData": [
    {
      "company": "Northwind Traders",
      "website": "https://www.northwind.example/?utm_source=directory",
      "email": "Sales@Northwind.example",
      "phone": "(415) 555-2671",
      "city": "San Francisco"
    },
    {
      "company": "Northwind Traders Inc.",
      "website": "http://northwind.example/",
      "email": "sales@northwind.example",
      "phone": "+1 415 555 2671",
      "employees": 120
    },
    {
      "company": "Contoso Ltd",
      "website": "https://contoso.example/about",
      "phone": "212-555-0187",
      "city": "New York"
    },
    {
      "company": "Contoso",
      "website": "https://www.contoso.example/about/#team",
      "email": "hello@contoso.example",
      "city": ""
    },
    {
      "company": "Fabrikam",
      "website": "https://fabrikam.example",
      "email": "info@fabrikam.example",
      "city": "Austin"
    },
    {
      "company": "Tailspin Toys",
      "website": "https://tailspin.example/shop?gclid=abc123",
      "city": "Denver"
    },
    {
      "company": "Tailspin Toys",
      "website": "https://tailspin.example/shop",
      "phone": "720 555 0144",
      "city": "Denver"
    }
  ],
  "dedupeFields": [
    "website"
  ],
  "urlFields": [
    "website"
  ],
  "keep": "merge"
}' |
apify call fernandosord/dataset-merge-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fernandosord/dataset-merge-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lfFTyNcrlpp4nwtlL/builds/uU2dIfQkmJ1cRG3fx/openapi.json