# Dataset Dedupe Merger (`inestimable_zoysia/dataset-dedupe-merger`) Actor

Merge multiple datasets into one clean dataset. Removes duplicates with exact, normalized, or fuzzy matching, unifies field names, and keeps the most complete version of every record — with missing fields back-filled from discarded duplicates. Built for cleaning multi-scraper output.

- **URL**: https://apify.com/inestimable\_zoysia/dataset-dedupe-merger.md
- **Developed by:** [Uncle Glooby](https://apify.com/inestimable_zoysia) (community)
- **Categories:** AI, Developer tools, Lead generation
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Dataset Deduplicator & Merger

Merge multiple datasets into one clean dataset — remove duplicates, unify field names, and keep the most complete version of every record.

Built for the most common post-scraping problem: you ran Google Maps, Yelp, and a directory scraper, and now you have three overlapping lists with different column names and thousands of duplicate businesses. This Actor turns them into a single deduplicated, backfilled dataset in one run.

### What it does

- **Merges any number of datasets** — pass a list of datasets, get one combined output.
- **Three matching modes:**
  - `exact` — byte-for-byte comparison.
  - `normalized` (recommended) — case-insensitive, ignores punctuation and accents, strips common company suffixes (LLC, Inc, Ltd...), and compares phone numbers digits-only, so `+1 (555) 123-4567` matches `5551234567`.
  - `fuzzy` — catches near-duplicates like `Joe's Pizza` vs `Joes Pizza LLC` using a similarity threshold you control.
- **Smart "most complete" keep strategy** — instead of blindly keeping the first duplicate, keeps the record with the most filled-in fields **and back-fills its empty fields from the discarded duplicates**. If one source had the email and the other had the phone number, the surviving record gets both.
- **Field mapping** — rename columns on the fly so `businessName` and `company_name` become one `name` column before merging.
- **Handles big datasets** — streams input in batches; fuzzy mode supports *blocking* (e.g. only compare rows within the same zip code) so 100k-row jobs stay fast.

### Input example

```json
{
    "datasetIds": ["abc123", "def456", "ghi789"],
    "dedupeFields": ["name", "address"],
    "matchMode": "fuzzy",
    "fuzzyThreshold": 85,
    "blockingField": "zipCode",
    "fieldMapping": { "businessName": "name", "company_name": "name" },
    "keepStrategy": "mostComplete"
}
```

### Output

One clean dataset, plus a `SUMMARY` record in the key-value store:

```json
{
    "datasetsMerged": 3,
    "rowsIn": 48210,
    "duplicatesRemoved": 9384,
    "rowsOut": 38826,
    "rowsWithoutDedupeKey": 112,
    "matchMode": "fuzzy",
    "keepStrategy": "mostComplete"
}
```

### Common use cases

- **Lead lists** — dedupe by `email` or `phone` across scraped sources before importing to your CRM.
- **Local business data** — merge Google Maps + Yelp + directory results; fuzzy match on `name` + `address`, block by `zipCode`.
- **E-commerce** — dedupe product listings by normalized title or SKU.
- **Job postings** — collapse the same posting syndicated across multiple boards.

### Tips

- Start with `normalized` mode. Switch to `fuzzy` only if you can see near-duplicates surviving.
- In fuzzy mode on large datasets, **always set `blockingField`** (zip, city, domain...). It dramatically speeds up the run without hurting accuracy.
- Dot notation works everywhere fields are referenced: `contact.email`.
- Rows where all dedupe fields are empty are never merged with each other — they pass through untouched and are counted in `rowsWithoutDedupeKey`.

### Pricing

Pay-per-result: billed per row in your **cleaned output** dataset. Duplicates removed along the way are processed free — you only pay for the clean rows you keep. Cleaning a 50,000-row lead list down to 38,000 unique records costs about $19 — a rounding error compared to the value of the list.

# Actor input Schema

## `datasetIds` (type: `array`):

One or more datasets to merge and deduplicate. Order matters for the 'first'/'last' keep strategies.

## `dedupeFields` (type: `array`):

Field name(s) that define a duplicate. Rows matching on ALL of these fields are considered duplicates. Examples: \['email'] or \['name', 'address']. Supports dot notation for nested fields (e.g. 'contact.email').

## `matchMode` (type: `string`):

How field values are compared. 'exact' = byte-for-byte. 'normalized' = case-insensitive, whitespace/punctuation stripped, phone numbers reduced to digits. 'fuzzy' = normalized + similarity matching for near-duplicates like "Joe's Pizza" vs "Joes Pizza LLC".

## `fuzzyThreshold` (type: `integer`):

Only used in fuzzy mode. Similarity percentage (50–100) required to consider two values a match. 85 is a good default: catches typos and suffix noise without merging genuinely different records.

## `blockingField` (type: `string`):

Optional but strongly recommended for fuzzy mode on 10k+ rows. Rows are only fuzzy-compared when they share the same normalized value in this field (e.g. 'zipCode' or 'city'). Keeps large jobs fast.

## `fieldMapping` (type: `object`):

Optional map of source field → target field, applied to every row before merging. Example: {"businessName": "name", "company\_name": "name"} unifies differently-named columns across datasets.

## `keepStrategy` (type: `string`):

Which record survives when duplicates are found. 'mostComplete' keeps the row with the fewest empty fields and back-fills its missing values from the discarded duplicates (best for lead lists).

## `outputFields` (type: `array`):

Optional. If set, output rows contain only these fields, in this order.

## Actor input object example

```json
{
  "dedupeFields": [
    "email"
  ],
  "matchMode": "normalized",
  "fuzzyThreshold": 85,
  "keepStrategy": "mostComplete"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupeFields": [
        "email"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("inestimable_zoysia/dataset-dedupe-merger").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "dedupeFields": ["email"] }

# Run the Actor and wait for it to finish
run = client.actor("inestimable_zoysia/dataset-dedupe-merger").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupeFields": [
    "email"
  ]
}' |
apify call inestimable_zoysia/dataset-dedupe-merger --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,inestimable_zoysia/dataset-dedupe-merger"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/w1168yVc62enB9FgA/builds/tTJZJPCYVv4DLDr9z/openapi.json