# Fuzzy Dedupe: Find Near-Duplicate Rows in Datasets (`ucchiii/dataset-fuzzy-dedupe`) Actor

Find near-duplicates that exact dedup misses: 'Acme Inc.' vs 'ACME, Inc', 'Main St.' vs 'Main Street'. Fuzzy match names, addresses, phones, URLs across Apify datasets; mark or merge clusters with reasons. Empty input = free demo. $0.50 per 1,000 rows.

- **URL**: https://apify.com/ucchiii/dataset-fuzzy-dedupe.md
- **Developed by:** [Actor Factory](https://apify.com/ucchiii) (community)
- **Categories:** Developer tools, Automation, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 row processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Fuzzy Dedupe — find near-duplicate rows that exact dedup misses

**For anyone with scraped or exported business lists** (leads, places, products, suppliers) where the same company
appears as "Acme Inc." and "ACME, Inc", or "12 Main St." and "12 Main Street". Exact dedup keeps both. This Actor
**finds the near-duplicates across one or more Apify datasets, groups them into clusters, and either marks every row
or merges each cluster into its most complete row**, with a score and a plain-text reason for every decision.

**Try it free: leave input empty and press Start** (sample rows, rows carry `_demo: true`, nothing is charged).

### Sample output (mark mode)

| company | clusterId | isPrimary | matchScore | matchReason |
|---|---|---|---|---|
| Acme Inc. | c1 | true | 1.0 | primary: most complete row of 2 near-duplicates |
| ACME, Inc | c1 | false | 1.0 | confirmed (4 fields agree, score 1.00): company exact; address exact; phone exact; website exact |
| Northwind Traders LLC | c3 | true | 1.0 | unique |
| Delta Dental Clinic | c10 | true | 0.96 | possible duplicate of demo:10 (best score 0.96) but not safe to auto-merge |

**Pricing:** $0.50 per 1,000 input rows processed ($0.0005 each). Example: 1,000 rows cost $0.50, 20,000 rows cost $10.
Merge mode charges for the input rows too (every row that went into a merged row), not only the rows written.
Failures, errors and the demo are free. Set a max charge per run to cap spend.

### What it does

- **Normalization**: case, punctuation, accents, legal suffixes (Inc, LLC, Ltd, Corp, GmbH, AG, S.A., 株式会社, 有限会社 ...), street abbreviations (St/Street, Ave, Rd, Hauptstr./Hauptstrasse, N/North ...), ordinal numbers, full-width/half-width characters, Japanese address units (1丁目2番3号 = 1-2-3), phone digits (country code and formatting ignored), URL host (www, scheme and path ignored).
- **Similarity** (via [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) 3.14.3): for names and addresses the average of Jaro-Winkler and token-sort ratio; for free text token-sort ratio; phones and URL hosts are exact. Rows whose numbers differ ("Store 12" vs "Store 21", "12 Main St" vs "21 Main St") are penalized.
- **Several fields, weighted**: each field has a `type` (`name`, `address`, `phone`, `url`, `text`) and a `weight`. The row score is the weighted average over the fields both rows have.
- **Blocking** to stay fast: only rows with the same key (first letters, postal code, or website domain) are compared. Huge blocks are compared with a sliding window.
- **Two modes**: `mark` keeps every row and adds `clusterId`, `isPrimary`, `matchScore`, `matchReason` (plus `matchStatus`, `clusterSize`, `rowId`, `primaryRowId`, `possibleDuplicateOf`). `merge` writes one row per cluster: the most complete row plus `mergedRowIds`.
- **Safety rule**: rows are merged into a cluster only when the weighted score is at least the threshold **and** either at least two independent fields agree, or a strong field (phone, URL host, or the only field you configured) agrees. Everything else that looks similar is only reported as `possible` (`possibleDuplicateOf`) and is never merged.
- **Memory-conscious**: rows are streamed; only compact normalized values are kept in memory, not the full rows. Up to **200,000 rows per run** (more are ignored and not charged) and 20 datasets.

### Input example

```json
{
  "datasetIds": ["aBcDeF123"],
  "matchFields": [
    {"field": "name", "type": "name", "weight": 2},
    {"field": "address", "type": "address", "weight": 1},
    {"field": "phone", "type": "phone", "weight": 1}
  ],
  "threshold": 0.9,
  "mode": "mark",
  "blockingKey": "firstLetters"
}
```

Run statistics (rows read, clusters, duplicates, possible matches, rows charged) are saved in the key-value store as `STATS`.

### Sample `matchReason` values

- `confirmed (4 fields agree, score 1.00): company exact ('acme' ~ 'acme'); address exact ('12 main street springfield 62701' ~ '12 main street springfield 62701'); phone exact; website exact`
- `confirmed (strong field agrees, score 0.94): name 0.88 ('globex corporation' ~ 'globex corp'); phone exact`
- `possible duplicate of demo:10 (best score 0.96) but not safe to auto-merge: fewer than two independent fields agree and no strong field (phone, url) matches`
- `linked through other similar rows; directly vs primary: ...` (A matches B and B matches C, but A and C only through B: check these)

### Disclaimer

**Fuzzy matching can be wrong.** Different companies can have similar names, and one company can legitimately have several
addresses or phone numbers. Treat results as suggestions: start with `mark` mode, review `matchReason`,
`possibleDuplicateOf` and low `matchScore` rows, and keep your source datasets. Your source datasets are never
changed, but the merged output only keeps the most complete row of each cluster.

### Tips

- Use a higher threshold (0.93-0.97) for names alone, lower (0.85) when phone or website also match.
- Blocking hides matches whose key differs (a typo in the first letters, a different postal code). For maximum recall on small data use `blockingKey: none` (comfortable up to roughly 20,000 rows; above that it falls back to a sliding window).
- Rows with an empty blocking value are only compared with each other.

### More tools from the same developer

- [Dataset Merge & Dedup](https://apify.com/ucchiii/dataset-dedup-merge) — merge Apify datasets and remove duplicates by the fields you choose
- [Google Maps Leads Cleaner](https://apify.com/ucchiii/maps-leads-cleaner) — dedupe Google Maps scraper results, format phones (E.164), CRM-ready leads
- [CSV / XLSX Converter](https://apify.com/ucchiii/csv-xlsx-converter) — convert CSV, TSV, Excel XLSX, JSON and NDJSON files or Apify datasets into each other
- [Dataset Transform](https://apify.com/ucchiii/dataset-transform) — flatten, filter, rename and convert any Apify dataset, export CSV / JSON / JSONL with a link
- [Japan Address Normalizer](https://apify.com/ucchiii/japan-address-normalizer) — split Japanese addresses into prefecture, city, town and rest, with coordinates

# Actor input Schema

## `datasetIds` (type: `array`):

Apify datasets to compare (the 'defaultDatasetId' of a finished run, e.g. 'WkzbQMuFYuamGv3YF'). Up to 20; rows from all datasets are compared with each other. You can paste a dataset URL from Apify Console too. Leave empty to run a free demo on sample data (not charged).

## `matchFields` (type: `array`):

List of objects: field (column name, dots for nested like 'address.city'), type (name, address, phone, url or text) and weight (default 1, higher = counts more). name: ignores case, punctuation and legal suffixes (Inc, LLC, Ltd, GmbH, 株式会社). address: street abbreviations and full-width characters. phone: digits only. url: host only. text: plain fuzzy text. If empty, a field called 'name' is compared.

## `threshold` (type: `number`):

Minimum weighted similarity for two rows to be treated as the same. Default 0.9. Lower finds more (and riskier) matches; 1 means identical after normalization.

## `mode` (type: `string`):

'mark' keeps every row and adds clusterId, isPrimary, matchScore and matchReason. 'merge' outputs one row per cluster: the most complete row plus mergedRowIds.

## `blockingKey` (type: `string`):

Only rows sharing the key are compared, which keeps big datasets fast. Rows with different keys are never matched, so pick a key that duplicates share (typos in the first letters can hide a match). Default: first letters.

## `blockingField` (type: `string`):

Column the blocking key is taken from (e.g. 'name', 'address', 'zip', 'website'). Leave empty to use the first match field of a suitable type.

## `blockingLength` (type: `integer`):

For the 'first letters' key: how many characters of the normalized value to use (1 to 10). Shorter = safer but slower.

## `maxRows` (type: `integer`):

Hard cap on rows processed per run (at most 200,000). Rows beyond the cap are ignored and not charged. Also useful as a cost limit.

## Actor input object example

```json
{
  "matchFields": [
    {
      "field": "name",
      "type": "name",
      "weight": 2
    },
    {
      "field": "address",
      "type": "address",
      "weight": 1
    },
    {
      "field": "phone",
      "type": "phone",
      "weight": 1
    }
  ],
  "threshold": 0.9,
  "mode": "mark",
  "blockingKey": "firstLetters",
  "blockingField": "name",
  "blockingLength": 3,
  "maxRows": 200000
}
```

# Actor output Schema

## `results` (type: `string`):

Mark mode: every input row with clusterId, isPrimary, matchScore and matchReason. Merge mode: one row per cluster with mergedRowIds.

## `stats` (type: `string`):

Record STATS: rows read, clusters, duplicate rows, rows with only possible matches, rows charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "matchFields": [
        {
            "field": "name",
            "type": "name",
            "weight": 2
        },
        {
            "field": "address",
            "type": "address",
            "weight": 1
        },
        {
            "field": "phone",
            "type": "phone",
            "weight": 1
        }
    ],
    "blockingField": "name"
};

// Run the Actor and wait for it to finish
const run = await client.actor("ucchiii/dataset-fuzzy-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "matchFields": [
        {
            "field": "name",
            "type": "name",
            "weight": 2,
        },
        {
            "field": "address",
            "type": "address",
            "weight": 1,
        },
        {
            "field": "phone",
            "type": "phone",
            "weight": 1,
        },
    ],
    "blockingField": "name",
}

# Run the Actor and wait for it to finish
run = client.actor("ucchiii/dataset-fuzzy-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "matchFields": [
    {
      "field": "name",
      "type": "name",
      "weight": 2
    },
    {
      "field": "address",
      "type": "address",
      "weight": 1
    },
    {
      "field": "phone",
      "type": "phone",
      "weight": 1
    }
  ],
  "blockingField": "name"
}' |
apify call ucchiii/dataset-fuzzy-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ucchiii/dataset-fuzzy-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1DUjKPikdxuujOgP7/builds/erLJcfJrnQp1faVbE/openapi.json