# 🧬 Entity Resolution - Merge Duplicate Records, No Shared ID (`that_red_bird/entity-resolver`) Actor

⚡ Find which messy records are the same real-world entity when there is no shared ID. ✅ Fuzzy name/company matching, blocking so 50k records don't need 1.25bn comparisons, probabilistic scoring, transitive clustering.

- **URL**: https://apify.com/that\_red\_bird/entity-resolver.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** Lead generation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Entity Resolution Engine

Work out which messy records are the **same real-world entity** when there is no shared ID —
then build one clean golden record per entity, with a full audit trail.

This is the problem behind every "our CRM is a mess" and "we merged two companies' customer
lists" project. Records arrive from different systems with different spellings, different
formatting and missing fields, and nothing joins them.

### What it actually does

**1. Blocking.** Comparing every pair is O(n²) — 50,000 records means 1.25 billion comparisons.
Records are grouped by cheap keys (email, email domain, phone suffix, company prefix, surname
Soundex, name initial) and only compared within a group. Multiple keys are used so one bad field
can't hide a true match. Oversized blocks (everyone sharing `gmail.com`) are skipped rather than
allowed to reintroduce the blowup. The run reports how many comparisons this avoided.

**2. Field-aware similarity.** Not one string distance for everything:

| Type | How it compares |
|---|---|
| `person` | surname-weighted; **initials handled** — "J. Adams" matches "Jennifer Adams"; nicknames expanded (Bob↔Robert, Jen↔Jennifer) |
| `company` | legal suffixes stripped repeatedly (`Acme Corp., Inc.` → `acme`), then token-set, trigram and Jaro-Winkler, best of |
| `email` | Gmail dots and `+tags` canonicalised, so `rob.ellis@` = `robellis+work@` |
| `phone` | digits only, country code handled |
| `address`, `text`, `numeric`, `exact` | order-independent tokens, trigrams, tolerance |

**3. Weighted scoring.** An email agreement is far stronger evidence than a first-name
agreement, and the weights encode that instead of averaging blindly. Missing fields are
**skipped, not scored zero** — absent data is not disagreement, and treating it as such hides
real matches in sparse data.

**4. Transitive clustering.** A~B and B~C means A, B and C are one entity, even if A and C were
never directly compared.

**5. A review band.** Pairs between `reviewThreshold` and `matchThreshold` are **reported, not
merged**, with the per-field evidence. Silently fusing two real people is far worse than missing
a duplicate.

**6. Survivorship.** Decides which value wins per field — most complete, longest, most common,
newest, first, or by source priority — and records **provenance** for every field so you can see
which input row each value came from.

### Input

Pass `records` inline and/or `sourceDatasetIds`. Leave `fields` empty and types are inferred
from column names; pass them explicitly for control:

```json
{ "fields": [
  { "field": "email",   "type": "email",   "weight": 5 },
  { "field": "name",    "type": "person",  "weight": 3 },
  { "field": "company", "type": "company", "weight": 2 }
], "matchThreshold": 85, "reviewThreshold": 70 }
```

Thresholds are 0–100 integers (Apify input schemas have no float type).

### Output

`entities` — every cluster with members, golden record, provenance and confidence ·
`golden` — one flat clean row per entity, ready to re-import · `duplicates` — only clusters that
actually merged · plus `review` rows for ambiguous pairs.

### Honest limitations

- Quality depends on having at least one **blockable** field (email, phone, name, company or
  postcode). Without one the run refuses rather than hanging on an O(n²) comparison.
- Nickname expansion covers common English given names; other languages fall back to string
  similarity.
- Thresholds are a genuine precision/recall trade-off. Raise `matchThreshold` for fewer false
  merges, lower it to catch more duplicates — and use the review band while you tune.
- Capped at 200,000 records per run.

# Actor input Schema

## `records` (type: `array`):

The records to resolve, as an array of flat objects. Combine freely with sourceDatasetIds.

## `sourceDatasetIds` (type: `array`):

Apify dataset IDs to pull records from. Rows are tagged with their dataset so sourcePriority survivorship can prefer one system over another.

## `fields` (type: `array`):

Leave empty to infer from the data. For control pass objects: {"field":"email","type":"email","weight":5}. Types: email, phone, person, company, address, numeric, text, exact. Higher weight = stronger evidence.

## `matchThreshold` (type: `integer`):

Pairs scoring at or above this are merged. Higher = fewer false merges, more missed duplicates. Expressed 0-100; 85 means 0.85.

## `reviewThreshold` (type: `integer`):

Pairs between this and the match threshold are reported for human review instead of being merged. Silently fusing two real entities is worse than missing a duplicate.

## `maxBlockSize` (type: `integer`):

Blocks larger than this are skipped to avoid an O(n²) blowup when many records share a generic key such as a common email domain.

## `survivorshipStrategy` (type: `string`):

Default rule for building the golden record from a cluster.

## `fieldStrategies` (type: `object`):

Override the rule for specific fields, e.g. {"email":"newest","company":"longest"}.

## `dateField` (type: `string`):

Field holding a record's last-updated date. Required by the "newest" strategy.

## `sourceField` (type: `string`):

Field naming the record's origin system. Defaults to the automatic \_source tag.

## `sourcePriority` (type: `array`):

Most trusted source first, e.g. \["crm","webform"]. Used by the sourcePriority strategy.

## `outputMode` (type: `string`):

entities = full clusters with members and provenance · golden = one clean row per entity · duplicates = only clusters that actually merged.

## `includeReviewPairs` (type: `boolean`):

Emit ambiguous pairs with their evidence so a human can decide.

## `includeScoredPairs` (type: `boolean`):

Emit all compared pairs with per-field similarity. Useful for tuning thresholds; verbose on large inputs.

## Actor input object example

```json
{
  "records": [
    {
      "name": "Jennifer Adams",
      "company": "Acme Corporation",
      "email": "jen@acme.com",
      "phone": "415-555-0100"
    },
    {
      "name": "Jen Adams",
      "company": "Acme Corp.",
      "email": "jen@acme.com",
      "phone": ""
    },
    {
      "name": "J. Adams",
      "company": "ACME, Inc.",
      "email": "",
      "phone": "(415) 555 0100"
    },
    {
      "name": "Michael Chen",
      "company": "Globex Ltd",
      "email": "mchen@globex.io",
      "phone": "212-555-0199"
    },
    {
      "name": "Mike Chen",
      "company": "Globex Limited",
      "email": "mchen@globex.io",
      "phone": ""
    }
  ],
  "matchThreshold": 85,
  "reviewThreshold": 70,
  "maxBlockSize": 500,
  "survivorshipStrategy": "mostComplete",
  "outputMode": "entities",
  "includeReviewPairs": true,
  "includeScoredPairs": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "records": [
        {
            "name": "Jennifer Adams",
            "company": "Acme Corporation",
            "email": "jen@acme.com",
            "phone": "415-555-0100"
        },
        {
            "name": "Jen Adams",
            "company": "Acme Corp.",
            "email": "jen@acme.com",
            "phone": ""
        },
        {
            "name": "J. Adams",
            "company": "ACME, Inc.",
            "email": "",
            "phone": "(415) 555 0100"
        },
        {
            "name": "Michael Chen",
            "company": "Globex Ltd",
            "email": "mchen@globex.io",
            "phone": "212-555-0199"
        },
        {
            "name": "Mike Chen",
            "company": "Globex Limited",
            "email": "mchen@globex.io",
            "phone": ""
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/entity-resolver").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "records": [
        {
            "name": "Jennifer Adams",
            "company": "Acme Corporation",
            "email": "jen@acme.com",
            "phone": "415-555-0100",
        },
        {
            "name": "Jen Adams",
            "company": "Acme Corp.",
            "email": "jen@acme.com",
            "phone": "",
        },
        {
            "name": "J. Adams",
            "company": "ACME, Inc.",
            "email": "",
            "phone": "(415) 555 0100",
        },
        {
            "name": "Michael Chen",
            "company": "Globex Ltd",
            "email": "mchen@globex.io",
            "phone": "212-555-0199",
        },
        {
            "name": "Mike Chen",
            "company": "Globex Limited",
            "email": "mchen@globex.io",
            "phone": "",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/entity-resolver").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "records": [
    {
      "name": "Jennifer Adams",
      "company": "Acme Corporation",
      "email": "jen@acme.com",
      "phone": "415-555-0100"
    },
    {
      "name": "Jen Adams",
      "company": "Acme Corp.",
      "email": "jen@acme.com",
      "phone": ""
    },
    {
      "name": "J. Adams",
      "company": "ACME, Inc.",
      "email": "",
      "phone": "(415) 555 0100"
    },
    {
      "name": "Michael Chen",
      "company": "Globex Ltd",
      "email": "mchen@globex.io",
      "phone": "212-555-0199"
    },
    {
      "name": "Mike Chen",
      "company": "Globex Limited",
      "email": "mchen@globex.io",
      "phone": ""
    }
  ]
}' |
apify call that_red_bird/entity-resolver --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/entity-resolver"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/QGcvgwdoRq1FiV9MS/builds/Wb9NtpTAARtIUB16F/openapi.json