Entity Resolution Engine

Work out which messy records are the same real-world entity when there is no shared ID — then build one clean golden record per entity, with a full audit trail.

This is the problem behind every "our CRM is a mess" and "we merged two companies' customer lists" project. Records arrive from different systems with different spellings, different formatting and missing fields, and nothing joins them.

What it actually does

1. Blocking. Comparing every pair is O(n²) — 50,000 records means 1.25 billion comparisons. Records are grouped by cheap keys (email, email domain, phone suffix, company prefix, surname Soundex, name initial) and only compared within a group. Multiple keys are used so one bad field can't hide a true match. Oversized blocks (everyone sharing gmail.com) are skipped rather than allowed to reintroduce the blowup. The run reports how many comparisons this avoided.

2. Field-aware similarity. Not one string distance for everything:

Type	How it compares
person	surname-weighted; initials handled — "J. Adams" matches "Jennifer Adams"; nicknames expanded (Bob↔Robert, Jen↔Jennifer)
company	legal suffixes stripped repeatedly (Acme Corp., Inc. → acme), then token-set, trigram and Jaro-Winkler, best of
email	Gmail dots and +tags canonicalised, so rob.ellis@ = robellis+work@
phone	digits only, country code handled
address, text, numeric, exact	order-independent tokens, trigrams, tolerance

3. Weighted scoring. An email agreement is far stronger evidence than a first-name agreement, and the weights encode that instead of averaging blindly. Missing fields are skipped, not scored zero — absent data is not disagreement, and treating it as such hides real matches in sparse data.

4. Transitive clustering. AB and BC means A, B and C are one entity, even if A and C were never directly compared.

5. A review band. Pairs between reviewThreshold and matchThreshold are reported, not merged, with the per-field evidence. Silently fusing two real people is far worse than missing a duplicate.

6. Survivorship. Decides which value wins per field — most complete, longest, most common, newest, first, or by source priority — and records provenance for every field so you can see which input row each value came from.

Input

Pass records inline and/or sourceDatasetIds. Leave fields empty and types are inferred from column names; pass them explicitly for control:

{ "fields": [
  { "field": "email",   "type": "email",   "weight": 5 },
  { "field": "name",    "type": "person",  "weight": 3 },
  { "field": "company", "type": "company", "weight": 2 }
], "matchThreshold": 85, "reviewThreshold": 70 }

Thresholds are 0–100 integers (Apify input schemas have no float type).

Output

entities — every cluster with members, golden record, provenance and confidence · golden — one flat clean row per entity, ready to re-import · duplicates — only clusters that actually merged · plus review rows for ambiguous pairs.

Honest limitations

• Quality depends on having at least one blockable field (email, phone, name, company or postcode). Without one the run refuses rather than hanging on an O(n²) comparison.
• Nickname expansion covers common English given names; other languages fall back to string similarity.
• Thresholds are a genuine precision/recall trade-off. Raise matchThreshold for fewer false merges, lower it to catch more duplicates — and use the review band while you tune.
• Capped at 200,000 records per run.