Entity Deduplication Matcher

Fuzzy-match company, product, location, or other entity rows and turn duplicate variants into traceable canonical record clusters.

What does Entity Deduplication Matcher do?

Entity Deduplication Matcher compares every bounded record pair with deterministic, configurable field scoring. It normalizes company suffixes, domains, email addresses, phone numbers, punctuation, casing, and whitespace before assigning each record a canonical, merge, unique, review, or invalid decision.

• Combine exact signals such as normalized domains and phones with fuzzy name and address similarity.
• Set field weights, automatic merge thresholds, and lower review thresholds.
• Select the most complete record as the canonical record for each duplicate cluster.
• Preserve original records, normalized comparison fields, match reasons, and scores.
• Read inline JSON records, quoted CSV text, or bounded public JSON/CSV URLs.
• Export dataset decisions, canonical-record JSON, summary JSON, and a Markdown report.

Why Use This Actor

Entity matching is easy to start and difficult to audit. Exact spreadsheet keys miss punctuation and naming variants, while opaque matching services can make merge decisions hard to explain. This actor keeps each score and decision visible.

Capability	Entity Deduplication Matcher	OpenRefine clustering	Spreadsheet formulas	Dedupe.io
Weighted multi-field scoring	Configurable	Limited by method	Manual	Available
Exact plus fuzzy fields	Yes	Yes	Usually exact only	Yes
Canonical record selection	Included	Manual	Manual	Workflow-dependent
Record-level reasons and scores	Included	Method-dependent	Formula-dependent	Product-dependent
Apify dataset and KVS exports	Included	No	No	Separate integration
Bounded public URL ingestion	Included with SSRF guards	File import	File import	Product-dependent
Primary processing price	$3.90 per 1,000 ready records	Separate product	Maintenance time	Separate product

Use it before CRM imports, account merges, catalog consolidation, lead routing, location cleanup, product identity resolution, or any automated workflow that needs a reviewable duplicate decision.

How to Use

{
  "records": [
    { "id": "acct-1", "name": "Acme Roofing Nashville LLC", "domain": "acmeroofing.example", "phone": "(615) 555-0199" },
    { "id": "acct-2", "name": "ACME Roofing - Nashville", "domain": "www.acmeroofing.example", "phone": "6155550199" },
    { "id": "acct-3", "name": "Brio Plumbing Austin", "domain": "brioplumbing.example", "phone": "(512) 555-0100" }
  ],
  "matchFields": ["name", "domain", "phone"],
  "exactFields": ["domain", "phone"],
  "matchThreshold": 0.82,
  "reviewThreshold": 0.67,
  "maxItems": 250,
  "includeReport": true
}

1. Supply records directly, paste CSV text, or configure a public source URL.
2. Choose stable identity fields and mark fields that should compare exactly after normalization.
3. Run with a conservative automatic threshold and inspect review rows.
4. Download canonical records and retain decision rows as the merge audit trail.

Company account cleanup

Use name, domain, email, phone, address, and city. Give domain and phone stronger weights when they are trustworthy. Keep the automatic threshold high enough that name similarity alone cannot merge unrelated businesses.

Product catalog matching

Map SKU or manufacturer part number as exact fields and product title as a fuzzy field. Review borderline rows before applying merges to inventory or pricing systems.

Location deduplication

Compare normalized name, address, city, postal code, and phone. Preserve each source record so downstream operators can trace a canonical location back to every import.

Input Configuration

Parameter	Type	Default	Description
targets	array	Included fixture	Multiple record, CSV, or URL sources.
records	array	[]	Records for one source when targets is empty.
csvText	string	Empty	Quoted CSV content for one source.
sourceUrl	string	Empty	Public HTTP(S) JSON or CSV source.
fetchUrls	boolean	false	Enables bounded public source retrieval.
idField	string	id	Field used as the record identifier.
matchFields	string array	Common identity fields	Fields participating in the weighted score.
exactFields	string array	Domain, email, phone	Fields scored as exact after normalization.
fieldWeights	object	Field-specific	Positive scoring weights keyed by field name.
matchThreshold	number	0.82	Minimum score for automatic clustering.
reviewThreshold	number	0.67	Minimum score for a review candidate.
maxTargets	integer	2	Source cap, with a hard maximum of 20.
maxItems	integer	250	Per-source record cap, with a hard maximum of 500.
maxTextBytes	integer	250000	Maximum fetched response size.
fetchTimeoutMs	integer	10000	Per-request timeout in milliseconds.
includeReport	boolean	true	Creates JSON and Markdown report files.

Public retrieval accepts HTTP and HTTPS only. Credentialed URLs, redirects, localhost, private networks, reserved ranges, private DNS answers, oversized bodies, and slow responses are rejected. Source-load failures produce a free diagnostic row rather than a paid match result.

Output Format

{
  "sourceId": "accounts",
  "recordId": "acct-2",
  "clusterId": "cluster_2be690d4b9f98c16",
  "matchedRecordId": "acct-1",
  "decision": "merge",
  "matchScore": 0.95,
  "matchReasons": ["name:similar", "domain:exact", "phone:exact"],
  "originalRecord": { "id": "acct-2", "name": "ACME Roofing - Nashville" },
  "canonicalRecord": { "id": "acct-1", "name": "Acme Roofing Nashville LLC" },
  "normalizedFields": { "name": "acme roofing nashville", "domain": "acmeroofing.example" },
  "issueCodes": [],
  "issueCount": 0,
  "status": "ready",
  "recommendation": "Merge this record into the canonical record after confirming source ownership rules."
}

Report files include ENTITY_DEDUPLICATION_MATCHER_RESULTS.json, ENTITY_DEDUPLICATION_MATCHER_CANONICAL_RECORDS.json, ENTITY_DEDUPLICATION_MATCHER_SUMMARY.json, and ENTITY_DEDUPLICATION_MATCHER_REPORT.md.

Integration Examples

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('junipr/entity-deduplication-matcher').call({
  records: accountRows,
  idField: 'accountId',
  matchFields: ['name', 'domain', 'phone', 'city'],
  exactFields: ['domain', 'phone'],
  maxItems: 250
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const approvedMerges = items.filter((row) => row.decision === 'merge');

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("junipr/entity-deduplication-matcher").call(run_input={
    "records": records,
    "idField": "accountId",
    "maxItems": 250,
})
decisions = client.dataset(run["defaultDatasetId"]).list_items().items
review_queue = [row for row in decisions if row["decision"] == "review"]

Tips and Advanced Usage

Tune thresholds safely

Start with a high matchThreshold, inspect the returned scores, then lower it only when known duplicates remain unique. Keep reviewThreshold below the automatic threshold so borderline candidates are surfaced without joining clusters.

Choose fields with intent

Do not give a weak field such as city the same influence as a trusted domain or normalized phone. Missing fields are excluded from a pair's denominator, so a pair can still match when one optional field is absent.

Control pair growth

The actor compares bounded record pairs deterministically. Use multiple sources or smaller runs for very large tables. The 500-record hard cap prevents accidental quadratic workloads and unbounded event charges.

Pricing

Prices follow the actor's locked pay-per-event contract and include platform usage for this bounded utility.

Event	Price	Charged when
actor-start	$0.00500	Run setup is accepted.
record-matched	$0.00390	A ready record decision is emitted.
issue-detected	$0.00372	A record-level review or validation issue is emitted.
qa-report-generated	$0.05000	Results, canonical records, summary, and report files are created.

FAQ

Does it automatically modify my source system?

No. It emits decisions and canonical records. Apply merges only after your own ownership and rollback checks.

Are fuzzy scores generated by an LLM?

No. Scores are deterministic combinations of normalized exact matches, edit similarity, and token overlap.

Can it read CSV with quoted commas?

Yes. The parser handles quoted commas, escaped quotes, embedded newlines, BOM markers, and duplicate headers.

Why is a pair marked review instead of merge?

Its best score reached reviewThreshold but stayed below matchThreshold.

Which record becomes canonical?

The most complete record in each automatic cluster wins; input order breaks ties.

Can it fetch private storage URLs?

Private and credentialed URLs are intentionally blocked. Use a time-limited public HTTPS URL without embedded credentials, or submit records directly.

Related Actors

• Domain Extractor Grouper
• CSV Deduper Normalizer
• CSV to Dashboard Summary
• URL Canonicalizer

Limitations and Safe Use

Fuzzy identity decisions are probabilistic data-cleaning signals, not proof that two real-world entities are identical. Review high-impact merges, keep source IDs, and avoid submitting personal or confidential fields that are unnecessary for matching.