Lead Cleaner — Fuzzy Deduplication & Normalization

Point it at the dataset your scraper just produced. It normalizes the contact fields, finds the duplicates that exact matching cannot see, and tells you why it merged every single one.

Restaurante El Puerto, S.L.   ana.ruiz@elpuerto.es    +34 915 55 12 34
RESTAURANTE EL PUERTO SL      ana.ruiz@elpuerto.es    915 551 234
El Puerto Restaurante SL      info@elpuerto.es        0034915551234

Three rows, one lead. An exact-match deduplicator keeps all three.

---

Why this exists

Lead-generation scrapers are among the highest-traffic tools on Apify, and their output is dirty by construction: the same business appears under several name spellings, phone formats and mailbox aliases. The deduplicators currently available treat this as a string-equality problem — lowercase the field, trim the whitespace, drop exact repeats — which resolves almost none of it.

This Actor compares records after normalizing them, so a legal-form difference, a phone written five ways or a +tag on a Gmail address stop being differences at all.

What it does

Normalization

Field	What happens
Phone	Parsed to E.164 per country, extensions separated, line type detected, Excel 3.4916E+10 mangling detected and refused rather than guessed
Company	Legal form split from the name and canonicalized across 40+ forms (S.L., GmbH & Co. KG, S.r.l., B.V., Lda., S.Coop.…)
Person	Apellido, Nombre reordered, honorifics and credentials stripped, Spanish particles kept with the family name, everyday names mapped to legal ones (Pepe → José)
Email	Wrappers unwrapped, Gmail dots and +tags folded, role mailboxes flagged, disposable domains flagged, optional MX check
Website	Reduced to the registrable domain, multi-label suffixes handled (acme.co.uk)
Tax ID	NIF, NIE and CIF check digits, EU VAT, plus non-EU national schemes (GB, CH, NO)
Address	Street type canonicalized (C/ → calle), number and floor separated, embedded postal code extracted, articles ignored in matching but kept in the output

Deduplication

• MinHash + LSH candidate generation on the company name, so it stays tractable instead of comparing every pair
• Phonetic name matching tuned for Spanish spelling (Gil/Jil, Vega/Bega, González/Gonsales)
• Evidence-based scoring: every merge carries the reasons that produced it, and confidence is combined as noisy-OR so a three-signal match scores above a one-signal match instead of both pinning at 1.0
• Conflicting evidence blocks a merge outright: two different valid tax IDs, two different company domains, two people at one organisation, or two branches of one brand in different cities
• Survivorship fills the surviving row's gaps from its duplicates, so merging never loses the only phone number in the group

Safety

• reportOnly annotates instead of merging. Run it first.
• Borderline pairs go to a separate review-pairs dataset instead of being merged silently.
• Rows with nothing comparable left after normalization are passed through untouched, never merged on absent evidence.

Quality benchmark

Reproducible without an Apify account:

$python benchmark/run_benchmark.py

Dataset: 63 labelled rows, 26 true duplicate pairs, covering Spanish, German, British, Italian, Dutch, French, Portuguese and US records — including 16 rows written specifically to break the matcher.

Method	Precision	Recall	F1	Wrong merges	Missed
Exact email match	0.909	0.385	0.541	1	16
Exact match on all fields	1.000	0.000	0.000	0	26
Casefolded company + phone digits	1.000	0.000	0.000	0	26
This Actor	1.000	1.000	1.000	0	0

Read this before trusting that last row. The benchmark dataset and the matching engine were written by the same author, so a perfect score measures internal consistency, not generalization.

What it is worth: the adversarial rows were added after the engine was working, and they broke it — a nickname that no similarity measure connects, a one-character typo in a scraped domain, and two branches of one chain wrongly merged into one row. F1 dropped to 0.939. The three rules written to fix those cases are in scoring.py and each is stated in general terms, not tuned to the example that exposed it. That is the honest claim: this benchmark has found real defects, and it will find more if you add your own cases. Every row in benchmark/dataset.jsonl carries a _note explaining what it tests, so you can judge whether those cases resemble yours.

The baseline rows are the unambiguous part: those methods are what the alternatives do, scored on the same data.

Note also that the labelled set cannot discriminate every parameter. Its duplicates are all reachable through deterministic keys, so it scores 1.000 at every candidate-generation threshold from 30 to 80; that setting was chosen from the scale measurements below instead.

Scale

$python benchmark/run_scale.py

Measured on synthetic lists with a 5% chain share and a 20% duplicate rate:

Rows	Runtime	Candidate pairs	Peak RSS
10,000	6 s	19,836	64 MB
50,000	36 s	463,059	365 MB
100,000	74 s	1,119,733	871 MB
200,000	163 s	3,520,685	2,151 MB

Candidate pairs grow faster than rows, so memory is the real constraint, not row count. Above 100,000 rows, run the Actor with at least 4 GB.

The maximum is 200,000 rows per run. A list whose rows share company names or domains heavily — a chain, a franchise network, or the same scrape concatenated several times — can hit the candidate-pair ceiling below that; the run then fails with the specific remedy rather than being killed for memory.

Input

Field	Type	Default	Notes
datasetId	string	—	The dataset to clean
datasetIds	array	[]	Deduplicate across several datasets at once
items	array	[]	Inline rows instead of a dataset
entityLevel	contact | company	contact	contact keeps two colleagues as two rows
reportOnly	boolean	false	Annotate without merging
threshold	50–95	75	Evidence required to merge
defaultCountry	string	—	ISO code for phones and tax IDs without a prefix
checkMx	boolean	false	DNS MX lookup per distinct domain
fieldMap	object	{}	Override column auto-detection
lshThreshold	30–80	65	Candidate generation sensitivity

Columns are detected automatically from their names — company_name, companyName and Company Name all work. Override with fieldMap only when the detection line in the log shows it picked the wrong one.

Output

Every output row keeps all its original fields and adds:

Field	Meaning
_normalized	Every normalized value, so you can see what was compared
_clusterId	Group this row belongs to
_mergedCount	How many input rows became this one
_mergedFromIndices	Which input rows were merged into it
_matchScore	Confidence, 0–1
_matchReasons	Why they were merged, pair by pair
_issues	Data problems found (invalid phone, disposable domain, failed checksum…)
alternateEmails / alternatePhones	Identifiers from the merged rows that were kept rather than discarded

In reportOnly mode you get one output row per input row, with _isCanonical and _duplicateOf instead of the merge fields.

Run statistics — including the detected field map and the blocking breakdown — are written to the STATS key of the run's key-value store.

Pricing

$0.40 per 1,000 input rows read, whether or not a row turns out to be a duplicate. The work is proportional to what was read, and charging per surviving row would mean you pay less the worse the deduplication is.

There is a second, standard Apify charge of $0.00005 per GB of allocated memory at Actor start — $0.0002 for a default 4 GB run. Platform compute is billed to you separately by Apify; it runs under 0.1% of the row charge at every size measured above.

That is the whole meter. No per-output-row fee, no per-field fee, and nothing charged for rows that were passed through untouched.

What it deliberately does not do

No SMTP probing. Email checks stop at syntax, MX and heuristics. Verifying deliverability by handshake requires IP warm-up, sender reputation and blocklist management; that maintenance burden would end up priced into every run. If you need mailbox-level verification, use a dedicated verifier after this step.

No VIES lookups. Tax IDs are checked for structure and check digits only. Confirming that a VAT number is registered requires the European Commission's VIES service, which is frequently unavailable and would make this Actor's runtime depend on it.

No enrichment. It cleans what you already have. It does not add fields.

Known limits

• Two records that share only a person's name, with no company and no shared mailbox, phone, domain or tax ID, are not compared. A name alone scores well below the merge threshold, so generating those pairs would cost time without ever producing a merge.
• Phonetic matching and the everyday-name table are tuned for Spanish; other languages fall back to standard Metaphone.
• The public-suffix list used for domains is a bundled subset covering the common cases, not the full IANA list.
• The disposable-domain list is a curated seed, refreshed per release rather than live.

Running it locally

pip install -r requirements.txt
python tests/test_normalize.py
python tests/test_engine.py
python benchmark/run_benchmark.py
python benchmark/run_scale.py

The leadclean package imports nothing from Apify, which is what lets the benchmark run anywhere.