# Lead Cleaner — Fuzzy Deduplication & Normalization (`javitax47/lead-cleaner`) Actor Deduplicate and normalize scraped lead lists. Matches records that exact deduplicators miss: legal-form variants, phone formats, email aliases and phonetic name spellings. Publishes a reproducible quality benchmark. - **URL**: https://apify.com/javitax47/lead-cleaner.md - **Developed by:** [Javier Camarena](https://apify.com/javitax47) (community) - **Categories:** Lead generation, Automation, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.40 / 1,000 result rows This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Lead Cleaner — Fuzzy Deduplication & Normalization Point it at the dataset your scraper just produced. It normalizes the contact fields, finds the duplicates that exact matching cannot see, and tells you why it merged every single one. ```` Restaurante El Puerto, S.L. ana.ruiz@elpuerto.es +34 915 55 12 34 RESTAURANTE EL PUERTO SL ana.ruiz@elpuerto.es 915 551 234 El Puerto Restaurante SL info@elpuerto.es 0034915551234 ```` Three rows, one lead. An exact-match deduplicator keeps all three. --- ### Why this exists Lead-generation scrapers are among the highest-traffic tools on Apify, and their output is dirty by construction: the same business appears under several name spellings, phone formats and mailbox aliases. The deduplicators currently available treat this as a string-equality problem — lowercase the field, trim the whitespace, drop exact repeats — which resolves almost none of it. This Actor compares records *after* normalizing them, so a legal-form difference, a phone written five ways or a `+tag` on a Gmail address stop being differences at all. ### What it does **Normalization** | Field | What happens | |---|---| | Phone | Parsed to E.164 per country, extensions separated, line type detected, Excel `3.4916E+10` mangling detected and refused rather than guessed | | Company | Legal form split from the name and canonicalized across 40+ forms (`S.L.`, `GmbH & Co. KG`, `S.r.l.`, `B.V.`, `Lda.`, `S.Coop.`…) | | Person | `Apellido, Nombre` reordered, honorifics and credentials stripped, Spanish particles kept with the family name, everyday names mapped to legal ones (`Pepe` → `José`) | | Email | Wrappers unwrapped, Gmail dots and `+tags` folded, role mailboxes flagged, disposable domains flagged, optional MX check | | Website | Reduced to the registrable domain, multi-label suffixes handled (`acme.co.uk`) | | Tax ID | NIF, NIE and CIF check digits, EU VAT, plus non-EU national schemes (GB, CH, NO) | | Address | Street type canonicalized (`C/` → `calle`), number and floor separated, embedded postal code extracted, articles ignored in matching but kept in the output | **Deduplication** - MinHash + LSH candidate generation on the company name, so it stays tractable instead of comparing every pair - Phonetic name matching tuned for Spanish spelling (`Gil`/`Jil`, `Vega`/`Bega`, `González`/`Gonsales`) - Evidence-based scoring: every merge carries the reasons that produced it, and confidence is combined as noisy-OR so a three-signal match scores above a one-signal match instead of both pinning at 1.0 - Conflicting evidence blocks a merge outright: two different valid tax IDs, two different company domains, two people at one organisation, or two branches of one brand in different cities - Survivorship fills the surviving row's gaps from its duplicates, so merging never loses the only phone number in the group **Safety** - `reportOnly` annotates instead of merging. Run it first. - Borderline pairs go to a separate `review-pairs` dataset instead of being merged silently. - Rows with nothing comparable left after normalization are passed through untouched, never merged on absent evidence. ### Quality benchmark Reproducible without an Apify account: ```bash python benchmark/run_benchmark.py ```` Dataset: 63 labelled rows, 26 true duplicate pairs, covering Spanish, German, British, Italian, Dutch, French, Portuguese and US records — including 16 rows written specifically to break the matcher. | Method | Precision | Recall | F1 | Wrong merges | Missed | |---|---|---|---|---|---| | Exact email match | 0.909 | 0.385 | 0.541 | 1 | 16 | | Exact match on all fields | 1.000 | 0.000 | 0.000 | 0 | 26 | | Casefolded company + phone digits | 1.000 | 0.000 | 0.000 | 0 | 26 | | **This Actor** | **1.000** | **1.000** | **1.000** | **0** | **0** | **Read this before trusting that last row.** The benchmark dataset and the matching engine were written by the same author, so a perfect score measures internal consistency, not generalization. What it is worth: the adversarial rows were added *after* the engine was working, and they broke it — a nickname that no similarity measure connects, a one-character typo in a scraped domain, and two branches of one chain wrongly merged into one row. F1 dropped to 0.939. The three rules written to fix those cases are in `scoring.py` and each is stated in general terms, not tuned to the example that exposed it. That is the honest claim: this benchmark has found real defects, and it will find more if you add your own cases. Every row in `benchmark/dataset.jsonl` carries a `_note` explaining what it tests, so you can judge whether those cases resemble yours. The baseline rows are the unambiguous part: those methods are what the alternatives do, scored on the same data. Note also that the labelled set cannot discriminate every parameter. Its duplicates are all reachable through deterministic keys, so it scores 1.000 at every candidate-generation threshold from 30 to 80; that setting was chosen from the scale measurements below instead. ### Scale ```bash python benchmark/run_scale.py ``` Measured on synthetic lists with a 5% chain share and a 20% duplicate rate: | Rows | Runtime | Candidate pairs | Peak RSS | |---|---|---|---| | 10,000 | 6 s | 19,836 | 64 MB | | 50,000 | 36 s | 463,059 | 365 MB | | 100,000 | 74 s | 1,119,733 | 871 MB | | 200,000 | 163 s | 3,520,685 | 2,151 MB | Candidate pairs grow faster than rows, so **memory is the real constraint, not row count**. Above 100,000 rows, run the Actor with at least 4 GB. The maximum is 200,000 rows per run. A list whose rows share company names or domains heavily — a chain, a franchise network, or the same scrape concatenated several times — can hit the candidate-pair ceiling below that; the run then fails with the specific remedy rather than being killed for memory. ### Input | Field | Type | Default | Notes | |---|---|---|---| | `datasetId` | string | — | The dataset to clean | | `datasetIds` | array | `[]` | Deduplicate across several datasets at once | | `items` | array | `[]` | Inline rows instead of a dataset | | `entityLevel` | `contact` | `company` | `contact` | `contact` keeps two colleagues as two rows | | `reportOnly` | boolean | `false` | Annotate without merging | | `threshold` | 50–95 | `75` | Evidence required to merge | | `defaultCountry` | string | — | ISO code for phones and tax IDs without a prefix | | `checkMx` | boolean | `false` | DNS MX lookup per distinct domain | | `fieldMap` | object | `{}` | Override column auto-detection | | `lshThreshold` | 30–80 | `65` | Candidate generation sensitivity | Columns are detected automatically from their names — `company_name`, `companyName` and `Company Name` all work. Override with `fieldMap` only when the detection line in the log shows it picked the wrong one. ### Output Every output row keeps all its original fields and adds: | Field | Meaning | |---|---| | `_normalized` | Every normalized value, so you can see what was compared | | `_clusterId` | Group this row belongs to | | `_mergedCount` | How many input rows became this one | | `_mergedFromIndices` | Which input rows were merged into it | | `_matchScore` | Confidence, 0–1 | | `_matchReasons` | Why they were merged, pair by pair | | `_issues` | Data problems found (invalid phone, disposable domain, failed checksum…) | | `alternateEmails` / `alternatePhones` | Identifiers from the merged rows that were kept rather than discarded | In `reportOnly` mode you get one output row per input row, with `_isCanonical` and `_duplicateOf` instead of the merge fields. Run statistics — including the detected field map and the blocking breakdown — are written to the `STATS` key of the run's key-value store. ### Pricing **$0.40 per 1,000 input rows read**, whether or not a row turns out to be a duplicate. The work is proportional to what was read, and charging per surviving row would mean you pay less the worse the deduplication is. There is a second, standard Apify charge of $0.00005 per GB of allocated memory at Actor start — $0.0002 for a default 4 GB run. Platform compute is billed to you separately by Apify; it runs under 0.1% of the row charge at every size measured above. That is the whole meter. No per-output-row fee, no per-field fee, and nothing charged for rows that were passed through untouched. ### What it deliberately does not do **No SMTP probing.** Email checks stop at syntax, MX and heuristics. Verifying deliverability by handshake requires IP warm-up, sender reputation and blocklist management; that maintenance burden would end up priced into every run. If you need mailbox-level verification, use a dedicated verifier after this step. **No VIES lookups.** Tax IDs are checked for structure and check digits only. Confirming that a VAT number is registered requires the European Commission's VIES service, which is frequently unavailable and would make this Actor's runtime depend on it. **No enrichment.** It cleans what you already have. It does not add fields. ### Known limits - Two records that share only a person's name, with no company and no shared mailbox, phone, domain or tax ID, are not compared. A name alone scores well below the merge threshold, so generating those pairs would cost time without ever producing a merge. - Phonetic matching and the everyday-name table are tuned for Spanish; other languages fall back to standard Metaphone. - The public-suffix list used for domains is a bundled subset covering the common cases, not the full IANA list. - The disposable-domain list is a curated seed, refreshed per release rather than live. ### Running it locally ```bash pip install -r requirements.txt python tests/test_normalize.py python tests/test_engine.py python benchmark/run_benchmark.py python benchmark/run_scale.py ``` The `leadclean` package imports nothing from Apify, which is what lets the benchmark run anywhere. # Actor input Schema ## `datasetId` (type: `string`): Dataset to clean — usually the output of the scraper you just ran. Copy the ID from that run's Storage tab. ## `datasetIds` (type: `array`): Merge several datasets and deduplicate across all of them at once. Use this to fold a new scrape into last month's list. ## `items` (type: `array`): Rows to clean, passed directly instead of from a dataset. Useful for testing and for API callers. ## `entityLevel` (type: `string`): contact — two people at the same company stay as two rows (default; this is what a sales list means by a duplicate). company — one row per organisation, contacts collapsed. ## `reportOnly` (type: `boolean`): Keep every input row and annotate it with its cluster, match score and reasons. Run this first: it shows exactly what would be merged before anything is removed. ## `threshold` (type: `integer`): How much evidence is required to merge, from 50 (aggressive) to 95 (conservative). 75 is the default: the labelled benchmark scores a perfect F1 anywhere from 60 to 75 and starts missing duplicates at 80. Pairs within 10 points below the threshold are written to the review-pairs dataset instead of being merged. ## `defaultCountry` (type: `string`): ISO two-letter code used to interpret phone numbers and tax IDs that have no country prefix. Set it when your list is single-country — it substantially improves phone matching. ## `checkMx` (type: `boolean`): Adds a DNS lookup per distinct domain (cached). Flags addresses whose domain cannot receive mail. No SMTP probing is performed and no message is ever sent. ## `fieldMap` (type: `object`): Only needed when auto-detection picks the wrong column. Keys: email, phone, company, person, domain, tax\_id, address, city, postal\_code, country. Values are the source column names. ## `lshThreshold` (type: `integer`): Advanced. How similar two company names must look before the records are compared in detail, from 30 (compare more pairs, much slower) to 80 (faster, may miss heavily mangled duplicates). Lowering it from the default 65 to 50 multiplies candidate pairs by about nine and finds roughly 0.7% more duplicates. ## Actor input object example ```json { "datasetIds": [], "items": [], "entityLevel": "contact", "reportOnly": false, "threshold": 75, "defaultCountry": "ES", "checkMx": false, "fieldMap": {}, "lshThreshold": 65 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("javitax47/lead-cleaner").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("javitax47/lead-cleaner").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call javitax47/lead-cleaner --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=javitax47/lead-cleaner", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Lead Cleaner — Fuzzy Deduplication & Normalization", "description": "Deduplicate and normalize scraped lead lists. Matches records that exact deduplicators miss: legal-form variants, phone formats, email aliases and phonetic name spellings. Publishes a reproducible quality benchmark.", "version": "0.1", "x-build-id": "zOcxZh3YX4zwqzbzB" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/javitax47~lead-cleaner/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-javitax47-lead-cleaner", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/javitax47~lead-cleaner/runs": { "post": { "operationId": "runs-sync-javitax47-lead-cleaner", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/javitax47~lead-cleaner/run-sync": { "post": { "operationId": "run-sync-javitax47-lead-cleaner", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "datasetId": { "title": "Source dataset ID", "type": "string", "description": "Dataset to clean — usually the output of the scraper you just ran. Copy the ID from that run's Storage tab." }, "datasetIds": { "title": "Additional dataset IDs", "type": "array", "description": "Merge several datasets and deduplicate across all of them at once. Use this to fold a new scrape into last month's list.", "default": [], "items": { "type": "string" } }, "items": { "title": "Inline records", "type": "array", "description": "Rows to clean, passed directly instead of from a dataset. Useful for testing and for API callers.", "default": [] }, "entityLevel": { "title": "What counts as one duplicate", "enum": [ "contact", "company" ], "type": "string", "description": "contact — two people at the same company stay as two rows (default; this is what a sales list means by a duplicate). company — one row per organisation, contacts collapsed.", "default": "contact" }, "reportOnly": { "title": "Report only, do not merge", "type": "boolean", "description": "Keep every input row and annotate it with its cluster, match score and reasons. Run this first: it shows exactly what would be merged before anything is removed.", "default": false }, "threshold": { "title": "Match threshold", "minimum": 50, "maximum": 95, "type": "integer", "description": "How much evidence is required to merge, from 50 (aggressive) to 95 (conservative). 75 is the default: the labelled benchmark scores a perfect F1 anywhere from 60 to 75 and starts missing duplicates at 80. Pairs within 10 points below the threshold are written to the review-pairs dataset instead of being merged.", "default": 75 }, "defaultCountry": { "title": "Default country", "type": "string", "description": "ISO two-letter code used to interpret phone numbers and tax IDs that have no country prefix. Set it when your list is single-country — it substantially improves phone matching." }, "checkMx": { "title": "Check email domains have MX records", "type": "boolean", "description": "Adds a DNS lookup per distinct domain (cached). Flags addresses whose domain cannot receive mail. No SMTP probing is performed and no message is ever sent.", "default": false }, "fieldMap": { "title": "Field mapping override", "type": "object", "description": "Only needed when auto-detection picks the wrong column. Keys: email, phone, company, person, domain, tax_id, address, city, postal_code, country. Values are the source column names.", "default": {} }, "lshThreshold": { "title": "Candidate generation sensitivity", "minimum": 30, "maximum": 80, "type": "integer", "description": "Advanced. How similar two company names must look before the records are compared in detail, from 30 (compare more pairs, much slower) to 80 (faster, may miss heavily mangled duplicates). Lowering it from the default 65 to 50 multiplies candidate pairs by about nine and finds roughly 0.7% more duplicates.", "default": 65 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```