Pricing

Pay per usage

Superclean Dedupe

Deduplicate lead records using fuzzy matching. Exact email/phone matching, Jaro-Winkler for names, token-set ratio for companies. Smart field auto-detection. Input: array of record objects. Output: clustered records with isCanonical, matchScore, matchReasons. Batch or instant pair API.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Superlative

Actor stats

Bookmarked

Total users

Monthly active users

a month ago

Last modified

Product Matching API

vivid_astronaut/product-matching

BRAINIALL Team

Array to Excel

hamza.alwan/array-to-excel

Converts any array of objects to Excel

Hamza Alwan

Fuzzy Search Dataset Actor

dtrungtin/fuzzy-search-dataset-actor

Search any Apify dataset using typo-tolerant fuzzy matching.

Tin

CRM Contact Cleanup & Dedupe Prep

critd/contact-cleanup

Turn supplied contact rows into a reimport-ready cleaned file plus focused review and possible-match files. Normalize mapped URL, email, and U.S.-leaning address values while preserving original fields. No scraping, finding, verification, enrichment, geocoding, survivor selection, or merging.

Critical Distinction

Entity Deduplication Matcher

junipr/entity-deduplication-matcher

Fuzzy-match and deduplicate company, product, location, or entity rows into canonical records.

junipr

JobMatch AI

peaceful_mix/JobMatch-AI

Intelligent system matching resumes to jobs using AI. 📄 Resume Analysis: Extracts data from PDFs. 🤖 AI Matching: Uses Gemini AI for accurate matching. 🎯 Smart Scoring: Provides a suitability score (0-100). 💡 Insights: Gives match reasons and prep tips. ⚡ Fast: Quick analysis of many postings.

Vidip Ghosh

Source Quote Verifier

vincesoft/source-quote-verifier

Check whether a public web source contains claimed text using exact, normalized, or deterministic fuzzy matching, with passage context and retrieval provenance.

VincSoft

Company Scout

tri_angle/company-scout

An Actor that finds YCombinator companies matching your criteria, enriches each with its matching LinkedIn company record, and scrapes LinkedIn job postings for the resulting shortlist.

Tri⟁angle

Full Sanctions Screener API

george.the.developer/full-sanctions-screener

Screen entities against OFAC SDN, EU, UN, and UK OFSI sanctions lists with fuzzy matching. Instant API with standby mode.

George Kioko

HubSpot Company Enrichment & Fuzzy Matcher for Clay

alizarin_refrigerator-owner/hubspot-company-enrichment-fuzzy-matcher-for-clay

Fuzzy match and enrich companies against your HubSpot CRM using multi-signal matching (domain, company name, phone, location). Returns HubSpot ID, lifecycle stage, deal status & confidence scores. Perfect for Clay workflows, lead deduplication, and outbound enrichment.

The Howlers

{ "name": "superclean-dedupe", "version": "0.1.0", "type": "module", "description": "Deduplicate lead records using fuzzy matching. Exact, fuzzy, and multi-field composite scoring with smart auto-detection.", "main": "src/main.js", "scripts": { "start": "node src/main.js", "test": "node --test tests/" }, "author": "Superlative", "license": "ISC", "dependencies": { "apify": "^3.3.2", "fuzzball": "^2.1.3" } }

{ "actorSpecification": 1, "name": "superclean-dedupe", "title": "Superclean Dedupe", "description": "Deduplicate lead records using fuzzy matching. Exact email/phone matching, Jaro-Winkler for names, token-set ratio for companies. Input: array of record objects. Output: records annotated with clusterId, isCanonical, matchScore, matchReasons. Batch or instant pair-comparison API.", "version": "0.1", "dockerfile": "../Dockerfile", "usesStandbyMode": true, "buildTag": "latest", "webServerSchema": "./web_server_openapi.json", "input": "./input_schema.json", "output": "./output_schema.json", "storages": { "dataset": { "actorSpecification": 1, "title": "Deduplicated records", "views": { "results": { "title": "Results", "transformation": { "fields": [ "id", "clusterId", "clusterSize", "isCanonical", "duplicateOf", "matchScore", "matchReasons", "confidence", "record" ] }, "display": { "component": "table", "properties": { "id": { "label": "ID", "format": "number" }, "clusterId": { "label": "Cluster", "format": "text" }, "clusterSize": { "label": "Size", "format": "number" }, "isCanonical": { "label": "Canonical", "format": "text" }, "duplicateOf": { "label": "Duplicate Of", "format": "number" }, "matchScore": { "label": "Score", "format": "number" }, "matchReasons": { "label": "Reasons", "format": "text" }, "confidence": { "label": "Confidence", "format": "number" }, "record": { "label": "Record", "format": "text" } } } } } } }, "defaultRunOptions": { "build": "latest", "memoryMbytes": 256, "timeoutSecs": 600 }, "minMemoryMbytes": 128, "maxMemoryMbytes": 1024 }

{ "title": "Superclean Dedupe Input", "type": "object", "schemaVersion": 1, "properties": { "records": { "title": "Records", "type": "array", "description": "Array of record objects to deduplicate. Each record should have fields like name, email, company, phone, etc.", "editor": "json", "prefill": [ { "name": "Jim Smith", "email": "jim@acme.com", "company": "ACME Corp" }, { "name": "James Smith", "email": "jim@acme.com", "company": "Acme Corporation" }, { "name": "Jane Doe", "email": "jane@example.com", "company": "Example Inc" }, { "name": "J. Doe", "email": "jane@example.com", "company": "Example" }, { "name": "Bob Wilson", "email": "bob@other.com", "company": "Other LLC" } ] }, "record": { "title": "Record (single)", "type": "object", "description": "Single record object — API shorthand. If both record and records are provided, record is prepended to the list.", "editor": "json", "sectionCaption": "API shorthand" }, "threshold": { "title": "Threshold", "type": "number", "description": "Similarity score threshold for considering two records as duplicates (0-1). Higher = stricter (fewer false merges). Lower = more aggressive (catches more duplicates but may over-merge).", "default": 0.85, "minimum": 0.1, "maximum": 1.0, "sectionCaption": "Configuration" }, "matchFields": { "title": "Match fields", "type": "array", "description": "Optional. Specify which fields to match on and how. If omitted, fields are auto-detected from record keys. Each item: {field, method, weight}. Methods: exact, jaro-winkler, levenshtein, token-set.", "editor": "json" } }, "required": [] }

{ "openapi": "3.0.0", "info": { "title": "Superclean Dedupe Standby API", "description": "Compare two records to check if they are duplicates. Returns similarity score, match reasons, and confidence.", "version": "0.1.0" }, "paths": { "/": { "get": { "summary": "Compare two records for duplicates", "parameters": [ { "name": "a", "in": "query", "required": true, "description": "First record as URL-encoded JSON object", "schema": { "type": "string" } }, { "name": "b", "in": "query", "required": true, "description": "Second record as URL-encoded JSON object", "schema": { "type": "string" } }, { "name": "threshold", "in": "query", "required": false, "description": "Similarity threshold (default 0.85)", "schema": { "type": "number", "default": 0.85 } } ], "responses": { "200": { "description": "Comparison result", "content": { "application/json": { "schema": { "type": "object", "properties": { "isDuplicate": { "type": "boolean" }, "score": { "type": "number" }, "reasons": { "type": "array", "items": { "type": "string" } }, "confidence": { "type": "number" }, "fieldsUsed": { "type": "array", "items": { "type": "string" } } } } } } }, "400": { "description": "Missing or invalid parameters" }, "405": { "description": "Method not allowed (use GET)" } } } } } }

{ "actorOutputSchemaVersion": 1, "title": "Superclean Dedupe Output", "description": "Deduplicated records with cluster membership, match scores, and canonical flags.", "properties": { "allResults": { "type": "string", "title": "All Results", "description": "All records annotated with cluster IDs, canonical flags, match scores, and reasons", "template": "{{links.apiDefaultDatasetUrl}}/items" } } }

1/** 2 * Union-Find (Disjoint Set Union) with path compression and union by rank. 3 * Used to cluster duplicate records via transitive closure: 4 * if A=B and B=C, then A, B, C are all in the same cluster. 5 */ 6export class UnionFind { 7 constructor(n) { 8 this.parent = Array.from({ length: n }, (_, i) => i); 9 this.rank = new Array(n).fill(0); 10 } 11 12 find(x) { 13 if (this.parent[x] !== x) { 14 this.parent[x] = this.find(this.parent[x]); // path compression 15 } 16 return this.parent[x]; 17 } 18 19 union(x, y) { 20 const rx = this.find(x); 21 const ry = this.find(y); 22 if (rx === ry) return false; 23 24 // union by rank 25 if (this.rank[rx] < this.rank[ry]) { 26 this.parent[rx] = ry; 27 } else if (this.rank[rx] > this.rank[ry]) { 28 this.parent[ry] = rx; 29 } else { 30 this.parent[ry] = rx; 31 this.rank[rx]++; 32 } 33 return true; 34 } 35 36 /** 37 * Returns clusters as Map<rootIndex, number[]> 38 * where each value is an array of member indices. 39 */ 40 clusters() { 41 const map = new Map(); 42 for (let i = 0; i < this.parent.length; i++) { 43 const root = this.find(i); 44 if (!map.has(root)) map.set(root, []); 45 map.get(root).push(i); 46 } 47 return map; 48 } 49}

1import { createRequire } from 'node:module'; 2const require = createRequire(import.meta.url); 3const fuzzball = require('fuzzball'); 4 5// ── Jaro-Winkler (custom, ~30 lines, better for short strings like names) ── 6 7function jaro(a, b) { 8 if (a === b) return 1; 9 const maxDist = Math.floor(Math.max(a.length, b.length) / 2) - 1; 10 if (maxDist < 0) return 0; 11 12 const aMatched = new Array(a.length).fill(false); 13 const bMatched = new Array(b.length).fill(false); 14 let matches = 0; 15 let transpositions = 0; 16 17 for (let i = 0; i < a.length; i++) { 18 const start = Math.max(0, i - maxDist); 19 const end = Math.min(b.length - 1, i + maxDist); 20 for (let j = start; j <= end; j++) { 21 if (bMatched[j] || a[i] !== b[j]) continue; 22 aMatched[i] = true; 23 bMatched[j] = true; 24 matches++; 25 break; 26 } 27 } 28 29 if (matches === 0) return 0; 30 31 let k = 0; 32 for (let i = 0; i < a.length; i++) { 33 if (!aMatched[i]) continue; 34 while (!bMatched[k]) k++; 35 if (a[i] !== b[k]) transpositions++; 36 k++; 37 } 38 39 return (matches / a.length + matches / b.length + (matches - transpositions / 2) / matches) / 3; 40} 41 42export function jaroWinkler(a, b, prefixScale = 0.1) { 43 if (!a || !b) return 0; 44 const al = a.toLowerCase(); 45 const bl = b.toLowerCase(); 46 const j = jaro(al, bl); 47 let prefix = 0; 48 for (let i = 0; i < Math.min(4, al.length, bl.length); i++) { 49 if (al[i] === bl[i]) prefix++; 50 else break; 51 } 52 return j + prefix * prefixScale * (1 - j); 53} 54 55// ── Levenshtein similarity (normalized 0-1) ── 56 57export function levenshteinSimilarity(a, b) { 58 if (!a && !b) return 1; 59 if (!a || !b) return 0; 60 const al = a.toLowerCase(); 61 const bl = b.toLowerCase(); 62 if (al === bl) return 1; 63 const maxLen = Math.max(al.length, bl.length); 64 if (maxLen === 0) return 1; 65 // Use fuzzball ratio (Levenshtein-based, 0-100) 66 return fuzzball.ratio(al, bl) / 100; 67} 68 69// ── Token Set Ratio (handles word-order differences) ── 70 71export function tokenSetRatio(a, b) { 72 if (!a && !b) return 1; 73 if (!a || !b) return 0; 74 return fuzzball.token_set_ratio(a.toLowerCase(), b.toLowerCase()) / 100; 75} 76 77// ── Exact match (after normalization) ── 78 79export function exactMatch(a, b) { 80 if (!a || !b) return false; 81 return a.toLowerCase().trim() === b.toLowerCase().trim(); 82} 83 84// ── Field normalizers ── 85 86export function normalizeEmail(email) { 87 if (!email || typeof email !== 'string') return ''; 88 let e = email.toLowerCase().trim(); 89 const at = e.indexOf('@'); 90 if (at < 0) return e; 91 92 let local = e.slice(0, at); 93 const domain = e.slice(at + 1); 94 95 // Strip plus addressing: user+tag@domain → user@domain 96 const plus = local.indexOf('+'); 97 if (plus >= 0) local = local.slice(0, plus); 98 99 // Gmail dot trick: j.o.h.n@gmail.com → john@gmail.com 100 if (domain === 'gmail.com' || domain === 'googlemail.com') { 101 local = local.replace(/\./g, ''); 102 } 103 104 return `${local}@${domain}`; 105} 106 107export function normalizePhone(phone) { 108 if (!phone || typeof phone !== 'string') return ''; 109 return phone.replace(/\D/g, ''); 110} 111 112export function extractDomain(value) { 113 if (!value || typeof value !== 'string') return ''; 114 const v = value.toLowerCase().trim(); 115 116 // If it's an email, extract domain 117 const at = v.indexOf('@'); 118 if (at >= 0) return v.slice(at + 1); 119 120 // If it's a URL, extract hostname 121 try { 122 const url = v.startsWith('http') ? v : `https://${v}`; 123 return new URL(url).hostname.replace(/^www\./, ''); 124 } catch { 125 return v.replace(/^www\./, ''); 126 } 127} 128 129// ── Method registry ── 130 131const METHODS = { 132 exact: (a, b) => exactMatch(a, b) ? 1 : 0, 133 'jaro-winkler': jaroWinkler, 134 levenshtein: levenshteinSimilarity, 135 'token-set': tokenSetRatio, 136}; 137 138/** 139 * Score a pair on a single field using the specified method. 140 * Returns a number 0-1. 141 */ 142export function scoreField(valueA, valueB, method) { 143 const fn = METHODS[method]; 144 if (!fn) throw new Error(`Unknown similarity method: ${method}`); 145 return fn(valueA, valueB); 146}

1import { scoreField, normalizeEmail, normalizePhone, extractDomain } from './similarity.js'; 2import { generateCandidatePairs } from './blocking.js'; 3import { UnionFind } from './union-find.js'; 4 5// ── Field auto-detection ── 6 7const FIELD_CONFIG = [ 8 { patterns: ['email'], method: 'exact', weight: 1.0 }, 9 { patterns: ['phone', 'phonenumber', 'telephone'], method: 'exact', weight: 0.9 }, 10 { patterns: ['name', 'fullname'], method: 'jaro-winkler', weight: 0.8 }, 11 { patterns: ['firstname'], method: 'jaro-winkler', weight: 0.5 }, 12 { patterns: ['lastname'], method: 'jaro-winkler', weight: 0.5 }, 13 { patterns: ['company', 'companyname', 'organization'], method: 'token-set', weight: 0.7 }, 14 { patterns: ['domain', 'website', 'url'], method: 'exact', weight: 0.7 }, 15 { patterns: ['address'], method: 'levenshtein', weight: 0.4 }, 16 { patterns: ['city'], method: 'exact', weight: 0.3 }, 17 { patterns: ['title', 'jobtitle'], method: 'token-set', weight: 0.3 }, 18]; 19 20/** 21 * Auto-detect matchable fields from record keys. 22 * Scans first few records and picks fields that match known patterns. 23 */ 24export function autoDetectFields(records) { 25 if (!records.length) return []; 26 27 // Collect all keys from up to 10 records 28 const allKeys = new Set(); 29 for (let i = 0; i < Math.min(10, records.length); i++) { 30 if (records[i] && typeof records[i] === 'object') { 31 for (const key of Object.keys(records[i])) { 32 allKeys.add(key); 33 } 34 } 35 } 36 37 const detected = []; 38 for (const config of FIELD_CONFIG) { 39 for (const key of allKeys) { 40 if (config.patterns.includes(key.toLowerCase())) { 41 detected.push({ 42 field: key, 43 method: config.method, 44 weight: config.weight, 45 }); 46 break; // one match per config entry 47 } 48 } 49 } 50 51 return detected; 52} 53 54// ── Pair scoring ── 55 56/** 57 * Get a field's value from a record (case-insensitive key lookup). 58 */ 59function getVal(record, field) { 60 if (!record || typeof record !== 'object') return ''; 61 if (record[field] !== undefined) return String(record[field] ?? ''); 62 const lower = field.toLowerCase(); 63 for (const key of Object.keys(record)) { 64 if (key.toLowerCase() === lower) return String(record[key] ?? ''); 65 } 66 return ''; 67} 68 69/** 70 * Normalize a field value before comparison based on its method. 71 */ 72function normalizeFieldValue(value, field) { 73 const fl = field.toLowerCase(); 74 if (fl === 'email') return normalizeEmail(value); 75 if (['phone', 'phonenumber', 'telephone'].includes(fl)) return normalizePhone(value); 76 if (['domain', 'website', 'url'].includes(fl)) return extractDomain(value); 77 return value; 78} 79 80/** 81 * Compare two records across all match fields. 82 * Returns { score, reasons, confidence }. 83 */ 84export function comparePair(recordA, recordB, matchFields) { 85 let weightedSum = 0; 86 let totalWeight = 0; 87 const reasons = []; 88 89 for (const { field, method, weight } of matchFields) { 90 let a = getVal(recordA, field); 91 let b = getVal(recordB, field); 92 93 // Skip if either value is empty — only score fields where both records have data 94 if (!a || !b) continue; 95 96 // Normalize before comparison 97 a = normalizeFieldValue(a, field); 98 b = normalizeFieldValue(b, field); 99 100 if (!a || !b) { 101 totalWeight += weight; 102 continue; 103 } 104 105 const fieldScore = scoreField(a, b, method); 106 weightedSum += fieldScore * weight; 107 totalWeight += weight; 108 109 if (fieldScore > 0.5) { 110 reasons.push(`${field}_${method}:${fieldScore.toFixed(2)}`); 111 } 112 if (fieldScore === 1 && method === 'exact') { 113 reasons[reasons.length - 1] = `${field}_exact`; 114 } 115 } 116 117 const score = totalWeight > 0 ? weightedSum / totalWeight : 0; 118 const confidence = Math.round(score * 100) / 100; 119 120 return { score, reasons, confidence }; 121} 122 123// ── Canonical record selection ── 124 125/** 126 * Pick the most complete record in a cluster (most non-null, non-empty fields). 127 */ 128function selectCanonical(records, indices) { 129 let bestIdx = indices[0]; 130 let bestCount = 0; 131 132 for (const idx of indices) { 133 const record = records[idx]; 134 if (!record || typeof record !== 'object') continue; 135 let count = 0; 136 for (const val of Object.values(record)) { 137 if (val !== null && val !== undefined && val !== '') count++; 138 } 139 if (count > bestCount) { 140 bestCount = count; 141 bestIdx = idx; 142 } 143 } 144 145 return bestIdx; 146} 147 148// ── Main dedup pipeline ── 149 150/** 151 * Deduplicate an array of records. 152 * 153 * @param {object[]} records - Array of record objects 154 * @param {object} options 155 * @param {Array<{field, method, weight}>} [options.matchFields] - Fields to match on (auto-detected if omitted) 156 * @param {number} [options.threshold=0.85] - Composite score threshold for dedup 157 * @returns {object[]} Annotated records with cluster info 158 */ 159export function deduplicateRecords(records, options = {}) { 160 const threshold = options.threshold ?? 0.85; 161 const matchFields = options.matchFields?.length ? options.matchFields : autoDetectFields(records); 162 163 if (!matchFields.length) { 164 // No matchable fields found — return records as-is (all unique) 165 return records.map((record, i) => ({ 166 id: i + 1, 167 record, 168 clusterId: `c_${i + 1}`, 169 clusterSize: 1, 170 isCanonical: true, 171 duplicateOf: null, 172 matchScore: null, 173 matchReasons: [], 174 confidence: 1, 175 })); 176 } 177 178 // 1. Generate candidate pairs via multi-pass blocking 179 const candidatePairs = generateCandidatePairs(records, matchFields); 180 181 // 2. Score each candidate pair 182 const uf = new UnionFind(records.length); 183 const pairScores = new Map(); // "i:j" → { score, reasons, confidence } 184 185 for (const pairKey of candidatePairs) { 186 const [i, j] = pairKey.split(':').map(Number); 187 const result = comparePair(records[i], records[j], matchFields); 188 189 if (result.score >= threshold) { 190 uf.union(i, j); 191 pairScores.set(pairKey, result); 192 } 193 } 194 195 // 3. Build clusters 196 const clusters = uf.clusters(); 197 198 // 4. Annotate each record 199 const output = new Array(records.length); 200 let clusterNum = 0; 201 202 for (const [root, members] of clusters) { 203 clusterNum++; 204 const clusterId = `c_${clusterNum}`; 205 const canonicalIdx = selectCanonical(records, members); 206 207 for (const idx of members) { 208 const isCanonical = idx === canonicalIdx; 209 210 // Find match score against canonical (if not canonical itself) 211 let matchScore = null; 212 let matchReasons = []; 213 let confidence = 1; 214 215 if (!isCanonical) { 216 const i = Math.min(idx, canonicalIdx); 217 const j = Math.max(idx, canonicalIdx); 218 const key = `${i}:${j}`; 219 const cached = pairScores.get(key); 220 if (cached) { 221 matchScore = cached.confidence; 222 matchReasons = cached.reasons; 223 confidence = cached.confidence; 224 } else { 225 // Direct comparison against canonical (may not have been a candidate pair) 226 const result = comparePair(records[idx], records[canonicalIdx], matchFields); 227 matchScore = result.confidence; 228 matchReasons = result.reasons; 229 confidence = result.confidence; 230 } 231 } 232 233 output[idx] = { 234 id: idx + 1, 235 record: records[idx], 236 clusterId, 237 clusterSize: members.length, 238 isCanonical, 239 duplicateOf: isCanonical ? null : canonicalIdx + 1, // 1-based 240 matchScore, 241 matchReasons, 242 confidence, 243 }; 244 } 245 } 246 247 return output; 248}

1import { normalizeEmail, normalizePhone, extractDomain } from './similarity.js'; 2 3/** 4 * Multi-pass blocking generates candidate index pairs. 5 * Each pass groups records by a blocking key, then generates pairs within each group. 6 * Returns a Set of "i:j" strings (i < j) — deduplicated across passes. 7 */ 8export function generateCandidatePairs(records, detectedFields) { 9 const pairs = new Set(); 10 const fieldNames = detectedFields.map((f) => f.field); 11 12 // Pass 1: Exact normalized email 13 if (fieldNames.includes('email')) { 14 blockByKey(records, pairs, (r) => { 15 const email = getFieldValue(r, 'email'); 16 return email ? normalizeEmail(email) : null; 17 }); 18 } 19 20 // Pass 2: Name prefix (first 3 chars of last name + first char of first name) 21 if (fieldNames.some((f) => ['name', 'fullName', 'lastName', 'firstName'].includes(f))) { 22 blockByKey(records, pairs, (r) => { 23 const name = getFieldValue(r, 'name') || getFieldValue(r, 'fullName') || ''; 24 const lastName = getFieldValue(r, 'lastName') || ''; 25 const firstName = getFieldValue(r, 'firstName') || ''; 26 27 let key = ''; 28 if (lastName) { 29 key = lastName.toLowerCase().slice(0, 3) + (firstName ? firstName.toLowerCase().charAt(0) : ''); 30 } else if (name) { 31 // Split "First Last" → use last part 32 const parts = name.trim().split(/\s+/); 33 const last = parts.length > 1 ? parts[parts.length - 1] : parts[0]; 34 const first = parts.length > 1 ? parts[0] : ''; 35 key = last.toLowerCase().slice(0, 3) + (first ? first.toLowerCase().charAt(0) : ''); 36 } 37 return key.length >= 3 ? key : null; 38 }); 39 } 40 41 // Pass 3: Phone suffix (last 7 digits) 42 if (fieldNames.some((f) => ['phone', 'phoneNumber', 'telephone'].includes(f))) { 43 blockByKey(records, pairs, (r) => { 44 const phone = getFieldValue(r, 'phone') || getFieldValue(r, 'phoneNumber') || getFieldValue(r, 'telephone') || ''; 45 const digits = normalizePhone(phone); 46 return digits.length >= 7 ? digits.slice(-7) : null; 47 }); 48 } 49 50 // Pass 4: Domain (from email or website) 51 if (fieldNames.some((f) => ['email', 'domain', 'website', 'url'].includes(f))) { 52 blockByKey(records, pairs, (r) => { 53 const email = getFieldValue(r, 'email') || ''; 54 const website = getFieldValue(r, 'domain') || getFieldValue(r, 'website') || getFieldValue(r, 'url') || ''; 55 const domain = extractDomain(email) || extractDomain(website); 56 // Skip very common domains (free email providers) — they don't signal same-entity 57 if (FREE_DOMAINS.has(domain)) return null; 58 return domain || null; 59 }); 60 } 61 62 // Pass 5: Company name prefix (first 4 chars after normalization) 63 if (fieldNames.some((f) => ['company', 'companyName', 'organization'].includes(f))) { 64 blockByKey(records, pairs, (r) => { 65 const company = getFieldValue(r, 'company') || getFieldValue(r, 'companyName') || getFieldValue(r, 'organization') || ''; 66 const normalized = company.toLowerCase().replace(/[^a-z0-9]/g, ''); 67 return normalized.length >= 4 ? normalized.slice(0, 4) : null; 68 }); 69 } 70 71 // Pass 6: Generic exact-value blocking for custom/unknown fields 72 // For any field not covered by the above passes, block by exact lowercase value 73 const knownFields = new Set([ 74 'email', 'phone', 'phonenumber', 'telephone', 75 'name', 'fullname', 'firstname', 'lastname', 76 'company', 'companyname', 'organization', 77 'domain', 'website', 'url', 78 ]); 79 for (const field of fieldNames) { 80 if (!knownFields.has(field.toLowerCase())) { 81 blockByKey(records, pairs, (r) => { 82 const val = getFieldValue(r, field); 83 return val ? val.toLowerCase().trim() : null; 84 }); 85 } 86 } 87 88 return pairs; 89} 90 91/** 92 * Group records by a blocking key, then generate pairs within each group. 93 */ 94function blockByKey(records, pairs, keyFn) { 95 const groups = new Map(); 96 for (let i = 0; i < records.length; i++) { 97 const key = keyFn(records[i]); 98 if (!key) continue; 99 if (!groups.has(key)) groups.set(key, []); 100 groups.get(key).push(i); 101 } 102 103 for (const members of groups.values()) { 104 if (members.length < 2 || members.length > 500) continue; // skip huge blocks (too generic) 105 for (let a = 0; a < members.length; a++) { 106 for (let b = a + 1; b < members.length; b++) { 107 const i = Math.min(members[a], members[b]); 108 const j = Math.max(members[a], members[b]); 109 pairs.add(`${i}:${j}`); 110 } 111 } 112 } 113} 114 115/** 116 * Get a field value from a record, trying common casings. 117 */ 118function getFieldValue(record, field) { 119 if (!record || typeof record !== 'object') return ''; 120 // Try exact field name first 121 if (record[field] !== undefined) return String(record[field] ?? ''); 122 // Try common alternatives 123 const lower = field.toLowerCase(); 124 for (const key of Object.keys(record)) { 125 if (key.toLowerCase() === lower) return String(record[key] ?? ''); 126 } 127 return ''; 128} 129 130const FREE_DOMAINS = new Set([ 131 'gmail.com', 'yahoo.com', 'hotmail.com', 'outlook.com', 'aol.com', 132 'protonmail.com', 'icloud.com', 'mail.com', 'zoho.com', 'yandex.com', 133]);

1import http from 'node:http'; 2import { Actor, log } from 'apify'; 3import { deduplicateRecords, comparePair, autoDetectFields } from './deduper.js'; 4 5// ── Input normalization ── 6 7function normalizeRecords(records) { 8 return records.flatMap((item) => { 9 if (Array.isArray(item)) { 10 log.warning(`Flattening nested array of ${item.length} records`); 11 return item.filter((r) => r && typeof r === 'object' && !Array.isArray(r)); 12 } 13 if (item && typeof item === 'object' && 'input' in item && typeof item.input === 'object') { 14 return item.input; 15 } 16 if (item && typeof item === 'object' && !Array.isArray(item)) return item; 17 return []; 18 }).filter(Boolean); 19} 20 21// ── Standby server ── 22 23function startStandbyServer() { 24 const port = Actor.config.get('containerPort'); 25 26 const server = http.createServer(async (req, res) => { 27 // Readiness probe 28 if (req.headers['x-apify-container-server-readiness-probe']) { 29 res.writeHead(200, { 'Content-Type': 'application/json' }); 30 res.end(JSON.stringify({ status: 'ready' })); 31 return; 32 } 33 34 if (req.method !== 'GET') { 35 res.writeHead(405, { 'Content-Type': 'application/json' }); 36 res.end(JSON.stringify({ error: 'Method not allowed. Use GET.' })); 37 return; 38 } 39 40 const reqUrl = new URL(req.url, `http://${req.headers.host}`); 41 const paramA = reqUrl.searchParams.get('a'); 42 const paramB = reqUrl.searchParams.get('b'); 43 44 // No params → landing page 45 if (!paramA && !paramB) { 46 res.writeHead(200, { 'Content-Type': 'text/html' }); 47 res.end(getLandingPageHtml()); 48 return; 49 } 50 51 // Pair comparison mode 52 if (!paramA || !paramB) { 53 res.writeHead(400, { 'Content-Type': 'application/json' }); 54 res.end(JSON.stringify({ error: 'Both "a" and "b" query parameters required for pair comparison.' })); 55 return; 56 } 57 58 try { 59 const recordA = JSON.parse(paramA); 60 const recordB = JSON.parse(paramB); 61 const threshold = parseFloat(reqUrl.searchParams.get('threshold') || '0.85'); 62 63 const matchFields = autoDetectFields([recordA, recordB]); 64 if (!matchFields.length) { 65 res.writeHead(400, { 'Content-Type': 'application/json' }); 66 res.end(JSON.stringify({ error: 'No matchable fields detected. Records must have fields like email, name, company, phone.' })); 67 return; 68 } 69 70 const result = comparePair(recordA, recordB, matchFields); 71 const response = { 72 isDuplicate: result.score >= threshold, 73 score: result.confidence, 74 reasons: result.reasons, 75 confidence: result.confidence, 76 fieldsUsed: matchFields.map((f) => f.field), 77 }; 78 79 // Push to dataset 80 await Actor.pushData(response); 81 82 res.writeHead(200, { 'Content-Type': 'application/json' }); 83 res.end(JSON.stringify(response)); 84 } catch (err) { 85 const statusCode = err instanceof SyntaxError ? 400 : 500; 86 const message = err instanceof SyntaxError 87 ? 'Invalid JSON in "a" or "b" parameter.' 88 : err.message; 89 res.writeHead(statusCode, { 'Content-Type': 'application/json' }); 90 res.end(JSON.stringify({ error: message })); 91 } 92 }); 93 94 server.listen(port, () => { 95 log.info(`Superclean Dedupe Standby server listening on port ${port}`); 96 }); 97} 98 99// ── Batch mode ── 100 101async function runBatchMode() { 102 const input = await Actor.getInput(); 103 let { records: rawRecords, record, matchFields, threshold } = input || {}; 104 105 // Handle singular "record" shorthand 106 if (record && typeof record === 'object' && !Array.isArray(record)) { 107 if (!Array.isArray(rawRecords)) rawRecords = []; 108 rawRecords = [record, ...rawRecords]; 109 } 110 111 if (!Array.isArray(rawRecords) || rawRecords.length === 0) { 112 await Actor.pushData({ error: 'missing_records', message: 'Input required: provide "records" (array of objects) or "record" (single object). Example: {"records": [{"name": "John", "email": "john@example.com"}]}' }); 113 await Actor.exit('Skipped: Input required: provide "records" (array of objects) or "record" (single object). Example: {"records": [{"name": "John", "email": "john@example.com"}]}'); 114 return; 115 } 116 117 const records = normalizeRecords(rawRecords); 118 119 log.info('Dedupe input:', { 120 recordCount: records.length, 121 matchFields: matchFields || 'auto-detect', 122 threshold: threshold ?? 0.85, 123 }); 124 125 if (records.length < 2) { 126 log.info('Only 1 record provided — nothing to deduplicate.'); 127 const result = { 128 id: 1, 129 record: records[0], 130 clusterId: 'c_1', 131 clusterSize: 1, 132 isCanonical: true, 133 duplicateOf: null, 134 matchScore: null, 135 matchReasons: [], 136 confidence: 1, 137 }; 138 await Actor.pushData(result); 139 return; 140 } 141 142 // Run dedup 143 const results = deduplicateRecords(records, { 144 matchFields, 145 threshold: threshold ?? 0.85, 146 }); 147 148 // Stats 149 const clusters = new Set(results.map((r) => r.clusterId)); 150 const duplicates = results.filter((r) => !r.isCanonical).length; 151 152 log.info(`Dedup complete: ${records.length} records → ${clusters.size} unique (${duplicates} duplicates found)`); 153 154 await Actor.pushData(results); 155 log.info(`Pushed ${results.length} results to dataset`); 156 157} 158 159// ── Entry point ── 160 161await Actor.init(); 162 163if (Actor.config.get('metaOrigin') === 'STANDBY') { 164 startStandbyServer(); 165} else { 166 await runBatchMode(); 167 await Actor.exit(); 168} 169 170// ── Landing page HTML ── 171 172function getLandingPageHtml() { 173 return `<!DOCTYPE html> 174<html lang="en"> 175<head> 176<meta charset="UTF-8"> 177<meta name="viewport" content="width=device-width, initial-scale=1.0"> 178<title>Superclean Dedupe — API</title> 179<style> 180:root{--color-bg:#0a0e14;--color-surface:#12171f;--color-border:#1e2632;--color-text:#e8eaed;--color-text-muted:#8b95a5;--color-accent:#8b5cf6;--color-accent-dim:rgba(139,92,246,0.15);--font-display:'Space Mono',monospace;--font-body:'DM Sans',sans-serif} 181*{margin:0;padding:0;box-sizing:border-box} 182body{font-family:var(--font-body);background:var(--color-bg);color:var(--color-text);line-height:1.6;min-height:100vh} 183.container{max-width:720px;margin:0 auto;padding:48px 24px} 184.logo{font-family:var(--font-display);font-size:13px;font-weight:700;letter-spacing:0.1em;text-transform:uppercase;color:var(--color-text-muted);margin-bottom:48px} 185h1{font-family:var(--font-display);font-size:28px;font-weight:700;margin-bottom:12px;letter-spacing:-0.02em} 186h1 .hl{color:var(--color-accent)} 187.subtitle{color:var(--color-text-muted);font-size:16px;max-width:520px;margin-bottom:32px;line-height:1.7} 188.link{color:var(--color-accent);text-decoration:none} 189.link:hover{text-decoration:underline} 190h2{font-family:var(--font-display);font-size:16px;font-weight:700;margin:48px 0 16px;padding-top:24px;border-top:1px solid var(--color-border)} 191p{color:var(--color-text-muted);margin-bottom:16px;font-size:14px;line-height:1.7} 192pre{background:var(--color-surface);border:1px solid var(--color-border);padding:16px;overflow-x:auto;font-family:var(--font-display);font-size:12px;line-height:1.6;margin:12px 0;color:var(--color-text)} 193code{font-family:var(--font-display);font-size:12px} 194.comment{color:var(--color-text-muted)} 195table{width:100%;border-collapse:collapse;margin:12px 0;font-size:13px} 196th{font-family:var(--font-display);font-size:10px;font-weight:700;letter-spacing:0.1em;text-transform:uppercase;color:var(--color-text-muted);text-align:left;padding:10px 12px;border-bottom:1px solid var(--color-border)} 197td{padding:8px 12px;border-bottom:1px solid var(--color-border);color:var(--color-text-muted)} 198td:first-child{color:var(--color-text);font-family:var(--font-display);font-size:12px} 199.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(200px,1fr));gap:12px;margin-top:16px} 200.card{background:var(--color-surface);border:1px solid var(--color-border);padding:16px;text-decoration:none;color:inherit;display:block;transition:border-color 0.2s} 201.card:hover{border-color:var(--color-accent)} 202.card-name{font-family:var(--font-display);font-size:12px;font-weight:700;margin-bottom:6px} 203.card-desc{font-size:12px;color:var(--color-text-muted);line-height:1.5} 204footer{margin-top:64px;padding-top:24px;border-top:1px solid var(--color-border);font-size:12px;color:var(--color-text-muted)} 205</style> 206<link rel="preconnect" href="https://fonts.googleapis.com"> 207<link href="https://fonts.googleapis.com/css2?family=DM+Sans:wght@400;700&family=Space+Mono:wght@400;700&display=swap" rel="stylesheet"> 208</head> 209<body> 210<div class="container"> 211<div class="logo">Superlative</div> 212<h1>Superclean <span class="hl">Dedupe</span></h1> 213<p class="subtitle">Deduplicate lead records using fuzzy matching. Compare two records instantly via this API, or run batch dedup on thousands of records via <a class="link" href="https://apify.com/superlativetech/superclean-dedupe?fpr=8e9l1">the Apify actor</a>.</p> 214 215<h2>Pair Comparison</h2> 216<p>Compare two records to check if they're duplicates:</p> 217<pre><span class="comment"># Are these the same person?</span> 218curl "https://superlativetech--superclean-dedupe.apify.actor?token=TOKEN\\ 219&a={\\"name\\":\\"Jim Smith\\",\\"email\\":\\"jim@acme.com\\"}\\ 220&b={\\"name\\":\\"James Smith\\",\\"email\\":\\"jim@acme.com\\"}" 221 222<span class="comment"># Response</span> 223{ 224 "isDuplicate": true, 225 "score": 0.92, 226 "reasons": ["email_exact", "name_jaro-winkler:0.87"], 227 "confidence": 0.92, 228 "fieldsUsed": ["email", "name"] 229}</pre> 230 231<h2>Query Parameters</h2> 232<table> 233<tr><th>Parameter</th><th>Required</th><th>Description</th></tr> 234<tr><td>a</td><td>Yes</td><td>First record as JSON object</td></tr> 235<tr><td>b</td><td>Yes</td><td>Second record as JSON object</td></tr> 236<tr><td>threshold</td><td>No</td><td>Similarity threshold (default 0.85)</td></tr> 237</table> 238 239<h2>Supported Fields</h2> 240<p>Fields are auto-detected from your records:</p> 241<table> 242<tr><th>Field</th><th>Method</th><th>Weight</th></tr> 243<tr><td>email</td><td>Exact (normalized)</td><td>1.0</td></tr> 244<tr><td>phone</td><td>Exact (digits)</td><td>0.9</td></tr> 245<tr><td>name / fullName</td><td>Jaro-Winkler</td><td>0.8</td></tr> 246<tr><td>company</td><td>Token Set Ratio</td><td>0.7</td></tr> 247<tr><td>domain / website</td><td>Exact (extracted)</td><td>0.7</td></tr> 248<tr><td>title / jobTitle</td><td>Token Set Ratio</td><td>0.3</td></tr> 249</table> 250 251<h2>Batch Dedup</h2> 252<p>For deduplicating hundreds or thousands of records, use the <a class="link" href="https://apify.com/superlativetech/superclean-dedupe?fpr=8e9l1">full actor</a> with batch input.</p> 253 254<h2>Pricing</h2> 255<p>Free to use — you only pay standard Apify platform usage.</p> 256 257<h2>More from Superlative</h2> 258<div class="cards"> 259<a class="card" href="https://apify.com/superlativetech/superclean-company-names?fpr=8e9l1"><div class="card-name">Company Names</div><div class="card-desc">Normalize company names from CRM exports</div></a> 260<a class="card" href="https://apify.com/superlativetech/superclean-emails?fpr=8e9l1"><div class="card-name">Emails</div><div class="card-desc">Validate emails, fix typos, check MX records</div></a> 261<a class="card" href="https://apify.com/superlativetech/superclean-person-names?fpr=8e9l1"><div class="card-name">Person Names</div><div class="card-desc">Clean person names for cold email</div></a> 262<a class="card" href="https://apify.com/superlativetech/superlead-icp-scorer?fpr=8e9l1"><div class="card-name">ICP Scorer</div><div class="card-desc">Score leads against your Ideal Customer Profile</div></a> 263<a class="card" href="https://apify.com/superlativetech/supernet-domain-health?fpr=8e9l1"><div class="card-name">Domain Health</div><div class="card-desc">Audit email sending domains (SPF, DKIM, DMARC)</div></a> 264<a class="card" href="https://apify.com/superlativetech/superclean-phone-numbers?fpr=8e9l1"><div class="card-name">Phone Numbers</div><div class="card-desc">Format and validate phone numbers</div></a> 265</div> 266 267<footer>© 2026 Superlative · <a class="link" href="https://superlative.tech">superlative.tech</a></footer> 268</div> 269</body> 270</html>`; 271}