# CRM Deduplication Tool (`enosgb/crm-deduplication-tool`) Actor Detects and merges duplicate contacts in CRM databases using advanced fuzzy matching algorithms - **URL**: https://apify.com/enosgb/crm-deduplication-tool.md - **Developed by:** [Enos Melo](https://apify.com/enosgb) (community) - **Categories:** AI, Lead generation, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 processed contacts This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## CRM Deduplication Tool Detects and merges duplicate contacts in CRM databases using advanced fuzzy matching algorithms. ### What does CRM Deduplication Tool do? CRM Deduplication Tool is a powerful serverless actor that identifies and merges duplicate contacts in any CRM database. Simply provide a list of contacts (from HubSpot, Salesforce, Pipedrive, or any other CRM), and the Actor uses advanced fuzzy matching algorithms to detect duplicates across email, name, phone, and company fields. It returns a complete report with confidence scores for each match and a clean, deduplicated list ready for re-import. Built for RevOps teams, sales managers, and marketing operations professionals who need to clean their CRM databases quickly without expensive monthly subscriptions. ### Why use CRM Deduplication Tool? - **Universal CRM compatibility** - Works with any CRM that exports to JSON or CSV - **Advanced fuzzy matching** - Detects duplicates even with typos, formatting differences, and variations - **Confidence scoring** - Every match includes a confidence score so you can review before merging - **Visual HTML report** - Get a beautiful HTML report showing all duplicates found - **Pay-per-use pricing** - No monthly subscription; pay only for what you use - **100% data privacy** - Your data never leaves your control; zero external requests ### How does it work? The deduplication process works in 5 phases: 1. **Normalization** - Each field is normalized (emails lowercased, phone numbers stripped of formatting, diacritics removed from names) 2. **Exact Email Matching** - Contacts with identical emails are immediately flagged as definite duplicates 3. **Fuzzy Name Matching** - Names are compared using Jaro-Winkler similarity and token-based matching 4. **Phone Matching** - Phone numbers are normalized and compared 5. **Company Matching** - Company names are normalized and checked for fuzzy matches The results are then clustered using Union-Find algorithm to group related duplicates together. ### Supported matching fields | Field | Matching Method | Confidence | | ------- | -------------------------------------------------- | ---------- | | Email | Exact match (after normalization) + typo detection | 85-100 | | Name | Jaro-Winkler + token sort + initials | 65-100 | | Phone | Normalized exact match + 1-digit typo | 60-90 | | Company | Jaro-Winkler + substring | 75-90 | ### Input Provide contacts either as an array or reference an Apify dataset: ```json { "contacts": [ { "email": "john@example.com", "name": "John Smith", "phone": "+1 555 123-4567", "company": "Acme Corp" }, { "email": "JOHN@EXAMPLE.COM", "name": "John Smith", "phone": "15551234567", "company": "Acme Corporation" } ], "confidenceThreshold": 70, "matchingFields": ["email", "name", "phone", "company"], "outputMode": "full", "mergeStrategy": "most-complete" } ```` #### Input fields | Field | Type | Required | Description | | ------------------- | ------ | -------- | ------------------------------------------------------ | | contacts | array | Yes\* | Array of contact objects | | datasetId | string | Yes\* | Apify dataset ID to fetch contacts from | | fieldMapping | object | No | Map your fields to email/name/phone/company | | matchingFields | array | No | Fields to use for matching (default: all 4) | | confidenceThreshold | number | No | Min score (50-100) to consider duplicate (default: 70) | | outputMode | string | No | "full", "duplicates-only", or "clean-list" | | mergeStrategy | string | No | "most-complete", "first", or "last" | \*Either contacts or datasetId is required. ### Output The Actor outputs a JSON dataset with: ```json { "deduplicationId": "uuid", "processedAt": "2024-01-15T10:30:00Z", "inputSummary": { "totalContactsReceived": 1500, "fieldsUsedForMatching": ["email", "name", "phone", "company"], "confidenceThreshold": 70 }, "summary": { "duplicateGroupsFound": 47, "totalDuplicateContacts": 112, "uniqueContactsAfterDedup": 1388, "duplicateRate": 7.47, "estimatedTimeSavedMinutes": 56 }, "duplicateGroups": [...], "cleanList": [...], "processingStats": {...} } ``` #### Confidence levels - **definite** (score ≥ 90): Almost certainly the same person - **likely** (score 70-89): Probably the same person - **possible** (score 50-69): Could be the same person, requires manual review ### Use cases 1. **HubSpot contact cleanup** - Remove duplicates accumulated from form submissions 2. **Salesforce dedup before migration** - Clean data before migrating from legacy systems 3. **Pipedrive list deduplication** - Merge contacts from multiple pipelines 4. **Marketing event attendee merge** - Combine attendee lists from multiple events 5. **Lead list validation** - Verify new leads against existing database before insertion 6. **CRM audit preparation** - Generate duplicate reports for quarterly reviews ### Performance | Contacts | Estimated Time | | -------- | -------------- | | 100 | < 1 second | | 1,000 | < 5 seconds | | 10,000 | < 60 seconds | | 50,000 | < 10 minutes | ### Limitations - Maximum 50,000 contacts per run - Requires at least 2 contacts with email or name - Currently does not directly merge in CRM (exports clean list only) ### Roadmap #### v1.1 (Planned) - CSV input support (paste CSV as string) - Better field auto-detection - Notes field explaining why duplicates were matched #### v2.0 (Planned) - Direct HubSpot integration (merge in CRM) - Direct Salesforce integration - Direct Pipedrive integration - Incremental mode (check new contacts against existing database) - CSV export for CRM re-import ### Pricing This Actor uses pay-per-use pricing. You only pay for the compute time used: - Approximately $0.001 per 100 contacts processed - No monthly subscription required Compare to Dedupely ($49-299/month), Insycle ($99/month), or Duplicate Check for Salesforce ($50/month). ### Getting started 1. Click **Run** in Apify Console 2. Paste your contacts as JSON or provide a dataset ID 3. Adjust confidence threshold if needed 4. Click **Start** The Actor will process your contacts and generate both a JSON dataset report and an HTML visual report. # Actor input Schema ## `contacts` (type: `array`): Array of contact objects to deduplicate. Provide either this field or datasetId. ## `datasetId` (type: `string`): Apify dataset ID to fetch contacts from. Use instead of contacts array. ## `fieldMapping` (type: `object`): Map your dataset fields to standard contact fields (email, name, phone, company) ## `matchingFields` (type: `array`): Fields to use for duplicate detection ## `confidenceThreshold` (type: `integer`): Minimum score (0-100) to consider two contacts as duplicates ## `outputMode` (type: `string`): What data to include in output ## `mergeStrategy` (type: `string`): How to select the master record when duplicates are found ## Actor input object example ```json { "contacts": [ { "email": "john@example.com", "name": "John Smith", "phone": "+1 555 123-4567", "company": "Acme Corp" }, { "email": "john@example.com", "name": "John SMITH", "phone": "15551234567", "company": "Acme Corp" }, { "email": "jane@other.com", "name": "Jane Doe", "phone": "+1 555 987-6543", "company": "Beta Inc" }, { "email": "jane@other.com", "name": "J. Doe", "phone": "15559876543", "company": "Beta Inc" }, { "email": "bob@company.com", "name": "Robert Johnson", "phone": "+1 555 111-2222", "company": "Gamma LLC" } ], "datasetId": "", "fieldMapping": { "emailField": "email", "nameField": "name", "phoneField": "phone", "companyField": "company" }, "matchingFields": [ "email", "name", "phone", "company" ], "confidenceThreshold": 70, "outputMode": "full", "mergeStrategy": "most-complete" } ``` # Actor output Schema ## `results` (type: `string`): Full deduplication report with duplicate groups, statistics, and clean list ## `htmlReport` (type: `string`): HTML visual report showing duplicate analysis # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "contacts": [ { "email": "john@example.com", "name": "John Smith", "phone": "+1 555 123-4567", "company": "Acme Corp" }, { "email": "john@example.com", "name": "John SMITH", "phone": "15551234567", "company": "Acme Corp" }, { "email": "jane@other.com", "name": "Jane Doe", "phone": "+1 555 987-6543", "company": "Beta Inc" }, { "email": "jane@other.com", "name": "J. Doe", "phone": "15559876543", "company": "Beta Inc" }, { "email": "bob@company.com", "name": "Robert Johnson", "phone": "+1 555 111-2222", "company": "Gamma LLC" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("enosgb/crm-deduplication-tool").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "contacts": [ { "email": "john@example.com", "name": "John Smith", "phone": "+1 555 123-4567", "company": "Acme Corp", }, { "email": "john@example.com", "name": "John SMITH", "phone": "15551234567", "company": "Acme Corp", }, { "email": "jane@other.com", "name": "Jane Doe", "phone": "+1 555 987-6543", "company": "Beta Inc", }, { "email": "jane@other.com", "name": "J. Doe", "phone": "15559876543", "company": "Beta Inc", }, { "email": "bob@company.com", "name": "Robert Johnson", "phone": "+1 555 111-2222", "company": "Gamma LLC", }, ] } # Run the Actor and wait for it to finish run = client.actor("enosgb/crm-deduplication-tool").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "contacts": [ { "email": "john@example.com", "name": "John Smith", "phone": "+1 555 123-4567", "company": "Acme Corp" }, { "email": "john@example.com", "name": "John SMITH", "phone": "15551234567", "company": "Acme Corp" }, { "email": "jane@other.com", "name": "Jane Doe", "phone": "+1 555 987-6543", "company": "Beta Inc" }, { "email": "jane@other.com", "name": "J. Doe", "phone": "15559876543", "company": "Beta Inc" }, { "email": "bob@company.com", "name": "Robert Johnson", "phone": "+1 555 111-2222", "company": "Gamma LLC" } ] }' | apify call enosgb/crm-deduplication-tool --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=enosgb/crm-deduplication-tool", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "CRM Deduplication Tool", "description": "Detects and merges duplicate contacts in CRM databases using advanced fuzzy matching algorithms", "version": "1.0", "x-build-id": "ksdoyt3ZKelykS13n" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/enosgb~crm-deduplication-tool/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-enosgb-crm-deduplication-tool", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/enosgb~crm-deduplication-tool/runs": { "post": { "operationId": "runs-sync-enosgb-crm-deduplication-tool", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/enosgb~crm-deduplication-tool/run-sync": { "post": { "operationId": "run-sync-enosgb-crm-deduplication-tool", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "contacts": { "title": "Contacts", "type": "array", "description": "Array of contact objects to deduplicate. Provide either this field or datasetId.", "default": [ { "email": "john@example.com", "name": "John Smith", "phone": "+1 555 123-4567", "company": "Acme Corp" }, { "email": "john@example.com", "name": "John SMITH", "phone": "15551234567", "company": "Acme Corp" }, { "email": "jane@other.com", "name": "Jane Doe", "phone": "+1 555 987-6543", "company": "Beta Inc" }, { "email": "jane@other.com", "name": "J. Doe", "phone": "15559876543", "company": "Beta Inc" }, { "email": "bob@company.com", "name": "Robert Johnson", "phone": "+1 555 111-2222", "company": "Gamma LLC" } ] }, "datasetId": { "title": "Dataset ID", "type": "string", "description": "Apify dataset ID to fetch contacts from. Use instead of contacts array.", "default": "" }, "fieldMapping": { "title": "Field Mapping", "type": "object", "description": "Map your dataset fields to standard contact fields (email, name, phone, company)", "default": { "emailField": "email", "nameField": "name", "phoneField": "phone", "companyField": "company" } }, "matchingFields": { "title": "Matching Fields", "type": "array", "description": "Fields to use for duplicate detection", "default": [ "email", "name", "phone", "company" ] }, "confidenceThreshold": { "title": "Confidence Threshold", "minimum": 50, "maximum": 100, "type": "integer", "description": "Minimum score (0-100) to consider two contacts as duplicates", "default": 70 }, "outputMode": { "title": "Output Mode", "enum": [ "full", "duplicates-only", "clean-list" ], "type": "string", "description": "What data to include in output", "default": "full" }, "mergeStrategy": { "title": "Merge Strategy", "enum": [ "most-complete", "first", "last" ], "type": "string", "description": "How to select the master record when duplicates are found", "default": "most-complete" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```