CRM Deduplication Tool

Detects and merges duplicate contacts in CRM databases using advanced fuzzy matching algorithms.

What does CRM Deduplication Tool do?

CRM Deduplication Tool is a powerful serverless actor that identifies and merges duplicate contacts in any CRM database. Simply provide a list of contacts (from HubSpot, Salesforce, Pipedrive, or any other CRM), and the Actor uses advanced fuzzy matching algorithms to detect duplicates across email, name, phone, and company fields. It returns a complete report with confidence scores for each match and a clean, deduplicated list ready for re-import.

Built for RevOps teams, sales managers, and marketing operations professionals who need to clean their CRM databases quickly without expensive monthly subscriptions.

Why use CRM Deduplication Tool?

• Universal CRM compatibility - Works with any CRM that exports to JSON or CSV
• Advanced fuzzy matching - Detects duplicates even with typos, formatting differences, and variations
• Confidence scoring - Every match includes a confidence score so you can review before merging
• Visual HTML report - Get a beautiful HTML report showing all duplicates found
• Pay-per-use pricing - No monthly subscription; pay only for what you use
• 100% data privacy - Your data never leaves your control; zero external requests

How does it work?

The deduplication process works in 5 phases:

1. Normalization - Each field is normalized (emails lowercased, phone numbers stripped of formatting, diacritics removed from names)
2. Exact Email Matching - Contacts with identical emails are immediately flagged as definite duplicates
3. Fuzzy Name Matching - Names are compared using Jaro-Winkler similarity and token-based matching
4. Phone Matching - Phone numbers are normalized and compared
5. Company Matching - Company names are normalized and checked for fuzzy matches

The results are then clustered using Union-Find algorithm to group related duplicates together.

Supported matching fields

Field	Matching Method	Confidence
Email	Exact match (after normalization) + typo detection	85-100
Name	Jaro-Winkler + token sort + initials	65-100
Phone	Normalized exact match + 1-digit typo	60-90
Company	Jaro-Winkler + substring	75-90

Input

Provide contacts either as an array or reference an Apify dataset:

{
    "contacts": [
        { "email": "john@example.com", "name": "John Smith", "phone": "+1 555 123-4567", "company": "Acme Corp" },
        { "email": "JOHN@EXAMPLE.COM", "name": "John Smith", "phone": "15551234567", "company": "Acme Corporation" }
    ],
    "confidenceThreshold": 70,
    "matchingFields": ["email", "name", "phone", "company"],
    "outputMode": "full",
    "mergeStrategy": "most-complete"
}

Input fields

Field	Type	Required	Description
contacts	array	Yes*	Array of contact objects
datasetId	string	Yes*	Apify dataset ID to fetch contacts from
fieldMapping	object	No	Map your fields to email/name/phone/company
matchingFields	array	No	Fields to use for matching (default: all 4)
confidenceThreshold	number	No	Min score (50-100) to consider duplicate (default: 70)
outputMode	string	No	"full", "duplicates-only", or "clean-list"
mergeStrategy	string	No	"most-complete", "first", or "last"

*Either contacts or datasetId is required.

Output

The Actor outputs a JSON dataset with:

{
  "deduplicationId": "uuid",
  "processedAt": "2024-01-15T10:30:00Z",
  "inputSummary": {
    "totalContactsReceived": 1500,
    "fieldsUsedForMatching": ["email", "name", "phone", "company"],
    "confidenceThreshold": 70
  },
  "summary": {
    "duplicateGroupsFound": 47,
    "totalDuplicateContacts": 112,
    "uniqueContactsAfterDedup": 1388,
    "duplicateRate": 7.47,
    "estimatedTimeSavedMinutes": 56
  },
  "duplicateGroups": [...],
  "cleanList": [...],
  "processingStats": {...}
}

Confidence levels

• definite (score ≥ 90): Almost certainly the same person
• likely (score 70-89): Probably the same person
• possible (score 50-69): Could be the same person, requires manual review

Use cases

1. HubSpot contact cleanup - Remove duplicates accumulated from form submissions
2. Salesforce dedup before migration - Clean data before migrating from legacy systems
3. Pipedrive list deduplication - Merge contacts from multiple pipelines
4. Marketing event attendee merge - Combine attendee lists from multiple events
5. Lead list validation - Verify new leads against existing database before insertion
6. CRM audit preparation - Generate duplicate reports for quarterly reviews

Performance

Contacts	Estimated Time
100	< 1 second
1,000	< 5 seconds
10,000	< 60 seconds
50,000	< 10 minutes

Limitations

• Maximum 50,000 contacts per run
• Requires at least 2 contacts with email or name
• Currently does not directly merge in CRM (exports clean list only)

Roadmap

v1.1 (Planned)

• CSV input support (paste CSV as string)
• Better field auto-detection
• Notes field explaining why duplicates were matched

v2.0 (Planned)

• Direct HubSpot integration (merge in CRM)
• Direct Salesforce integration
• Direct Pipedrive integration
• Incremental mode (check new contacts against existing database)
• CSV export for CRM re-import

Pricing

This Actor uses pay-per-use pricing. You only pay for the compute time used:

• Approximately $0.001 per 100 contacts processed
• No monthly subscription required

Compare to Dedupely ($49-299/month), Insycle ($99/month), or Duplicate Check for Salesforce ($50/month).

Getting started

1. Click Run in Apify Console
2. Paste your contacts as JSON or provide a dataset ID
3. Adjust confidence threshold if needed
4. Click Start

The Actor will process your contacts and generate both a JSON dataset report and an HTML visual report.