Dataset Deduplicator & Merger

Merge multiple datasets into one clean dataset — remove duplicates, unify field names, and keep the most complete version of every record.

Built for the most common post-scraping problem: you ran Google Maps, Yelp, and a directory scraper, and now you have three overlapping lists with different column names and thousands of duplicate businesses. This Actor turns them into a single deduplicated, backfilled dataset in one run.

What it does

• Merges any number of datasets — pass a list of datasets, get one combined output.
• Three matching modes:
  • exact — byte-for-byte comparison.
  • normalized (recommended) — case-insensitive, ignores punctuation and accents, strips common company suffixes (LLC, Inc, Ltd...), and compares phone numbers digits-only, so +1 (555) 123-4567 matches 5551234567.
  • fuzzy — catches near-duplicates like Joe's Pizza vs Joes Pizza LLC using a similarity threshold you control.
• Smart "most complete" keep strategy — instead of blindly keeping the first duplicate, keeps the record with the most filled-in fields and back-fills its empty fields from the discarded duplicates. If one source had the email and the other had the phone number, the surviving record gets both.
• Field mapping — rename columns on the fly so businessName and company_name become one name column before merging.
• Handles big datasets — streams input in batches; fuzzy mode supports blocking (e.g. only compare rows within the same zip code) so 100k-row jobs stay fast.

Input example

{
    "datasetIds": ["abc123", "def456", "ghi789"],
    "dedupeFields": ["name", "address"],
    "matchMode": "fuzzy",
    "fuzzyThreshold": 85,
    "blockingField": "zipCode",
    "fieldMapping": { "businessName": "name", "company_name": "name" },
    "keepStrategy": "mostComplete"
}

Output

One clean dataset, plus a SUMMARY record in the key-value store:

{
    "datasetsMerged": 3,
    "rowsIn": 48210,
    "duplicatesRemoved": 9384,
    "rowsOut": 38826,
    "rowsWithoutDedupeKey": 112,
    "matchMode": "fuzzy",
    "keepStrategy": "mostComplete"
}

Common use cases

• Lead lists — dedupe by email or phone across scraped sources before importing to your CRM.
• Local business data — merge Google Maps + Yelp + directory results; fuzzy match on name + address, block by zipCode.
• E-commerce — dedupe product listings by normalized title or SKU.
• Job postings — collapse the same posting syndicated across multiple boards.

Tips

• Start with normalized mode. Switch to fuzzy only if you can see near-duplicates surviving.
• In fuzzy mode on large datasets, always set blockingField (zip, city, domain...). It dramatically speeds up the run without hurting accuracy.
• Dot notation works everywhere fields are referenced: contact.email.
• Rows where all dedupe fields are empty are never merged with each other — they pass through untouched and are counted in rowsWithoutDedupeKey.

Pricing

Pay-per-result: billed per row in your cleaned output dataset. Duplicates removed along the way are processed free — you only pay for the clean rows you keep. Cleaning a 50,000-row lead list down to 38,000 unique records costs about $19 — a rounding error compared to the value of the list.