Dataset Deduplicator & Cleaner

🔗 Part of the Apify actors collection — 6 reliable, tested scrapers & data tools.

Quick start: paste a JSON array into Inline data (or give a Source dataset ID), then Run.

Turn messy, duplicate-ridden data into analysis-ready records in one deterministic run. Point it at any Apify dataset (or paste JSON directly), and get back deduplicated, cleaned, type-consistent records — plus a data-quality report. No scraping, no external APIs, no maintenance.

---

✨ What it does

• Deduplicates records — exact, normalized, or fuzzy (near-duplicate) matching.
• Cleans fields by type — emails, phones, URLs, whitespace, HTML, empty values, and more.
• Reports quality — input/output counts, duplicates removed, and per-field fill rates, saved to the key-value store as QUALITY_REPORT.
• Two ways in — pipe another actor's output via Source dataset ID, or paste Inline data.
• Zero maintenance — deterministic, no third-party services, pay only per output record.

🎯 Who is it for

Role	What it solves
Scraper users	Clean and de-dupe an actor's output before loading it into a DB or spreadsheet.
Data / ops teams	Standardize messy exports (leads, catalogs, contacts) without writing scripts.
Analysts	De-dupe and normalize lists, then check fill rates at a glance.

---

⚙️ Deduplication modes (dedupMode)

Mode	Description
none	Don't deduplicate — clean only.
exact	Byte-identical values.
normalized	Ignores case, whitespace and punctuation — catches "John Smith" vs "john smith!". (default)
fuzzy	Also merges near-duplicates by string similarity (similarityThreshold, 0.5–0.99). Best for a few thousand rows; on very large or low-diversity inputs it automatically degrades to normalized and tells you in the run status (so a run never hangs).

Set dedupKeys to the fields that identify a duplicate (e.g. email, url); leave it empty to compare the whole record. When duplicates are found, keepStrategy decides which one survives:

• most_complete — keep the record with the most filled fields (default)
• first / last — keep the first or last occurrence

---

🧹 Field cleaning

Fields are cleaned by type. With smartFields on (default), emails, phones and URLs are detected by field name; everything else is treated as text.

Option	What it does
cleanEmails	Lowercase + trim values that look like a valid email (invalid ones are left as text, not dropped).
cleanPhones	Compact phone-like fields to digits, keeping a leading +/00 country prefix.
cleanUrls	Lowercase the scheme and host, drop a trailing slash.
stripHtml	Remove HTML tags and unescape entities (& → &).
emptyToNull	Turn empty / blank values into null for consistency. (default on)
dropEmptyFields	Remove empty fields entirely instead of keeping them as null.
case	Force plain-text fields to lower / upper / title case.
coerceTypes	Convert numeric and true/false strings into real numbers and booleans.

Whitespace collapse, end-trimming and control-character stripping are always applied to text.

---

📥 Input

Provide either sourceDatasetId or data:

{
  "sourceDatasetId": "abc123XYZ",
  "dedupMode": "normalized",
  "dedupKeys": ["email"],
  "keepStrategy": "most_complete",
  "cleanEmails": true,
  "cleanPhones": true,
  "cleanUrls": true,
  "emptyToNull": true,
  "includeReport": true,
  "maxItems": 0
}

Or clean records inline:

{
  "data": [
    { "name": "  Acme Corp  ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/", "notes": "<p>Top client</p>" },
    { "name": "Acme Corp",     "email": "sales@acme.com", "phone": "+15551234567",     "website": "http://acme.com",  "notes": "Top client" },
    { "name": "Globex Inc",    "email": "info@globex.io", "phone": "",                  "website": "https://globex.io/", "notes": "" }
  ],
  "dedupMode": "normalized",
  "dedupKeys": ["name"],
  "stripHtml": true
}

Output (deduplicated on name, HTML stripped, empty → null):

[
  { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com", "notes": "Top client" },
  { "name": "Globex Inc", "email": "info@globex.io", "phone": null, "website": "https://globex.io", "notes": null }
]

---

📊 Quality report

When includeReport is on, a report is saved to the key-value store under QUALITY_REPORT:

{
  "inputRecords": 3,
  "outputRecords": 2,
  "duplicatesRemoved": 1,
  "dedupMode": "normalized",
  "fieldValuesCleaned": 7,
  "fieldCount": 5,
  "fields": {
    "email":   { "filled": 2, "empty": 0, "fillRate": 1.0 },
    "phone":   { "filled": 1, "empty": 1, "fillRate": 0.5 },
    "website": { "filled": 2, "empty": 0, "fillRate": 1.0 },
    "notes":   { "filled": 1, "empty": 1, "fillRate": 0.5 }
  }
}

---

🚀 How to use

1. Open the actor in Apify Console (or call it via API).
2. Pick a data source — a Source dataset ID from a previous run, or paste a JSON array into Inline data.
3. Adjust dedup and cleaning if you like — the defaults (normalized dedup + sensible cleaning) work out of the box.
4. Run. Clean records land in the default dataset; the quality report goes to the key-value store as QUALITY_REPORT.
5. Export as JSON / CSV / Excel, or push onward with Apify integrations.

---

💰 Pricing & reliability

• Pay per result — you're billed only for the clean records written to the dataset.
• Deterministic — same input and config always produce the same output.
• No external dependencies — no third-party APIs, no browser, nothing to break.
• Handles up to 50,000 records per run (exact/normalized). Fuzzy is meant for a few thousand rows and degrades gracefully above that. Larger source datasets are paginated in full, and if the 50,000 cap truncates the input the run says so in its status and quality report.

License

Licensed under the Apache License 2.0.