Data Cleaning & Deduplication

A reusable, data-only Actor that turns a messy record dump into a clean, deduplicated, profiled dataset. It runs entirely on the records you give it — no scraping, no proxies — so it is the natural post-processing step after any scraper in your pipeline.

What does this Actor do?

It runs a six-stage pipeline over your input records:

1. Column normalization — unifies key casing and whitespace into canonical column names (so Company, company and company merge).
2. Field & type normalization — per-field normalization for emails, phones, company names, product names, addresses and URLs; invalid emails/phones/URLs are flagged.
3. Empty-value handling — blank / N/A / null-like values become real null; optionally drop columns that are empty across every row.
4. Duplicate detection & removal — key-based (business or product) or fuzzy name similarity.
5. Invalid-record separation — all-empty rows, or rows with an invalid normalized email/phone/URL, are set aside.
6. Dataset profiling — per-column fill rate, inferred type and top values, plus an overall data-quality summary.

Why use it?

• Merge and de-duplicate leads or product feeds collected from several sources into one clean list.
• Standardise emails and phone numbers before importing into a CRM.
• Get a quick data-quality report (fill rates, duplicate counts, invalid rows) for any dataset.

How to use it

1. Provide records inline (records), as csv / json text, or point at an Apify dataset (datasetId) or a public datasetUrl.
2. Choose a dedupeStrategy, optionally enable fuzzyDedupe, and map fields to normalize via normalizeFields.
3. Run and export the cleaned dataset.

Input

Field	Type	Description
records	array	Inline records to clean.
csv / json	string	CSV or JSON text (used if records empty).
datasetId / datasetUrl	string	Load from an Apify dataset or a public URL.
dedupeStrategy	string	business, product, or none.
fuzzyDedupe	boolean	Fuzzy name matching instead of exact keys.
fuzzyThreshold	number	Similarity threshold (0–1) for fuzzy dedupe.
normalizeFields	object	{fieldName: "email|phone|company|product|address|url"}.
dropEmptyColumns	boolean	Drop all-empty columns.
includeRemoved	boolean	Also emit removed duplicates / invalid rows.

Input example

{
  "records": [
    { "company": "Acme, Inc.", "email": "INFO@acme.com", "phone": "(555) 123-4567" },
    { "company": "Acme Inc",  "email": "info@acme.com", "phone": "555.123.4567" }
  ],
  "dedupeStrategy": "business",
  "normalizeFields": { "email": "email", "phone": "phone", "company": "company" }
}

Output

Cleaned records (recordType: "record"), optionally removed duplicates (duplicate) and invalid rows (invalid), and a final summary record.

Output example

{
  "recordType": "summary",
  "inputCount": 2,
  "cleanedCount": 1,
  "duplicatesRemoved": 1,
  "invalidRecords": 0,
  "columnsDropped": [],
  "dataQualitySummary": {
    "totalRecords": 2, "cleanedRecords": 1, "duplicatesRemoved": 1,
    "invalidRecords": 0, "columns": 3, "averageFillRate": 100.0
  }
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Environment variables

None. The Actor processes only the records you provide.