Dataset Cleaner & Deduplicator

Scraped data is messy. Field names arrive in three different casings, half the values are "N/A" instead of empty, phone numbers carry random punctuation, and the same company shows up four times under slightly different names.

This Actor takes any dataset and hands you back a clean one. Point it at the output of any scraper, or paste your own JSON.

What it fixes

Problem	What you get back
"First Name", "firstName", "FIRST_NAME"	one consistent first_name key
" Jean Dupont "	"Jean Dupont"
"N/A", "null", "-", "unknown"	real null values
"+33 (0)6 12.34-56 78"	"+330612345678"
"1 234,56" and "1,234.56"	1234.56
"Société Dupont" vs "SOCIETE DUPONT"	one row, not two

Two ways to run it

Chain it after a scraper. Copy the Dataset ID from any finished run and paste it into inputDatasetId. This is the common case: scrape, then clean.

Paste data directly. Drop a JSON array into items for a one-off cleanup.

Deduplication

Exact mode compares values literally, case-insensitively. Pick which fields identify a duplicate with dedupeFields — usually email or url. Leave it empty to compare whole rows.

Fuzzy mode catches near-duplicates that exact matching misses. Set fuzzyField to the column that identifies a record, typically a company or person name.

The fuzzyThreshold controls how aggressive matching is:

Threshold	Behaviour
95+	Only trivial variants: casing, accents, extra spaces
90	Safe default — the same entity written slightly differently
80	Catches suffix differences like SARL, Ltd, Inc
Below 80	Aggressive. Expect false positives

Legal suffixes cost more similarity than they look: Societe Dupont and SOCIETE DUPONT SARL score just under 85, so they stay separate at the default threshold. Lower fuzzyThreshold to 80 if you want those merged.

Accents and casing are normalized before comparison, so Société Dupont and SOCIETE DUPONT always match regardless of threshold.

Output

Cleaned records go to the default dataset. A STATS record is written to the key-value store:

{
  "inputItems": 5000,
  "outputItems": 4212,
  "duplicatesRemoved": 703,
  "droppedIncomplete": 85,
  "dedupeMode": "fuzzy"
}

Notes

Fuzzy deduplication is quadratic in the worst case. Above 50,000 rows the Actor automatically falls back to exact matching rather than burning your compute budget. Records are compared inside blocks of similar values, so real-world runs stay fast well below that ceiling.

Field names are normalized before requiredFields, dedupeFields and fuzzyField are applied — so write them in snake_case, or just use the original name and let the Actor normalize it for you. If you turn normalizeFieldNames off, these three settings are matched against your original keys instead, exactly as they appear in the data.