csv-cleaner

Deterministic typed sniff / normalise / dedupe for messy CSV — built for agent pipelines that receive dirty lead lists and need machine-readable cleaning reports, not guesses.

Pure local computation: zero network at runtime, zero keys, zero scraping.

What it does

• Parse CSV text, an array of line strings, or a {"rows": [...]} object array (serialised to CSV text with a header from the union of keys before parsing — same clean output as the equivalent CSV string): quoted fields, CRLF, BOM stripping, blank-line skipping, encoding normalisation to trimmed UTF-8 strings.
• Header normalisation: trim, lowercase, collapse internal whitespace, dedupe collisions.
• Typed column sniffing per column: int | float | date | postcode | email | url | bool | text, with a confidence score (share of unambiguous values matching the chosen type). Slash dates resolve via dateorder (auto/dmy/mdy); genuinely ambiguous date columns stay text and are annotated — the value is never guessed.
• Normalisation (only where unambiguous): dates → ISO YYYY-MM-DD, emails lowercased, UK postcodes canonicalised (sw1a1aa → SW1A 1AA), URLs lowercased host + trailing dot trimmed, leading-zero values kept verbatim as text.
• Dedupe: exact and near-duplicate rows (case/whitespace-insensitive key). First occurrence kept; every removal appears in dropped[] with a reason. Nothing is silently dropped.
• Invalid-row report: per-row reason (which column failed validation and why).
• Column hints: force a column type; values that fail the forced type mark the row invalid rather than being rewritten.
• Honest statuses: report status is ok | invalid | ambiguous. ambiguous means a column could not be typed without guessing — the raw value is preserved.

What it does NOT do

• No network calls at runtime — no fetch, no enrichment, no geocoding, no lookups.
• No enrichment of any kind: it will not append data it does not have.
• No fuzzy rewriting of values: normalisation is only the deterministic rules listed above; ambiguous values are kept as-is and annotated.
• No type guessing on ambiguous columns; no silent row drops (every drop has a reported reason).
• No persistence: one input in, one report out.

Quickstart

cd /home/hermes/venture/csv-cleaner
node main.js --fixtures          # runs the fixture suite; prints "N/N passed", exit 0 on pass
node main.js input.json          # process { "csv": "..." } or { "rows": [...] }, prints report JSON

input.json example:

{
  "csv": "Name, Email\n  Jon Smith ,jon@ex.com\n",
  "hints": {},
  "dateorder": "auto",
  "strictwidth": true
}

Object-array input is also accepted: {"rows": [{"name": "Jon", "email": "jon@ex.com"}, …]} is serialised to CSV text (header = union of keys, first-seen order) before parsing, producing the same report as the equivalent CSV string. {"rows": []} returns the documented invalid / empty input report (no crash). Nested object/array values are not flattenable: they render as empty cells and are counted in summary.nested_object_cells.

Inputs / outputs

• input_schema.json — Apify-style input: csv (string) or rows, hints, dateorder, strictwidth. All property names lowercase with prefill defaults.
• output_schema.json — documents every field of the cleaning report (status, reason, headers, columns[], rows[], dropped[], summary).

Fixtures

• fixtures/cases.json — 48 hand-checked cases across 9 suites (type sniffing, header normalisation, CSV parse edges incl. BOM/CRLF, column resolution, exact + near dedupe, malformed rows, hint enforcement, line-string rows, object-array rows). node main.js --fixtures emits 48 dataset items.

Positioning

P1 lead-hygiene companion rail: lead lists arrive as messy CSVs; P1 sells freshness/verification — hygiene is the adjacent deterministic step. Differentiator: bundled-offline execution + typed-schema machine-readable output (not a web-wrapper).