CSV Deduper Normalizer

Normalize CSV-style rows, detect duplicate keys and invalid values, and export clean JSON and CSV files for downstream imports.

What It Cleans

• Leading, trailing, and repeated whitespace.
• Domain names, including protocol, path, query, and www removal.
• URLs, including hostname casing, fragments, trailing slashes, and common tracking parameters.
• Email casing and basic email syntax.
• Configured lowercase and title-case fields.
• Duplicate rows based on one or more normalized key fields.
• Missing key fields and duplicate CSV header names.

The CSV parser handles quoted delimiters, escaped quotes, embedded newlines, CRLF input, and UTF-8 byte-order marks. Structured row objects are also accepted.

Input

Provide one source: csvText, rows, or a public csvUrl with fetchUrls enabled.

Field	Purpose
csvText	CSV content with a header row.
csvUrl	Public CSV URL fetched only when fetchUrls is true and csvText is empty.
rows	Structured row objects used instead of CSV text.
delimiter	Single-character delimiter; comma by default.
keyFields	Normalized fields combined into the duplicate key.
domainFields	Fields normalized as domains.
urlFields	Fields normalized as URLs.
emailFields	Fields lowercased and checked for basic email syntax.
lowercaseFields	Additional fields converted to lowercase.
titleCaseFields	Fields converted to title case.
trimWhitespace / collapseWhitespace	String whitespace controls.
includeDuplicates	Include duplicate rows in the dataset; clean files always contain kept rows only.
maxItems	Row cap, from 1 through 50,000.
maxCsvBytes	Retrieved CSV size cap, up to 5,000,000 bytes.
includeReport	Write result, summary, clean-file, and Markdown report records.

{
  "csvText": "company,domain,url,email\n ACME INC ,https://www.Example.com/path,https://Example.com/pricing/?utm_source=ad,Sales@Example.com\nAcme Inc,example.com,example.com/pricing,sales@example.com",
  "keyFields": ["domain", "email"],
  "domainFields": ["domain"],
  "urlFields": ["url"],
  "emailFields": ["email"],
  "titleCaseFields": ["company"],
  "includeDuplicates": true,
  "maxItems": 100,
  "includeReport": true
}

Dataset Rows

Each row describes one kept, duplicate, invalid, or diagnostic result. Important fields include:

• rowIndex, status, action, dedupeKey, and duplicateOfRowIndex.
• originalRow, normalizedRow, and changedFields.
• issueCount and issues, with codes such as duplicate-row, missing-key-field, and invalid-email.
• sourceType, sourceId, processedAt, and recommendation.
• diagnosticCode and sourceError for blocked retrieval or missing data rows.

Row statuses are clean, changed, duplicate, and invalid. A duplicate points to the zero-based index of the first matching normalized key.

Stored Artifacts

• Dataset: emitted row results after the includeDuplicates filter.
• CSV_DEDUPER_NORMALIZER_RESULTS.json: the same emitted result rows.
• CSV_DEDUPER_NORMALIZER_SUMMARY.json: full processed, kept, duplicate, changed, invalid, issue, diagnostic, and charge-limit counts.
• CSV_DEDUPER_NORMALIZER_CLEAN.csv: normalized kept rows.
• CSV_DEDUPER_NORMALIZER_CLEAN.json: normalized kept rows as JSON.
• CSV_DEDUPER_NORMALIZER_REPORT.md: readable QA report.

Usage Charges

The actor uses pay per event. Platform usage is included in these event prices.

Event	Price	When charged
actor-start	$0.00500	Once before paid work starts.
row-checked	$0.00390	Before each non-diagnostic dataset row is emitted.
issue-detected	$0.00372	Once per issue on a row, before that row is emitted.
qa-report-generated	$0.05000	Before result, summary, clean-file, and report records are written.

If Apify reports that a charge was limited or only partly accepted, the actor stops before the corresponding paid output. Set includeReport to false when KVS artifacts are not needed, and set includeDuplicates to false when duplicate dataset rows are not needed.

Public Tasks

Five capped task inputs cover domain and email deduplication, URL cleanup, company-name casing, missing-key QA, and clean-file export. Each task supplies concrete data and bounded row counts.

Retrieval Safety

• Only public http and https CSV URLs are accepted.
• Credentials in URLs and redirects are rejected.
• Loopback, link-local, and private network destinations are blocked unless allowPrivateUrls is explicitly enabled for controlled testing.
• Response length, retained CSV bytes, row count, and request duration are bounded.
• Fetch failures and empty inputs produce explicit diagnostic rows rather than fabricated data.

Limitations

• Deduplication is exact after configured normalization; fuzzy entity matching is not performed.
• Email checks validate syntax only and do not prove mailbox ownership or deliverability.
• Domain and URL normalization does not verify that a host exists.
• Inputs larger than the configured byte or row cap must be split by the caller.