# CSV Deduper Normalizer (`junipr/csv-deduper-normalizer`) Actor Deduplicate and normalize CSV-style rows. Clean whitespace, casing, domains, URLs, and emails, then output kept and duplicate rows plus clean CSV/JSON files. - **URL**: https://apify.com/junipr/csv-deduper-normalizer.md - **Developed by:** [junipr](https://apify.com/junipr) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.95 / 1,000 item processeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## CSV Deduper Normalizer Normalize CSV-style rows, detect duplicate keys and invalid values, and export clean JSON and CSV files for downstream imports. ### What It Cleans - Leading, trailing, and repeated whitespace. - Domain names, including protocol, path, query, and `www` removal. - URLs, including hostname casing, fragments, trailing slashes, and common tracking parameters. - Email casing and basic email syntax. - Configured lowercase and title-case fields. - Duplicate rows based on one or more normalized key fields. - Missing key fields and duplicate CSV header names. The CSV parser handles quoted delimiters, escaped quotes, embedded newlines, CRLF input, and UTF-8 byte-order marks. Structured row objects are also accepted. ### Input Provide one source: `csvText`, `rows`, or a public `csvUrl` with `fetchUrls` enabled. | Field | Purpose | | --- | --- | | `csvText` | CSV content with a header row. | | `csvUrl` | Public CSV URL fetched only when `fetchUrls` is true and `csvText` is empty. | | `rows` | Structured row objects used instead of CSV text. | | `delimiter` | Single-character delimiter; comma by default. | | `keyFields` | Normalized fields combined into the duplicate key. | | `domainFields` | Fields normalized as domains. | | `urlFields` | Fields normalized as URLs. | | `emailFields` | Fields lowercased and checked for basic email syntax. | | `lowercaseFields` | Additional fields converted to lowercase. | | `titleCaseFields` | Fields converted to title case. | | `trimWhitespace` / `collapseWhitespace` | String whitespace controls. | | `includeDuplicates` | Include duplicate rows in the dataset; clean files always contain kept rows only. | | `maxItems` | Row cap, from 1 through 50,000. | | `maxCsvBytes` | Retrieved CSV size cap, up to 5,000,000 bytes. | | `includeReport` | Write result, summary, clean-file, and Markdown report records. | ```json { "csvText": "company,domain,url,email\n ACME INC ,https://www.Example.com/path,https://Example.com/pricing/?utm_source=ad,Sales@Example.com\nAcme Inc,example.com,example.com/pricing,sales@example.com", "keyFields": ["domain", "email"], "domainFields": ["domain"], "urlFields": ["url"], "emailFields": ["email"], "titleCaseFields": ["company"], "includeDuplicates": true, "maxItems": 100, "includeReport": true } ```` ### Dataset Rows Each row describes one kept, duplicate, invalid, or diagnostic result. Important fields include: - `rowIndex`, `status`, `action`, `dedupeKey`, and `duplicateOfRowIndex`. - `originalRow`, `normalizedRow`, and `changedFields`. - `issueCount` and `issues`, with codes such as `duplicate-row`, `missing-key-field`, and `invalid-email`. - `sourceType`, `sourceId`, `processedAt`, and `recommendation`. - `diagnosticCode` and `sourceError` for blocked retrieval or missing data rows. Row statuses are `clean`, `changed`, `duplicate`, and `invalid`. A duplicate points to the zero-based index of the first matching normalized key. ### Stored Artifacts - Dataset: emitted row results after the `includeDuplicates` filter. - `CSV_DEDUPER_NORMALIZER_RESULTS.json`: the same emitted result rows. - `CSV_DEDUPER_NORMALIZER_SUMMARY.json`: full processed, kept, duplicate, changed, invalid, issue, diagnostic, and charge-limit counts. - `CSV_DEDUPER_NORMALIZER_CLEAN.csv`: normalized kept rows. - `CSV_DEDUPER_NORMALIZER_CLEAN.json`: normalized kept rows as JSON. - `CSV_DEDUPER_NORMALIZER_REPORT.md`: readable QA report. ### Usage Charges The actor uses pay per event. Platform usage is included in these event prices. | Event | Price | When charged | | --- | ---: | --- | | `actor-start` | $0.00500 | Once before paid work starts. | | `row-checked` | $0.00390 | Before each non-diagnostic dataset row is emitted. | | `issue-detected` | $0.00372 | Once per issue on a row, before that row is emitted. | | `qa-report-generated` | $0.05000 | Before result, summary, clean-file, and report records are written. | If Apify reports that a charge was limited or only partly accepted, the actor stops before the corresponding paid output. Set `includeReport` to false when KVS artifacts are not needed, and set `includeDuplicates` to false when duplicate dataset rows are not needed. ### Public Tasks Five capped task inputs cover domain and email deduplication, URL cleanup, company-name casing, missing-key QA, and clean-file export. Each task supplies concrete data and bounded row counts. ### Retrieval Safety - Only public `http` and `https` CSV URLs are accepted. - Credentials in URLs and redirects are rejected. - Loopback, link-local, and private network destinations are blocked unless `allowPrivateUrls` is explicitly enabled for controlled testing. - Response length, retained CSV bytes, row count, and request duration are bounded. - Fetch failures and empty inputs produce explicit diagnostic rows rather than fabricated data. ### Limitations - Deduplication is exact after configured normalization; fuzzy entity matching is not performed. - Email checks validate syntax only and do not prove mailbox ownership or deliverability. - Domain and URL normalization does not verify that a host exists. - Inputs larger than the configured byte or row cap must be split by the caller. # Actor input Schema ## `csvText` (type: `string`): CSV data with a header row. ## `csvUrl` (type: `string`): CSV URL fetched only when enabled and csvText is empty. ## `rows` (type: `array`): Structured rows used instead of CSV text. ## `delimiter` (type: `string`): Single-character CSV delimiter. ## `keyFields` (type: `array`): Fields combined into the duplicate key. ## `domainFields` (type: `array`): Field names whose values should be normalized as domains. ## `urlFields` (type: `array`): Field names whose values should be normalized as URLs. ## `emailFields` (type: `array`): Field names whose values should be validated as email addresses. ## `lowercaseFields` (type: `array`): Field names whose string values should be lowercased. ## `titleCaseFields` (type: `array`): Field names whose string values should be converted to title case. ## `trimWhitespace` (type: `boolean`): Trim leading and trailing whitespace from string fields. ## `collapseWhitespace` (type: `boolean`): Collapse repeated internal whitespace in normalized string fields. ## `includeDuplicates` (type: `boolean`): Include duplicate rows in the dataset output instead of summary-only counts. ## `fetchUrls` (type: `boolean`): Fetch csvUrl when csvText and inline rows are empty. ## `fetchTimeoutMs` (type: `integer`): Maximum time to wait when fetching a public CSV URL. ## `allowPrivateUrls` (type: `boolean`): Allow fetching private or local network CSV URLs. ## `maxCsvBytes` (type: `integer`): Maximum number of CSV bytes to read from text or fetched content. ## `maxItems` (type: `integer`): Maximum number of CSV or inline rows to process. ## `includeReport` (type: `boolean`): Write cleaned CSV, duplicate summary, and Markdown report artifacts. ## `dryRun` (type: `boolean`): Validate and summarize input without writing paid dataset rows. ## `debug` (type: `boolean`): Enable additional diagnostic log output. ## Actor input object example ```json { "csvText": "", "csvUrl": "", "rows": [], "delimiter": ",", "keyFields": [ "domain" ], "domainFields": [ "domain" ], "urlFields": [ "url" ], "emailFields": [ "email" ], "lowercaseFields": [], "titleCaseFields": [], "trimWhitespace": true, "collapseWhitespace": true, "includeDuplicates": true, "fetchUrls": false, "fetchTimeoutMs": 10000, "allowPrivateUrls": false, "maxCsvBytes": 500000, "maxItems": 100, "includeReport": true, "dryRun": false, "debug": false } ``` # Actor output Schema ## `results` (type: `string`): Normalize, validate, deduplicate, and export CSV-style rows. ## `artifact1` (type: `string`): CSV\_DEDUPER\_NORMALIZER\_RESULTS.json ## `artifact2` (type: `string`): CSV\_DEDUPER\_NORMALIZER\_SUMMARY.json ## `artifact3` (type: `string`): CSV\_DEDUPER\_NORMALIZER\_REPORT.md ## `artifact4` (type: `string`): CSV\_DEDUPER\_NORMALIZER\_CLEAN.csv ## `artifact5` (type: `string`): CSV\_DEDUPER\_NORMALIZER\_CLEAN.json # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("junipr/csv-deduper-normalizer").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("junipr/csv-deduper-normalizer").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call junipr/csv-deduper-normalizer --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=junipr/csv-deduper-normalizer", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "CSV Deduper Normalizer", "description": "Deduplicate and normalize CSV-style rows. Clean whitespace, casing, domains, URLs, and emails, then output kept and duplicate rows plus clean CSV/JSON files.", "version": "1.0", "x-build-id": "rEZDTnE2fPJOKwbR9" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/junipr~csv-deduper-normalizer/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-junipr-csv-deduper-normalizer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/junipr~csv-deduper-normalizer/runs": { "post": { "operationId": "runs-sync-junipr-csv-deduper-normalizer", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/junipr~csv-deduper-normalizer/run-sync": { "post": { "operationId": "run-sync-junipr-csv-deduper-normalizer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "csvText": { "title": "CSV text", "type": "string", "description": "CSV data with a header row.", "default": "" }, "csvUrl": { "title": "Public CSV URL", "type": "string", "description": "CSV URL fetched only when enabled and csvText is empty.", "default": "" }, "rows": { "title": "Inline rows", "type": "array", "description": "Structured rows used instead of CSV text.", "items": { "type": "object" }, "default": [] }, "delimiter": { "title": "Delimiter", "type": "string", "description": "Single-character CSV delimiter.", "default": "," }, "keyFields": { "title": "Dedupe key fields", "type": "array", "description": "Fields combined into the duplicate key.", "items": { "type": "string" }, "default": [ "domain" ] }, "domainFields": { "title": "Domain fields", "type": "array", "description": "Field names whose values should be normalized as domains.", "items": { "type": "string" }, "default": [ "domain" ] }, "urlFields": { "title": "URL fields", "type": "array", "description": "Field names whose values should be normalized as URLs.", "items": { "type": "string" }, "default": [ "url" ] }, "emailFields": { "title": "Email fields", "type": "array", "description": "Field names whose values should be validated as email addresses.", "items": { "type": "string" }, "default": [ "email" ] }, "lowercaseFields": { "title": "Lowercase fields", "type": "array", "description": "Field names whose string values should be lowercased.", "items": { "type": "string" }, "default": [] }, "titleCaseFields": { "title": "Title-case fields", "type": "array", "description": "Field names whose string values should be converted to title case.", "items": { "type": "string" }, "default": [] }, "trimWhitespace": { "title": "Trim whitespace", "type": "boolean", "description": "Trim leading and trailing whitespace from string fields.", "default": true }, "collapseWhitespace": { "title": "Collapse whitespace", "type": "boolean", "description": "Collapse repeated internal whitespace in normalized string fields.", "default": true }, "includeDuplicates": { "title": "Include duplicate rows", "type": "boolean", "description": "Include duplicate rows in the dataset output instead of summary-only counts.", "default": true }, "fetchUrls": { "title": "Fetch CSV URL", "type": "boolean", "description": "Fetch csvUrl when csvText and inline rows are empty.", "default": false }, "fetchTimeoutMs": { "title": "Fetch timeout", "minimum": 1000, "maximum": 30000, "type": "integer", "description": "Maximum time to wait when fetching a public CSV URL.", "default": 10000 }, "allowPrivateUrls": { "title": "Allow private URLs", "type": "boolean", "description": "Allow fetching private or local network CSV URLs.", "default": false }, "maxCsvBytes": { "title": "Max CSV bytes", "minimum": 1000, "maximum": 5000000, "type": "integer", "description": "Maximum number of CSV bytes to read from text or fetched content.", "default": 500000 }, "maxItems": { "title": "Max rows", "minimum": 1, "maximum": 50000, "type": "integer", "description": "Maximum number of CSV or inline rows to process.", "default": 100 }, "includeReport": { "title": "Include clean files and report", "type": "boolean", "description": "Write cleaned CSV, duplicate summary, and Markdown report artifacts.", "default": true }, "dryRun": { "title": "Dry run", "type": "boolean", "description": "Validate and summarize input without writing paid dataset rows.", "default": false }, "debug": { "title": "Debug logs", "type": "boolean", "description": "Enable additional diagnostic log output.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```