# Dataset Deduplicator & Cleaner β€” Dedup + Report (`eszetael_lab/dataset-deduplicator-cleaner`) Actor Deduplicate and clean any dataset: exact, normalized and fuzzy dedup, per-type field cleaning (emails, phones, URLs, HTML) and a data-quality report. - **URL**: https://apify.com/eszetael\_lab/dataset-deduplicator-cleaner.md - **Developed by:** [RadosΕ‚aw Szal](https://apify.com/eszetael_lab) (community) - **Categories:** Developer tools, AI, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Dataset Deduplicator & Cleaner > πŸ”— Part of the **[Apify actors collection](https://github.com/Eszetael/apify-actors)** β€” 6 reliable, tested scrapers & data tools. **Quick start:** paste a JSON array into **Inline data** (or give a **Source dataset ID**), then Run. **Turn messy, duplicate-ridden data into analysis-ready records in one deterministic run.** Point it at any Apify dataset (or paste JSON directly), and get back deduplicated, cleaned, type-consistent records β€” plus a data-quality report. No scraping, no external APIs, no maintenance. --- ### ✨ What it does - **Deduplicates** records β€” exact, normalized, or fuzzy (near-duplicate) matching. - **Cleans fields by type** β€” emails, phones, URLs, whitespace, HTML, empty values, and more. - **Reports quality** β€” input/output counts, duplicates removed, and per-field fill rates, saved to the key-value store as `QUALITY_REPORT`. - **Two ways in** β€” pipe another actor's output via **Source dataset ID**, or paste **Inline data**. - **Zero maintenance** β€” deterministic, no third-party services, pay only per output record. ### 🎯 Who is it for | Role | What it solves | |------|----------------| | **Scraper users** | Clean and de-dupe an actor's output before loading it into a DB or spreadsheet. | | **Data / ops teams** | Standardize messy exports (leads, catalogs, contacts) without writing scripts. | | **Analysts** | De-dupe and normalize lists, then check fill rates at a glance. | --- ### βš™οΈ Deduplication modes (`dedupMode`) | Mode | Description | |------|-------------| | `none` | Don't deduplicate β€” clean only. | | `exact` | Byte-identical values. | | `normalized` | Ignores case, whitespace and punctuation β€” catches `"John Smith"` vs `"john smith!"`. **(default)** | | `fuzzy` | Also merges near-duplicates by string similarity (`similarityThreshold`, 0.5–0.99). Best for a few thousand rows; on very large or low-diversity inputs it automatically degrades to `normalized` and tells you in the run status (so a run never hangs). | Set **`dedupKeys`** to the fields that identify a duplicate (e.g. `email`, `url`); leave it empty to compare the whole record. When duplicates are found, **`keepStrategy`** decides which one survives: - `most_complete` β€” keep the record with the most filled fields *(default)* - `first` / `last` β€” keep the first or last occurrence --- ### 🧹 Field cleaning Fields are cleaned by type. With **`smartFields`** on (default), emails, phones and URLs are detected by field name; everything else is treated as text. | Option | What it does | |--------|--------------| | `cleanEmails` | Lowercase + trim values that look like a valid email (invalid ones are left as text, not dropped). | | `cleanPhones` | Compact phone-like fields to digits, keeping a leading `+`/`00` country prefix. | | `cleanUrls` | Lowercase the scheme and host, drop a trailing slash. | | `stripHtml` | Remove HTML tags and unescape entities (`&` β†’ `&`). | | `emptyToNull` | Turn empty / blank values into `null` for consistency. *(default on)* | | `dropEmptyFields` | Remove empty fields entirely instead of keeping them as `null`. | | `case` | Force plain-text fields to `lower` / `upper` / `title` case. | | `coerceTypes` | Convert numeric and `true`/`false` strings into real numbers and booleans. | Whitespace collapse, end-trimming and control-character stripping are always applied to text. --- ### πŸ“₯ Input Provide **either** `sourceDatasetId` **or** `data`: ```json { "sourceDatasetId": "abc123XYZ", "dedupMode": "normalized", "dedupKeys": ["email"], "keepStrategy": "most_complete", "cleanEmails": true, "cleanPhones": true, "cleanUrls": true, "emptyToNull": true, "includeReport": true, "maxItems": 0 } ```` Or clean records inline: ```json { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/", "notes": "

Top client

" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "http://acme.com", "notes": "Top client" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "", "website": "https://globex.io/", "notes": "" } ], "dedupMode": "normalized", "dedupKeys": ["name"], "stripHtml": true } ``` **Output** (deduplicated on `name`, HTML stripped, empty β†’ null): ```json [ { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com", "notes": "Top client" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": null, "website": "https://globex.io", "notes": null } ] ``` *** ### πŸ“Š Quality report When `includeReport` is on, a report is saved to the **key-value store** under `QUALITY_REPORT`: ```json { "inputRecords": 3, "outputRecords": 2, "duplicatesRemoved": 1, "dedupMode": "normalized", "fieldValuesCleaned": 7, "fieldCount": 5, "fields": { "email": { "filled": 2, "empty": 0, "fillRate": 1.0 }, "phone": { "filled": 1, "empty": 1, "fillRate": 0.5 }, "website": { "filled": 2, "empty": 0, "fillRate": 1.0 }, "notes": { "filled": 1, "empty": 1, "fillRate": 0.5 } } } ``` *** ### πŸš€ How to use 1. Open the actor in Apify Console (or call it via API). 2. Pick a data source β€” a **Source dataset ID** from a previous run, or paste a JSON array into **Inline data**. 3. Adjust dedup and cleaning if you like β€” the defaults (normalized dedup + sensible cleaning) work out of the box. 4. Run. Clean records land in the default dataset; the quality report goes to the key-value store as `QUALITY_REPORT`. 5. Export as JSON / CSV / Excel, or push onward with Apify integrations. *** ### πŸ’° Pricing & reliability - **Pay per result** β€” you're billed only for the clean records written to the dataset. - **Deterministic** β€” same input and config always produce the same output. - **No external dependencies** β€” no third-party APIs, no browser, nothing to break. - Handles up to **50,000 records** per run (`exact`/`normalized`). Fuzzy is meant for a few thousand rows and degrades gracefully above that. Larger source datasets are paginated in full, and if the 50,000 cap truncates the input the run says so in its status and quality report. ### License Licensed under the Apache License 2.0. # Actor input Schema ## `sourceDatasetId` (type: `string`): ID of an existing Apify dataset to clean (e.g. the output of another actor). Use this OR paste records into 'Inline data' below. ## `data` (type: `array`): Records to clean as a JSON array of objects. Use this instead of a source dataset ID for ad-hoc or piped data. ## `dedupMode` (type: `string`): How to detect duplicates. Exact = byte-identical. Normalized = ignores case/whitespace/punctuation. Fuzzy = also merges near-duplicates by similarity. None = only clean, keep every row. ## `dedupKeys` (type: `array`): Field names that identify a duplicate (e.g. email, url). Leave empty to compare the whole record. ## `similarityThreshold` (type: `number`): For fuzzy mode only: 0.5–0.99. Higher = stricter (fewer merges). 0.9 is a good start. ## `keepStrategy` (type: `string`): When duplicates are found, which record survives. ## `smartFields` (type: `boolean`): Auto-detect emails, phones and URLs by field name and clean them accordingly. ## `cleanEmails` (type: `boolean`): Lowercase and trim values in email-like fields (only if they look like a valid address). ## `cleanPhones` (type: `boolean`): Normalize phone-like fields to a compact digits (+country) form. ## `cleanUrls` (type: `boolean`): Lowercase the scheme and host and drop trailing slashes in URL-like fields. ## `stripHtml` (type: `boolean`): Remove HTML tags and unescape entities from text fields. ## `coerceTypes` (type: `boolean`): Convert numeric and true/false strings into real numbers and booleans (AI-ready output). ## `emptyToNull` (type: `boolean`): Turn empty strings and blank values into null for consistency. ## `dropEmptyFields` (type: `boolean`): Remove empty fields from each record entirely instead of keeping them as null. ## `case` (type: `string`): Optionally force the case of plain text fields (emails/phones/URLs are handled separately). ## `includeReport` (type: `boolean`): Save a data-quality report (counts, duplicates removed, per-field fill rates) to the key-value store as QUALITY\_REPORT. ## `maxItems` (type: `integer`): Cap on delivered clean records (0 = up to the safety limit of 50000). ## Actor input object example ```json { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/" } ], "dedupMode": "normalized", "similarityThreshold": 0.9, "keepStrategy": "most_complete", "smartFields": true, "cleanEmails": true, "cleanPhones": true, "cleanUrls": true, "stripHtml": false, "coerceTypes": false, "emptyToNull": true, "dropEmptyFields": false, "case": "", "includeReport": true, "maxItems": 0 } ``` # Actor output Schema ## `cleanRecords` (type: `string`): The deduplicated, cleaned dataset. Fields mirror your input records. ## `qualityReport` (type: `string`): JSON report under the QUALITY\_REPORT key: input/output counts, duplicates removed, dedup mode, per-field fill rates, and any warnings (truncation, fuzzy degrade). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("eszetael_lab/dataset-deduplicator-cleaner").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/", }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com", }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/", }, ] } # Run the Actor and wait for it to finish run = client.actor("eszetael_lab/dataset-deduplicator-cleaner").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/" } ] }' | apify call eszetael_lab/dataset-deduplicator-cleaner --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=eszetael_lab/dataset-deduplicator-cleaner", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Dataset Deduplicator & Cleaner β€” Dedup + Report", "description": "Deduplicate and clean any dataset: exact, normalized and fuzzy dedup, per-type field cleaning (emails, phones, URLs, HTML) and a data-quality report.", "version": "0.1", "x-build-id": "5udcqoVP9El9IGbZD" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/eszetael_lab~dataset-deduplicator-cleaner/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-eszetael_lab-dataset-deduplicator-cleaner", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/eszetael_lab~dataset-deduplicator-cleaner/runs": { "post": { "operationId": "runs-sync-eszetael_lab-dataset-deduplicator-cleaner", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/eszetael_lab~dataset-deduplicator-cleaner/run-sync": { "post": { "operationId": "run-sync-eszetael_lab-dataset-deduplicator-cleaner", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "sourceDatasetId": { "title": "Source dataset ID", "type": "string", "description": "ID of an existing Apify dataset to clean (e.g. the output of another actor). Use this OR paste records into 'Inline data' below." }, "data": { "title": "Inline data", "type": "array", "description": "Records to clean as a JSON array of objects. Use this instead of a source dataset ID for ad-hoc or piped data." }, "dedupMode": { "title": "Deduplication mode", "enum": [ "none", "exact", "normalized", "fuzzy" ], "type": "string", "description": "How to detect duplicates. Exact = byte-identical. Normalized = ignores case/whitespace/punctuation. Fuzzy = also merges near-duplicates by similarity. None = only clean, keep every row.", "default": "normalized" }, "dedupKeys": { "title": "Deduplication keys", "type": "array", "description": "Field names that identify a duplicate (e.g. email, url). Leave empty to compare the whole record.", "items": { "type": "string" } }, "similarityThreshold": { "title": "Fuzzy similarity threshold", "minimum": 0.5, "maximum": 0.99, "type": "number", "description": "For fuzzy mode only: 0.5–0.99. Higher = stricter (fewer merges). 0.9 is a good start.", "default": 0.9 }, "keepStrategy": { "title": "Which duplicate to keep", "enum": [ "most_complete", "first", "last" ], "type": "string", "description": "When duplicates are found, which record survives.", "default": "most_complete" }, "smartFields": { "title": "Smart field cleaning", "type": "boolean", "description": "Auto-detect emails, phones and URLs by field name and clean them accordingly.", "default": true }, "cleanEmails": { "title": "Clean emails", "type": "boolean", "description": "Lowercase and trim values in email-like fields (only if they look like a valid address).", "default": true }, "cleanPhones": { "title": "Clean phone numbers", "type": "boolean", "description": "Normalize phone-like fields to a compact digits (+country) form.", "default": true }, "cleanUrls": { "title": "Clean URLs", "type": "boolean", "description": "Lowercase the scheme and host and drop trailing slashes in URL-like fields.", "default": true }, "stripHtml": { "title": "Strip HTML", "type": "boolean", "description": "Remove HTML tags and unescape entities from text fields.", "default": false }, "coerceTypes": { "title": "Coerce types", "type": "boolean", "description": "Convert numeric and true/false strings into real numbers and booleans (AI-ready output).", "default": false }, "emptyToNull": { "title": "Empty values to null", "type": "boolean", "description": "Turn empty strings and blank values into null for consistency.", "default": true }, "dropEmptyFields": { "title": "Drop empty fields", "type": "boolean", "description": "Remove empty fields from each record entirely instead of keeping them as null.", "default": false }, "case": { "title": "Text case", "enum": [ "", "lower", "upper", "title" ], "type": "string", "description": "Optionally force the case of plain text fields (emails/phones/URLs are handled separately).", "default": "" }, "includeReport": { "title": "Include quality report", "type": "boolean", "description": "Save a data-quality report (counts, duplicates removed, per-field fill rates) to the key-value store as QUALITY_REPORT.", "default": true }, "maxItems": { "title": "Max items", "type": "integer", "description": "Cap on delivered clean records (0 = up to the safety limit of 50000).", "default": 0 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```