# Data Cleaning & Deduplication (`moving_beacon-owner1/data-cleaning-deduplication`) Actor

Clean, normalize, deduplicate, and profile messy datasets using only the records you provide. Standardize fields, remove duplicates, flag invalid records, handle empty values, and generate an auditable data-quality summary—with no scraping or proxies required.

- **URL**: https://apify.com/moving\_beacon-owner1/data-cleaning-deduplication.md
- **Developed by:** [Jamshaid Arif](https://apify.com/moving_beacon-owner1) (community)
- **Categories:** Automation, Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $9.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Data Cleaning & Deduplication

A reusable, **data-only** Actor that turns a messy record dump into a clean, deduplicated, profiled dataset. It runs entirely on the records you give it — no scraping, no proxies — so it is the natural post-processing step after any scraper in your pipeline.

### What does this Actor do?

It runs a six-stage pipeline over your input records:

1. **Column normalization** — unifies key casing and whitespace into canonical column names (so `Company`, `company ` and `company` merge).
2. **Field & type normalization** — per-field normalization for emails, phones, company names, product names, addresses and URLs; invalid emails/phones/URLs are flagged.
3. **Empty-value handling** — blank / `N/A` / `null`-like values become real `null`; optionally drop columns that are empty across every row.
4. **Duplicate detection & removal** — key-based (business or product) or fuzzy name similarity.
5. **Invalid-record separation** — all-empty rows, or rows with an invalid normalized email/phone/URL, are set aside.
6. **Dataset profiling** — per-column fill rate, inferred type and top values, plus an overall data-quality summary.

### Why use it?

- Merge and de-duplicate leads or product feeds collected from several sources into one clean list.
- Standardise emails and phone numbers before importing into a CRM.
- Get a quick data-quality report (fill rates, duplicate counts, invalid rows) for any dataset.

### How to use it

1. Provide records inline (`records`), as `csv` / `json` text, or point at an Apify dataset (`datasetId`) or a public `datasetUrl`.
2. Choose a `dedupeStrategy`, optionally enable `fuzzyDedupe`, and map fields to normalize via `normalizeFields`.
3. Run and export the cleaned dataset.

### Input

| Field | Type | Description |
| --- | --- | --- |
| `records` | array | Inline records to clean. |
| `csv` / `json` | string | CSV or JSON text (used if `records` empty). |
| `datasetId` / `datasetUrl` | string | Load from an Apify dataset or a public URL. |
| `dedupeStrategy` | string | `business`, `product`, or `none`. |
| `fuzzyDedupe` | boolean | Fuzzy name matching instead of exact keys. |
| `fuzzyThreshold` | number | Similarity threshold (0–1) for fuzzy dedupe. |
| `normalizeFields` | object | `{fieldName: "email\|phone\|company\|product\|address\|url"}`. |
| `dropEmptyColumns` | boolean | Drop all-empty columns. |
| `includeRemoved` | boolean | Also emit removed duplicates / invalid rows. |

#### Input example

```json
{
  "records": [
    { "company": "Acme, Inc.", "email": "INFO@acme.com", "phone": "(555) 123-4567" },
    { "company": "Acme Inc",  "email": "info@acme.com", "phone": "555.123.4567" }
  ],
  "dedupeStrategy": "business",
  "normalizeFields": { "email": "email", "phone": "phone", "company": "company" }
}
```

### Output

Cleaned records (`recordType: "record"`), optionally removed duplicates (`duplicate`) and invalid rows (`invalid`), and a final `summary` record.

#### Output example

```json
{
  "recordType": "summary",
  "inputCount": 2,
  "cleanedCount": 1,
  "duplicatesRemoved": 1,
  "invalidRecords": 0,
  "columnsDropped": [],
  "dataQualitySummary": {
    "totalRecords": 2, "cleanedRecords": 1, "duplicatesRemoved": 1,
    "invalidRecords": 0, "columns": 3, "averageFillRate": 100.0
  }
}
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

### Environment variables

None. The Actor processes only the records you provide.

# Actor input Schema

## `records` (type: `array`):

Inline JSON array of record objects to clean. Alternatively use csv/json/datasetId/datasetUrl.

## `csv` (type: `string`):

CSV data (first row is the header). Used if 'records' is empty.

## `json` (type: `string`):

A JSON array as text. Used if 'records' and 'csv' are empty.

## `datasetId` (type: `string`):

Load records from an existing Apify dataset by id or name.

## `datasetUrl` (type: `string`):

Public URL returning a JSON array of records.

## `dedupeStrategy` (type: `string`):

Key used to detect duplicates.

## `fuzzyDedupe` (type: `boolean`):

Use fuzzy name similarity instead of exact keys.

## `fuzzyThreshold` (type: `number`):

Similarity (0-1) above which two records are duplicates when fuzzyDedupe is on.

## `normalizeFields` (type: `object`):

Map of fieldName -> type in {email, phone, company, product, address, url}. Invalid emails/phones/urls flag the record as invalid.

## `dropEmptyColumns` (type: `boolean`):

Remove columns that are empty across every cleaned record.

## `includeRemoved` (type: `boolean`):

Also output removed duplicates and invalid rows (tagged by recordType).

## Actor input object example

```json
{
  "records": [
    {
      "company": "Acme, Inc.",
      "email": "INFO@acme.com",
      "phone": "(555) 123-4567"
    },
    {
      "company": "Acme Inc",
      "email": "info@acme.com",
      "phone": "555.123.4567"
    }
  ],
  "dedupeStrategy": "business",
  "fuzzyDedupe": false,
  "fuzzyThreshold": 0.9,
  "normalizeFields": {},
  "dropEmptyColumns": false,
  "includeRemoved": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "records": [
        {
            "company": "Acme, Inc.",
            "email": "INFO@acme.com",
            "phone": "(555) 123-4567"
        },
        {
            "company": "Acme Inc",
            "email": "info@acme.com",
            "phone": "555.123.4567"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("moving_beacon-owner1/data-cleaning-deduplication").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "records": [
        {
            "company": "Acme, Inc.",
            "email": "INFO@acme.com",
            "phone": "(555) 123-4567",
        },
        {
            "company": "Acme Inc",
            "email": "info@acme.com",
            "phone": "555.123.4567",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("moving_beacon-owner1/data-cleaning-deduplication").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "records": [
    {
      "company": "Acme, Inc.",
      "email": "INFO@acme.com",
      "phone": "(555) 123-4567"
    },
    {
      "company": "Acme Inc",
      "email": "info@acme.com",
      "phone": "555.123.4567"
    }
  ]
}' |
apify call moving_beacon-owner1/data-cleaning-deduplication --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,moving_beacon-owner1/data-cleaning-deduplication"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/OvK81OlHNwJbOjHgE/builds/iPs0YK6JkMCpGmPuj/openapi.json