# 🧽 Dataset Deduplicator - Clean Any Scraped Dataset (`that_red_bird/dataset-deduplicator`) Actor

🧽 Upload any dataset — scraped rows, a RAG corpus, an export — and get back exact AND near-duplicates removed, plus a per-field data quality report. ✅ MinHash + LSH banding finds near-duplicate text at scale without O(n²) pairwise comparison.

- **URL**: https://apify.com/that\_red\_bird/dataset-deduplicator.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Dataset Deduplicator & Quality Scorer

Upload any dataset — scraped rows, a RAG corpus you're about to embed, a CRM or catalogue export,
the combined output of three different scrapers pointed at the same site — and get back a clean
version with exact duplicates gone, near-duplicates gone, and a per-field data quality report
telling you how dirty the input actually was.

### What it does

1. **Exact duplicates** — a sha256 content hash over every record's sorted fields catches
   byte-identical rows in a single pass, no comparisons needed.
2. **Near duplicates** — the harder problem. Two scraped rows describing the same article/listing
   are rarely byte-identical (different whitespace, a re-scrape days later, a paragraph edited).
   This actor implements **MinHash + LSH banding from scratch** (`src/lsh.js`) to find those pairs
   without comparing every record to every other record:
   - each text field is broken into overlapping word shingles (falling back to character shingles
     for short strings),
   - each document's shingle set is summarised into a short MinHash signature,
   - the signature is split into bands and hashed into buckets — only records sharing a bucket
     become "candidates",
   - every candidate is then verified with an **exact** Jaccard similarity check on the real
     shingle sets before it counts as a duplicate.
     This keeps the run close to O(n) instead of O(n²): a bucket of size *b* only costs *b²/2*
     comparisons, and unrelated records never share a bucket at all.
3. **Quality report** — for every field: completeness %, uniqueness %, dominant type + type
   consistency %, blank count, and (for numeric fields) an IQR-based outlier count. These roll up
   into one 0-100 quality score that also factors in how much of the dataset was duplicated.

### Honest limits — read before you rely on the near-duplicate count

**MinHash + LSH is probabilistic by design.** Banding trades recall for speed: a genuine
near-duplicate pair can fail to land in the same bucket in any band and simply never becomes a
candidate — a **false negative**. Raising `numBands` (or lowering `numHashes` per band) increases
recall at the cost of more candidate pairs to verify and more CPU. There are **no false
positives** from this stage, though: every candidate is re-checked with an exact Jaccard
similarity on the real shingle sets before being reported, so nothing is flagged as a duplicate on
the MinHash estimate alone.

Other things this actor will not do:

- It does not understand meaning — two records that say the same thing in completely different
  words will not be caught. This is lexical (shingle-overlap) similarity, not semantic similarity.
- Exact-duplicate hashing is structural: two JSON objects with the same field values in a
  different nested-object key order will not hash identically. Top-level field order does not
  matter; deeply nested object key order does.
- It caps at 200,000 rows per run and skips any oversized LSH bucket (e.g. thousands of rows with
  blank text all landing in one bucket) to avoid a runaway comparison count.
- Auto-detected text fields require the column to be a string in at least 30% of sampled rows
  with an average length over `textFieldMinAvgLength` — a dataset with no such column (e.g. purely
  numeric) skips the near-duplicate stage entirely and runs exact-dedup + quality report only.

### Input

```json
{
  "records": [
    { "title": "Best wireless headphones 2024", "body": "Great sound and battery life for the price." },
    { "title": "Best wireless headphones 2024", "body": "Great sound and battery life for the price." },
    { "title": "Best wireless headphones of 2024", "body": "Great sound quality and long battery life for the price." }
  ],
  "similarityThreshold": 80
}
```

Only `records` and/or `datasetIds` is required. Everything else — similarity threshold, shingle
size, MinHash/LSH sizing, which fields count as text — has a sensible default or is auto-detected.

### Output

One dataset, rows tagged by `type`:

| type | what it is |
|---|---|
| `kept` | A surviving, deduplicated record — the original fields plus `id` and `_duplicatesRemoved` (how many rows were folded into it). |
| `duplicate` | A removed row, with `duplicateOf` (the id of the record it duplicates), `similarity` (0-100), and `method` (`exact` or `near`). |
| `qualityReport` | One row: `overallScore` (0-100) plus a per-field breakdown of completeness, uniqueness, dominant type, type consistency, blank count and outlier count. |

The key-value store's `SUMMARY` reports counts, the LSH candidate-generation stats (how many of
the possible pairs were actually checked), and the overall quality score.

### Who uses it

Anyone about to embed a scraped corpus into a vector database and doesn't want the same chunk
indexed five times · data teams merging several scrapers' output on the same target · marketplaces
cleaning a product catalogue pulled from multiple feeds · researchers auditing a dataset's quality
before training on it.

# Actor input Schema

## `records` (type: `array`):

The dataset to clean, as an array of flat row objects. Combine freely with datasetIds. Nested arrays/objects per row are not supported.

## `datasetIds` (type: `array`):

Apify dataset IDs to pull additional rows from (e.g. the output of a scraping actor). Rows are appended to "records" before deduplication.

## `idField` (type: `string`):

Field to use as each record's identifier in the output (e.g. "url" or "sku"). Leave empty to auto-generate an id (R000001, R000002, ...).

## `textFields` (type: `array`):

Which fields hold free text to check for near-duplicates. Leave empty to auto-detect long string columns (average length above "Auto-detect text length threshold").

## `textFieldMinAvgLength` (type: `integer`):

When textFields is empty, a column is treated as free text if its average string length across the sampled rows is at least this many characters.

## `similarityThreshold` (type: `integer`):

Two records are treated as near-duplicates when their text overlap (Jaccard similarity over shingles) is at or above this. Higher = fewer false merges, more missed near-duplicates. Expressed 0-100; 80 means 0.80.

## `shingleSize` (type: `integer`):

Length of the word n-grams used to fingerprint each document. Smaller catches shorter edits but is noisier; larger is stricter. Short text automatically falls back to character shingles instead.

## `numHashes` (type: `integer`):

How many hash functions make up each document's MinHash signature. Higher gives a more accurate similarity estimate for candidate generation but costs more CPU. Must be >= numBands.

## `numBands` (type: `integer`):

How many bands the MinHash signature is split into for LSH bucketing. More bands catch weaker matches (higher recall, more candidate pairs to verify); fewer bands are stricter and faster.

## `maxBucketSize` (type: `integer`):

An LSH bucket larger than this is skipped when generating candidate pairs, to avoid an O(n²) blowup if many records hash into the same bucket (e.g. many rows with blank text).

## `includeKept` (type: `boolean`):

Emit the deduplicated, cleaned records (type: "kept").

## `includeRemoved` (type: `boolean`):

Emit every removed row (type: "duplicate") with the id of the record it duplicates, the similarity score, and whether it was an exact or near match.

## `includeQualityReport` (type: `boolean`):

Emit one summary row (type: "qualityReport") with per-field completeness/uniqueness/type-consistency/outliers and an overall 0-100 quality score.

## Actor input object example

```json
{
  "records": [
    {
      "title": "Best wireless headphones 2024",
      "body": "These wireless headphones deliver excellent sound quality and battery life for the price."
    },
    {
      "title": "Best wireless headphones 2024",
      "body": "These wireless headphones deliver excellent sound quality and battery life for the price."
    },
    {
      "title": "Best wireless headphones of 2024",
      "body": "These wireless headphones deliver great sound quality and long battery life for the price."
    },
    {
      "title": "Top budget laptops this year",
      "body": "A roundup of the best budget laptops you can buy right now, covering performance and battery."
    }
  ],
  "textFieldMinAvgLength": 40,
  "similarityThreshold": 80,
  "shingleSize": 3,
  "numHashes": 32,
  "numBands": 8,
  "maxBucketSize": 500,
  "includeKept": true,
  "includeRemoved": true,
  "includeQualityReport": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "records": [
        {
            "title": "Best wireless headphones 2024",
            "body": "These wireless headphones deliver excellent sound quality and battery life for the price."
        },
        {
            "title": "Best wireless headphones 2024",
            "body": "These wireless headphones deliver excellent sound quality and battery life for the price."
        },
        {
            "title": "Best wireless headphones of 2024",
            "body": "These wireless headphones deliver great sound quality and long battery life for the price."
        },
        {
            "title": "Top budget laptops this year",
            "body": "A roundup of the best budget laptops you can buy right now, covering performance and battery."
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/dataset-deduplicator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "records": [
        {
            "title": "Best wireless headphones 2024",
            "body": "These wireless headphones deliver excellent sound quality and battery life for the price.",
        },
        {
            "title": "Best wireless headphones 2024",
            "body": "These wireless headphones deliver excellent sound quality and battery life for the price.",
        },
        {
            "title": "Best wireless headphones of 2024",
            "body": "These wireless headphones deliver great sound quality and long battery life for the price.",
        },
        {
            "title": "Top budget laptops this year",
            "body": "A roundup of the best budget laptops you can buy right now, covering performance and battery.",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/dataset-deduplicator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "records": [
    {
      "title": "Best wireless headphones 2024",
      "body": "These wireless headphones deliver excellent sound quality and battery life for the price."
    },
    {
      "title": "Best wireless headphones 2024",
      "body": "These wireless headphones deliver excellent sound quality and battery life for the price."
    },
    {
      "title": "Best wireless headphones of 2024",
      "body": "These wireless headphones deliver great sound quality and long battery life for the price."
    },
    {
      "title": "Top budget laptops this year",
      "body": "A roundup of the best budget laptops you can buy right now, covering performance and battery."
    }
  ]
}' |
apify call that_red_bird/dataset-deduplicator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/dataset-deduplicator"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/seTCHdSRFWy3D9B8w/builds/hiUxapSn440GMmew5/openapi.json