Dataset Deduplicator & Quality Scorer

Upload any dataset — scraped rows, a RAG corpus you're about to embed, a CRM or catalogue export, the combined output of three different scrapers pointed at the same site — and get back a clean version with exact duplicates gone, near-duplicates gone, and a per-field data quality report telling you how dirty the input actually was.

What it does

1. Exact duplicates — a sha256 content hash over every record's sorted fields catches byte-identical rows in a single pass, no comparisons needed.
2. Near duplicates — the harder problem. Two scraped rows describing the same article/listing are rarely byte-identical (different whitespace, a re-scrape days later, a paragraph edited). This actor implements MinHash + LSH banding from scratch (src/lsh.js) to find those pairs without comparing every record to every other record:
   • each text field is broken into overlapping word shingles (falling back to character shingles for short strings),
   • each document's shingle set is summarised into a short MinHash signature,
   • the signature is split into bands and hashed into buckets — only records sharing a bucket become "candidates",
   • every candidate is then verified with an exact Jaccard similarity check on the real shingle sets before it counts as a duplicate. This keeps the run close to O(n) instead of O(n²): a bucket of size b only costs b²/2 comparisons, and unrelated records never share a bucket at all.
3. Quality report — for every field: completeness %, uniqueness %, dominant type + type consistency %, blank count, and (for numeric fields) an IQR-based outlier count. These roll up into one 0-100 quality score that also factors in how much of the dataset was duplicated.

Honest limits — read before you rely on the near-duplicate count

MinHash + LSH is probabilistic by design. Banding trades recall for speed: a genuine near-duplicate pair can fail to land in the same bucket in any band and simply never becomes a candidate — a false negative. Raising numBands (or lowering numHashes per band) increases recall at the cost of more candidate pairs to verify and more CPU. There are no false positives from this stage, though: every candidate is re-checked with an exact Jaccard similarity on the real shingle sets before being reported, so nothing is flagged as a duplicate on the MinHash estimate alone.

Other things this actor will not do:

• It does not understand meaning — two records that say the same thing in completely different words will not be caught. This is lexical (shingle-overlap) similarity, not semantic similarity.
• Exact-duplicate hashing is structural: two JSON objects with the same field values in a different nested-object key order will not hash identically. Top-level field order does not matter; deeply nested object key order does.
• It caps at 200,000 rows per run and skips any oversized LSH bucket (e.g. thousands of rows with blank text all landing in one bucket) to avoid a runaway comparison count.
• Auto-detected text fields require the column to be a string in at least 30% of sampled rows with an average length over textFieldMinAvgLength — a dataset with no such column (e.g. purely numeric) skips the near-duplicate stage entirely and runs exact-dedup + quality report only.

Input

{
  "records": [
    { "title": "Best wireless headphones 2024", "body": "Great sound and battery life for the price." },
    { "title": "Best wireless headphones 2024", "body": "Great sound and battery life for the price." },
    { "title": "Best wireless headphones of 2024", "body": "Great sound quality and long battery life for the price." }
  ],
  "similarityThreshold": 80
}

Only records and/or datasetIds is required. Everything else — similarity threshold, shingle size, MinHash/LSH sizing, which fields count as text — has a sensible default or is auto-detected.

Output

One dataset, rows tagged by type:

type	what it is
kept	A surviving, deduplicated record — the original fields plus id and _duplicatesRemoved (how many rows were folded into it).
duplicate	A removed row, with duplicateOf (the id of the record it duplicates), similarity (0-100), and method (exact or near).
qualityReport	One row: overallScore (0-100) plus a per-field breakdown of completeness, uniqueness, dominant type, type consistency, blank count and outlier count.

The key-value store's SUMMARY reports counts, the LSH candidate-generation stats (how many of the possible pairs were actually checked), and the overall quality score.

Who uses it

Anyone about to embed a scraped corpus into a vector database and doesn't want the same chunk indexed five times · data teams merging several scrapers' output on the same target · marketplaces cleaning a product catalogue pulled from multiple feeds · researchers auditing a dataset's quality before training on it.