Dataset Quality & Deduplication Engine

A deterministic utility for cleaning Apify datasets and structured JSON without an AI model or external API.

What it does

• exact duplicate detection
• bounded fuzzy duplicate detection with a hard comparison limit
• case, whitespace, and URL normalization
• common tracking-parameter removal
• missing-value and completeness analysis
• field type-consistency analysis
• email syntax validation only
• cleaned records, duplicate/rejected records, and a transparent quality score

Quick start

{
  "records": [
    { "name": " Acme  ", "url": "https://example.com/product/?utm_source=demo" },
    { "name": "acme", "url": "https://example.com/product/" }
  ],
  "keyFields": ["name", "url"],
  "fuzzyThreshold": 0.9
}

For an existing Apify dataset, provide datasetId instead of records.

Quality score

The score starts at 100 and applies documented penalties for duplicates, malformed records, missing values, invalid email syntax, and fields with inconsistent non-null types. It is a dataset-health indicator, not a guarantee that values are factually correct.

Pricing

This Actor is intended as an acquisition utility under pay-per-usage pricing: users pay only standard Apify platform resources and no developer event fee. The Actor uses 256 MB by default and caps rows and fuzzy comparisons to keep compute predictable.

Limitations

Fuzzy matching uses token similarity inside bounded buckets and may miss reordered or very short near-duplicates. Nested objects are preserved and deterministically serialized, but quality metrics focus on top-level fields. Email validation checks format only; it does not contact mail servers, verify ownership, enrich identities, or harvest addresses. Review cleaned output before replacing production data.