Dataset Quality & Deduplication Engine

A deterministic utility for cleaning Apify datasets and structured JSON without an AI model or external API.

What it does

• exact duplicate detection
• bounded fuzzy duplicate detection with a hard comparison limit
• case, whitespace, and URL normalization
• common tracking-parameter removal
• missing-value and completeness analysis
• field type-consistency analysis
• email syntax validation only
• cleaned records, duplicate/rejected records, and a transparent quality score

Quick start

{
  "records": [
    { "name": " Acme  ", "url": "https://example.com/product/?utm_source=demo" },
    { "name": "acme", "url": "https://example.com/product/" }
  ],
  "keyFields": ["name", "url"],
  "fuzzyThreshold": 0.9
}

For an existing Apify dataset, pick it in the datasetId field (resource picker, read-only access is granted to the selected dataset only, private datasets work). Via API, pass the dataset ID as a string.

Quality score

The score starts at 100 and applies documented penalties for duplicates, malformed records, missing values, invalid email syntax, and fields with inconsistent non-null types. It is a dataset-health indicator, not a guarantee that values are factually correct.

Pricing

Pay per event: $0.10 per 1,000 input records analyzed, plus the displayed Actor-start event ($0.00005 per GB, minimum one event). Each input row is counted once, including duplicates or rejected records. Summary and output rows are not billed separately. Summary-only mode still analyzes and bills input records. The Store Pricing tab is authoritative.

Free Apify plans receive a sample of up to 100 input rows per run. An Apify paid plan enables larger batches, within maxRows and the run spending limit. The Actor stops before processing more rows than the remaining budget allows.

Clean scraper exports before your next workflow

1. Export product records with Product Data Extractor.
2. Supply that run’s datasetId here, with keyFields such as url or SKU.
3. Inspect duplicate and rejected rows before importing the cleaned output into your CRM, spreadsheet, or catalog.

Save the input as an Apify Task for repeat jobs. Use a Schedule for time-based runs or an integration after an upstream dataset finishes. This Actor does not modify the source dataset.

Limitations

Fuzzy matching uses token similarity inside bounded buckets and may miss reordered or very short near-duplicates. Nested objects are preserved and deterministically serialized, but quality metrics focus on top-level fields. Email validation checks format only; it does not contact mail servers, verify ownership, enrich identities, or harvest addresses. Review cleaned output before replacing production data.

FAQ

How do I remove duplicates from an Apify dataset? Pick the dataset (or paste JSON records), choose the key fields, and run. You get cleaned records, the duplicates that were removed and a quality summary.

Does it use AI? No. Exact and bounded fuzzy matching are deterministic, so the same input always gives the same result.

How much does it cost? $0.10 per 1,000 records processed. The Pricing tab is authoritative.

Can I run it through the API or on a schedule? Yes. Save your input as a task, schedule it in Apify Console, or call it from the Apify API. Results export as JSON, CSV or Excel and connect to Google Sheets, Slack, Make, Zapier or a webhook.

Related Actors

• E-commerce Product Scraper
• ATS Job Scraper
• Company Website Contact Enricher