Dataset Dedupe + Schema Cleanup

Clean, deduplicate, normalize, and schema-check JSON datasets with structured input.

What it does

• Removes duplicate records using selected keys.
• Trims and normalizes supported fields.
• Validates records against optional schema rules.
• Separates invalid records from clean records.
• Returns a summary of the cleanup.
• Writes clean records to the run's default Apify Dataset.
• Writes the complete result to OUTPUT in the run's default Key-value store.

Example input

{
  "records": [
    {"email": " alice@example.com ", "name": "Alice"},
    {"email": "alice@example.com", "name": "Alice"},
    {"email": "bob@example.com", "name": "Bob"}
  ],
  "dedupeKeys": ["email"],
  "normalization": {
    "trimStrings": true
  }
}

Expected result

• 3 input records
• 2 clean records
• 1 duplicate removed
• 0 invalid records

Output

The OUTPUT object contains:

• cleanRecords: normalized records that passed validation and were not duplicates.
• removedDuplicates: records removed because their selected dedupe-key values matched an earlier record.
• invalidRecords: records that failed the optional schema or rules.
• summary: counts for input, clean, duplicate, and invalid records.

Clean records are also written as individual items to the default Apify Dataset for tabular inspection and downstream integrations.

Limits and safety

• Maximum 5,000 records.
• Maximum 1 MB serialized input.
• Bounded execution.
• No URLs, scraping, or arbitrary code.
• No external network access is required.

Use in Apify

In Apify Console, open the Actor, enter the JSON input, and click Start. After the run completes, use the Dataset tab for clean records and the Key-value store tab to open the OUTPUT record.

For API or SDK runs, send the same JSON object as the Actor input. Read the run's default Dataset for cleanRecords, and retrieve the default Key-value store record named OUTPUT for the complete result. The Actor's generated input and output schemas are also available in Console and through the Apify API.