Dataset Deduplication & Merge Planner

Deduplicate, Merge & Transform Datasets

datacach/deduplicate-datasets

Merge multiple datasets, deduplicate items by a combination of fields, and apply custom transforms — powered by Polars.

Data Deduplicator

parsebird/dataset-deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows while keeping the first or last occurrence. Supports case-insensitive matching and whitespace trimming.

Remove duplicate rows from an Apify Dataset or inline JSON. Deduplicate by selected fields, trim strings, and write the cleaned rows to a new Dataset.

Use this Actor to monitor dataset diff and change changes and return decision-ready change signals. Compare two Apify datasets or inline snapshots and explain added, removed, and changed records plus field-level change frequencies.

Use this Actor to analyze dataset pii and sensitive data exposure and return decision-ready structured signals. Scan Apify datasets for likely personal or sensitive data patterns and return masked, field-level exposure evidence for privacy and pipeline controls.

Use this Actor to analyze dataset cardinality and anomaly profiler and return decision-ready structured signals. Profile field completeness, cardinality, type distributions, and numeric ranges, then compare them with a baseline to flag data-shape anomalies.

Use this Actor to monitor social bio positioning change changes and return decision-ready change signals. Track how brands and executives reposition themselves across recurring social-profile snapshots.

Use this Actor to analyze llm model pricing, deprecation and migration and return decision-ready structured signals. Track model pricing, deprecations, context and availability changes across AI providers, then rank migration risk, cost impact and replacement priorities.

Use this Actor to monitor dataset schema drift and contract changes and return decision-ready change signals. Detect top-level field additions, removals, and type changes in Apify datasets with deterministic schema fingerprints and optional pipeline stopping.

Use this Actor to analyze warn layoff and corporate distress and return decision-ready structured signals. Turn WARN layoff notices into company-level layoff acceleration, affected-headcount, geographic concentration and distress signals.