Dedupe and Merge Datasets: Only New Items

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Merge & Dedup Datasets: Remove Duplicates, Only New Items

charlesmarkette/merge-deduplicate-datasets-only-new-items

Merge any number of Apify datasets and remove duplicates by the fields you choose (or the whole item). Keep the first or last copy, or output only items not seen in earlier runs: ideal after a scheduled scraper. Streams millions of items. Duplicates are free; $0.0019 per 1,000 items output.

Data Deduplicator

parsebird/dataset-deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows while keeping the first or last occurrence. Supports case-insensitive matching and whitespace trimming.

CSV & Excel Dataset Merger / Splitter [Bulk Optimizer]

thecfmarketing/csv-xlsx-dataset-merger-pattern-splitter

High-speed tool to merge multiple CSV & Excel files or split large datasets into smaller chunks by rows, file size, or specific patterns. Clean duplicates, optimize data formats, and structure columns. The ultimate utility for handling millions of B2B cold email leads and data lists.

Dataset Transformer - Filter, Dedupe, Merge & Export CSV

seemuapps/dataset-transformer

Filter, deduplicate, merge, sort, rename and flatten Apify datasets from any scraper, then export clean rows to a dataset, CSV or JSON file.

Merge & Deduplicate Datasets - Reliable, Any Size

geese_chief/dataset-merge-dedup

Merge any number of Apify datasets into one and remove duplicates by the fields you choose (url, id, email...). Keep unique items or only the duplicates, pick output fields, tag each item with its source. Streams data, so millions of items work. Pick datasets from your storage: no permission errors.

Fetch public security.txt and trust-center pages. Try is Stripe’s file, not a pentest. $0.01 / vendor.

Compare 2-10 Apify datasets to detect changes, new/removed records, and duplicates. Features field-level diffs, smart merging, schema validation, data cleaning, and anomaly detection. Perfect for price monitoring, lead deduplication, and data quality tracking.

CSV Doctor | Repair Broken CSV Files, With a Fix Report

broomwagon/csv-doctor

Repair the CSV that will not load: wrong delimiters, unquoted values, ragged rows, blank and duplicate headers, mojibake. Get back rectangular typed records plus a report of exactly what changed. Nothing is invented and nothing is dropped, so ragged rows are named, never truncated.

Only New Items | Cross-Run Dedupe Ledger for Any Dataset

broomwagon/only-new-items

Run your scraper on a schedule and get back only what you have never seen before. Name the field that identifies an item, pipe any dataset in, and this actor holds back everything already delivered. Records pass through untouched. The ledger is yours, not a database you rent.