Pricing

from $10.00 / 1,000 1,000 items processeds

Try for free

Go to Apify Store

Dataset Deduplicate & Merge (Fuzzy)

Try for free

Merge multiple Apify datasets or CSV/JSON/XLSX files and remove duplicates with normalised or fuzzy matching. Only-new mode returns just items you have not seen before.

Pricing

from $10.00 / 1,000 1,000 items processeds

Rating

0.0

(0)

Developer

Larchrow

Actor stats

Bookmarked

Total users

Monthly active users

8 days ago

Last modified

Categories

Developer tools

Automation

Security

EN · Merge any number of Apify datasets (plus JSON/CSV URLs) and remove duplicates that simple exact-match dedupers miss:

Normalisers per field: url (drops www, trailing slash, utm_/gclid/fbclid tracking params, sorts query), phone (E.164, German 0-prefix → +49), email (case, Gmail dots and +tags), company (drops GmbH, UG, AG, KG, Ltd, LLC, Inc … and punctuation), text (case, accents, whitespace), exact.
Fuzzy merge of near-identical names (typos, missing letters) with a safety rule: values that differ by a real word – e.g. Hospital Rijeka vs Hospital Osijek, NL Nord vs NL Süd – are never merged.
Merge strategies: keep first, keep last, fill empty fields from duplicates, or fill and union list fields. Every output item gets __dupes (how many records were merged) and __source.
Only-new mode: remembers keys across runs, so a scheduled scraper + this actor outputs only records you have never seen.
Streaming reads in 10k pages, works with datasets you own (no extra permissions needed). Stats in the key-value store record STATS.

Pricing: pay per event, per 1,000 input items processed. No charge for empty runs.

DE · Führt beliebig viele Apify-Datasets (und JSON/CSV-URLs) zusammen und entfernt Duplikate, die einfache Exakt-Vergleiche übersehen: URL-, Telefon-, E-Mail- und Firmennamen-Normalisierung (GmbH & Co. KG, UG, AG …), unscharfer Abgleich mit Schutzregel gegen Fehlzusammenführungen, vier Merge-Strategien und ein „nur neue“-Modus für Monitoring.

Hinweis: Das Tool verarbeitet nur Daten, die du selbst einbringst. Für personenbezogene Daten in deinen Datasets bist du Verantwortlicher im Sinne der DSGVO.

Example input:

{"datasetIds":["<run-1-dataset>","<run-2-dataset>"],"fields":["website:url"],"fuzzyField":"name:company","mergeStrategy":"fill"}

New (merged from dataset-tools): XLSX/JSONL input, mergeStrategy: most-complete (keeps the duplicate with the most filled fields), renameFields, fieldsToKeep, fieldsToDrop, requiredFields. Neu: XLSX/JSONL-Eingabe, Strategie „most-complete“ (vollständigster Eintrag), Felder umbenennen/auswählen/entfernen, Pflichtfelder.

Merge Datasets & Remove Duplicates — Deduplicate, Filter, Excel

tidydata/merge-dedupe-datasets

Merge datasets and remove duplicates in one call: combine Apify datasets, CSV, Excel, JSON or Google Sheets, dedupe by any field (smart URL/email/phone matching), filter, sort, keep only new rows, export CSV/Excel/JSON. Works from AI agents via Apify MCP (Claude, ChatGPT), n8n and the API.

tidydata

Deduplicate, Merge & Transform Datasets

datacach/deduplicate-datasets

Merge multiple datasets, deduplicate items by a combination of fields, and apply custom transforms — powered by Polars.

DataCach

Monitoring Checker Duplication

apify/monitoring-checker-duplication

The Monitoring checker duplication is a part of the Apify Monitoring Suite (apify/monitoring). See its readme for more information and how to use this.

Apify

5.0

(5)

Merge, Dedup & Transform Datasets – Filter, CSV, Excel

gazidev/dataset-filter-transform

No-code filter, merge and dedupe for Apify datasets, CSV, XLSX and JSON files: combine several datasets, remove duplicates by fields, filter rules (equals, contains, regex, ranges, dates, AND/OR), only-new-since-last-run, rename/select/drop, flatten, sort. Export CSV/XLSX/JSON. Pay per row.

Cemal Atakli

Remove Duplicates & Merge Datasets: Dedup, Only New

brasspage/dataset-dedup-merge

Dedup and merge Apify datasets: removes duplicate items by any fields, or by the whole item, and returns the unique items (or only the duplicates) with their original fields. Can skip items seen in earlier runs, for daily new results. Works with limited permissions. $0.02 per 1,000 items.

Brasspage

AI Dataset Search

dtrungtin/ai-dataset-search

The actor lets you query any dataset using **plain English** instead of code. Point it at a dataset — for example the output of a scraper run — describe what you're looking for ("videos about race day at the Belgian Grand Prix", "items with more than 100k plays"), and it returns the matching items.

Tin

Dataset Deduplicator & Cleaner

starshaped_bullsnake/dataset-deduplicator-cleaner

Remove duplicate rows from an Apify Dataset or inline JSON. Deduplicate by selected fields, trim strings, and write the cleaned rows to a new Dataset.

Starshape Tools

Dataset Transformer — Filter, Dedupe, Sort & Reshape Data

nexgensignal/dataset-transformer-api

Transform Apify datasets or JSON / CSV links: filter rows, remove duplicates, sort, flatten nested fields, unwind lists, pick and rename columns. Merge several datasets into one clean result for export as JSON, CSV or Excel. Part of NexGen Signal — data tools and official-source data products

NexGen Signal

Dataset Cleaner & Deduplicator | JSON Data QA

egeusta/dataset-quality-deduplication-engine

Clean JSON records and Apify datasets with exact and bounded fuzzy deduplication. Normalize text and URLs, detect missing fields and email-format issues, and export cleaned records, duplicates and a quality summary.

Scopras Data Labs

Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset

broomwagon/dataset-deduper

Collapse duplicates in any dataset, in two passes you control. Exact matching ignores case, punctuation, and word order; an optional fuzzy pass catches the near-duplicates normalization cannot. Every decision is explained in an audit trail, and a cross-run ledger remembers what you already got.