Pricing

Pay per usage

Try for free

Go to Apify Store

Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset

Try for free

Collapse duplicates in any dataset, in two passes you control. Exact matching ignores case, punctuation, and word order; an optional fuzzy pass catches the near-duplicates normalization cannot. Every decision is explained in an audit trail, and a cross-run ledger remembers what you already got.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Brandon Mensing

Actor stats

Bookmarked

Total users

Monthly active users

a month ago

Last modified

Categories

Developer tools

Agents

Automation

You can access the Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "items": [
12        {
13            "name": "Blue Bottle Coffee",
14            "address": "300 Webster St, Oakland, CA",
15            "phone": "+1 510-653-3394"
16        },
17        {
18            "name": "blue bottle coffee",
19            "address": "300 webster st, oakland ca",
20            "phone": "(510) 653-3394"
21        },
22        {
23            "name": "Verve Coffee",
24            "address": "2101 Pacific Ave, Santa Cruz, CA",
25            "phone": "+1 831-600-7784"
26        },
27        {
28            "name": "Verve Coffee Roasters",
29            "address": "2101 Pacific Avenue, Santa Cruz",
30            "phone": "831-600-7784"
31        }
32    ],
33    "exactFields": [
34        "name",
35        "address"
36    ],
37    "fuzzyFields": [
38        "name",
39        "address"
40    ],
41    "vetoFields": [
42        "phone"
43    ]
44};
45
46// Run the Actor and wait for it to finish
47const run = await client.actor("broomwagon/dataset-deduper").call(input);
48
49// Fetch and print Actor results from the run's dataset (if any)
50console.log('Results from dataset');
51console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
52const { items } = await client.dataset(run.defaultDatasetId).listItems();
53items.forEach((item) => {
54    console.dir(item);
55});
56
57// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Dataset Deduper: Exact and Fuzzy Duplicate Removal API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset API in Python

Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset API through CLI

Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset OpenAPI definition

Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset API

Fuzzy Search Dataset Actor

dtrungtin/fuzzy-search-dataset-actor

Search any Apify dataset using typo-tolerant fuzzy matching.

Tin

Only New Items | Cross-Run Dedupe Ledger for Any Dataset

broomwagon/only-new-items

Run your scraper on a schedule and get back only what you have never seen before. Name the field that identifies an item, pipe any dataset in, and this actor holds back everything already delivered. Records pass through untouched. The ledger is yours, not a database you rent.

Brandon Mensing

Google Maps Scraper Deduper | Merge Duplicates | Cross-Run

broomwagon/google-maps-deduper

Merge duplicate places from Google Maps scraper runs — by placeId, phone, or fuzzy name+address — including duplicates from previous runs. Only get what's new, cut enrichment costs, and keep a full audit trail. Stateless: you hold the memory, we store nothing.

Brandon Mensing

Dataset Deduplicator & Cleaner

eszetael_lab/dataset-deduplicator-cleaner

Returns your records with duplicates removed and values normalized — same fields as the input, nothing added or renamed. Exact, normalized or fuzzy matching on keys you choose. A data quality report lands under QUALITY_REPORT. Takes a dataset ID or inline JSON, so it chains after any scraper.

Radosław Szal

🧽 Dataset Deduplicator - Clean Any Scraped Dataset

that_red_bird/dataset-deduplicator

🧽 Upload any dataset — scraped rows, a RAG corpus, an export — and get back exact AND near-duplicates removed, plus a per-field data quality report. ✅ MinHash + LSH banding finds near-duplicate text at scale without O(n²) pairwise comparison.

mohamed alaya

Dataset Cleaner & Deduplicator

luminous_i/dataset-cleaner-deduplicator

Clean, normalize and deduplicate any dataset: field names, whitespace, empty values, numbers, plus exact and fuzzy duplicate removal.

Steve

Dataset Toolkit — Dedupe, Merge, Clean, Diff

glueworks/dataset-toolkit

Post-process any Apify dataset: remove duplicates, merge multiple datasets, clean and normalize fields, or diff two crawls. Outputs a new dataset plus a summary record.