# Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset (`broomwagon/dataset-deduper`) Actor
Collapse duplicates in any dataset, in two passes you control. Exact matching ignores case, punctuation, and word order; an optional fuzzy pass catches the near-duplicates normalization cannot. Every decision is explained in an audit trail, and a cross-run ledger remembers what you already got.
- **URL**: https://apify.com/broomwagon/dataset-deduper.md
- **Developed by:** [Brandon Mensing](https://apify.com/broomwagon) (community)
- **Categories:** Developer tools, Agents, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $1.00 / 1,000 record examineds
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Dataset Deduper
> Exact and fuzzy dedupe for any dataset, with an audit trail for every decision and a cross-run ledger so tomorrow's run knows what today's already delivered.
Scraped data arrives with the same thing in it more than once: the same business under
two spellings, the same product from two category pages, the same listing today and
tomorrow. This actor collapses those, in two passes you control, and shows its work.
Part of the **Broomwagon** family: deterministic tools that watch scraped and AI data
change. No LLM decides what matches, so the same input always produces the same output,
and the AUDIT record explains every merge.
### Two passes, both yours to configure
1. **Exact, always on.** Records match when the fields you name in `exactFields` agree
after normalizing: case, punctuation, extra spaces, and word order are ignored.
"The Blue Cafe" equals "blue cafe, the". Leave `exactFields` empty and the whole
record has to match.
2. **Fuzzy, opt in.** For the near-duplicates normalization can't join, name fields in
`fuzzyFields` and set a `fuzzyThreshold`. Similarity is averaged across the fields
you named; a field missing on either side is skipped rather than counted as a
difference. "Verve Coffee" at 2101 Pacific Ave and "Verve Coffee Roasters" at 2101
Pacific Avenue merge at 90. Two different cafés on the same street do not.
3. **Cross-run, if you keep the memory.** Anything delivered in an earlier run is
dropped, so a daily scrape only ever hands you what you have not seen.
Guardrails worth knowing:
- **`vetoFields`** stop a fuzzy merge when two records disagree on something decisive,
e.g. `["phone"]` or `["sku"]`. A record missing the field is never vetoed by it, and
values that are the same number written differently (`+1 831-600-7784` and
`831-600-7784`) count as agreeing.
- **`blockField`** tells the actor to only compare records sharing a value, e.g.
`"city"`. Without it, records are grouped by the distinctive words in their fuzzy
fields, which works well until one word is everywhere ("Inc", "Ltd"); very large
groups get capped and the run report says so.
- **First occurrence wins**, always. The record that arrives first is the one delivered,
so the output does not depend on how the run was chunked.
### Which deduper do I want
- **This one** when you want configurable matching over any shape of data, including
fuzzy.
- **Only New Items** when you only care about "have I been given this before", identity
is one exact field, and you want the cheapest possible per-record price.
- **Google Maps Deduper** when the data is places and you want the places rules
(place IDs, phone identity, brand suffixes) already built in.
### Stateless, with memory
The cross-run ledger lives in the `MEMORY` key-value record every run returns: a
compressed Bloom filter of fingerprints, never your records. Pass it back next run
(`memory`), or point the actor at a key-value store in **your** account
(`memoryStoreId`) and scheduled runs chain themselves. Broomwagon stores nothing.
Aliases count. When a near-duplicate is merged away, its identity still goes into the
ledger, so the same variant arriving tomorrow is recognized instead of sailing through.
### Try it in two minutes (no scraper needed)
Leave the four sample records in **Inline records** and run. Two of them differ only in
case and punctuation and merge on the exact pass; the Verve pair merges on the fuzzy
pass. Read the `AUDIT` record to see which rule caught which, then set
`fuzzyThreshold` to 99 and run again to watch the fuzzy merge stop happening.
### Input
- `datasetId` | `items` | `fileUrl`: the records. A picked dataset wins.
- `exactFields`, `fuzzyFields`, `fuzzyThreshold`, `vetoFields`, `blockField`: matching,
as above.
- `ledgerCapacity`: how many distinct records to size the cross-run ledger for.
- `memory` / `memoryStoreId`: the ledger.
### Output
- Dataset: the surviving records, byte-for-byte as they arrived.
- `AUDIT`: the first 1,000 decisions, each with the action, the reason, what it matched,
and the similarity score. This is the record to read when a merge surprises you.
- `OUTPUT`: run report. `MEMORY`: the ledger, to send back next run.
### Pricing (pay-per-event)
| Event | What you pay for | Price |
| --- | --- | --- |
| `record-processed` | Per record examined | $0.001 |
| `apify-actor-start` | Run start, per GB of run memory | $0.005 |
**Worked example:** deduping a 50,000-row scrape: 50,000 × $0.001 = **$50** (plus a
penny to start at the default 2 GB), once.
Daily on a 5,000-row scrape: $5 per run, and every run after the first only hands
downstream the records that are actually new. Cheaper per record than the places-specific
deduper because it does not carry the places rules; cheaper still if all you need is
"have I seen this", which is **Only New Items** at a fifth the price.
Set a max charge on any run (`Maximum cost per run` in Console, or
`ACTOR_MAX_TOTAL_CHARGE_USD` via API) and the actor stops cleanly at your budget.
### Integrations
- **Apify Schedule (zero glue):** create a key-value store once, put its ID in
`memoryStoreId`, and schedule this actor right after your scraper.
- **API:** `POST https://api.apify.com/v2/acts/broomwagon~dataset-deduper/runs` with
`{"datasetId": "", "exactFields": ["name","address"], "memory": }`.
- **n8n / Make:** scraper node → this actor → your warehouse or CRM load. The load step
stops re-upserting rows it already has.
- **MCP / AI agents:** callable as a tool via the Apify MCP server, so an agent can
collapse a messy result set before reasoning over it, at a fixed cost per record
instead of a token bill that scales with the mess.
### Roadmap
Keep-richest merging (survivorship rules picking the most complete record rather than
the first), and per-field merge strategies for combining duplicates instead of dropping
them.
### The Broomwagon family
This actor is one of nine deterministic post-processing tools from [Broomwagon](https://apify.com/broomwagon): the layer that follows your scrapers and agents, cleaning and watching what they produce. Same input, same output, every time.
- [Scraper Output Monitor](https://apify.com/broomwagon/scraper-output-monitor) — catch the day your scrape silently breaks.
- [Only New Items](https://apify.com/broomwagon/only-new-items) — deliver only records you have never delivered before.
- [CSV Doctor](https://apify.com/broomwagon/csv-doctor) — repair broken CSV files, with a report of every fix.
- [LLM Output Guard](https://apify.com/broomwagon/llm-output-guard) — validate LLM and agent JSON against your schema.
- [PII Redactor](https://apify.com/broomwagon/pii-redactor) — strip emails, phones, SSNs, cards, and addresses, deterministically.
- [Record Linker](https://apify.com/broomwagon/record-linker) — fuzzy join two datasets that share no key.
- [Google Maps Scraper Deduper](https://apify.com/broomwagon/google-maps-deduper) — merge duplicate places across Google Maps scrapes.
- [AI Search Visibility Monitor](https://apify.com/broomwagon/ai-search-visibility-monitor) — track your brand's share of voice in AI answers.
# Actor input Schema
## `datasetId` (type: `string`):
Any actor's output dataset. Pipe it straight in; a picked dataset always wins over the sample records below.
## `items` (type: `array`):
Records provided inline as a JSON array, for trying the actor out before wiring a scraper up. Ignored when a dataset is picked above.
## `fileUrl` (type: `string`):
URL of a JSON or NDJSON file of records.
## `exactFields` (type: `array`):
Fields that must match for two records to be the same. Compared after normalizing: case, punctuation, extra spaces, and word order are ignored, so "The Blue Cafe" equals "blue cafe, the". Leave empty to compare the whole record.
## `fuzzyFields` (type: `array`):
Fields compared by similarity for records the exact pass did not catch, e.g. \["name", "address"]. Leave empty to turn fuzzy matching off entirely.
## `fuzzyThreshold` (type: `integer`):
How close is close enough, 0 to 100, averaged across the fuzzy fields. 90 is a good starting point: high enough to reject different businesses, loose enough to join abbreviations and suffixes. Raise it if you see wrong merges in the AUDIT record.
## `vetoFields` (type: `array`):
Fields where two different non-empty values forbid a fuzzy merge, e.g. \["phone"] or \["sku"]. A record missing the field is never vetoed by it. This is how you stop a kiosk inside a venue from merging into the venue.
## `blockField` (type: `string`):
Only compare records that share this field's value, e.g. "city" or "category". Leave empty and the actor groups by the distinctive words in your fuzzy fields instead, which is usually fine but caps very large groups.
## `ledgerCapacity` (type: `integer`):
How many distinct records the cross-run ledger is sized for. Going far past this raises the rate at which a new record is mistaken for one already delivered. The run report always tells you where you stand.
## `memory` (type: `object`):
An object beginning with "memoryVersion", copied from the MEMORY record in your last run's key-value store (not your records, those go in the fields above). Without it, each run dedupes only within itself.
## `memoryStoreId` (type: `string`):
Optional: a key-value store in YOUR account. The actor reads the ledger from it before running and writes the update back after, so scheduled runs chain automatically with nothing to copy by hand.
## Actor input object example
```json
{
"items": [
{
"name": "Blue Bottle Coffee",
"address": "300 Webster St, Oakland, CA",
"phone": "+1 510-653-3394"
},
{
"name": "blue bottle coffee",
"address": "300 webster st, oakland ca",
"phone": "(510) 653-3394"
},
{
"name": "Verve Coffee",
"address": "2101 Pacific Ave, Santa Cruz, CA",
"phone": "+1 831-600-7784"
},
{
"name": "Verve Coffee Roasters",
"address": "2101 Pacific Avenue, Santa Cruz",
"phone": "831-600-7784"
}
],
"exactFields": [
"name",
"address"
],
"fuzzyFields": [
"name",
"address"
],
"fuzzyThreshold": 90,
"vetoFields": [
"phone"
],
"ledgerCapacity": 100000
}
```
# Actor output Schema
## `deduped` (type: `string`):
The surviving records, passed through untouched. First occurrence wins.
## `audit` (type: `string`):
Why each record was kept, merged, or dropped, with the matched record and similarity score.
## `memory` (type: `string`):
The ledger of everything delivered so far. Send it back as `memory` on the next run, or skip this by using `memoryStoreId`.
## `report` (type: `string`):
Records seen, kept, merged within the run, and dropped as already delivered.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"items": [
{
"name": "Blue Bottle Coffee",
"address": "300 Webster St, Oakland, CA",
"phone": "+1 510-653-3394"
},
{
"name": "blue bottle coffee",
"address": "300 webster st, oakland ca",
"phone": "(510) 653-3394"
},
{
"name": "Verve Coffee",
"address": "2101 Pacific Ave, Santa Cruz, CA",
"phone": "+1 831-600-7784"
},
{
"name": "Verve Coffee Roasters",
"address": "2101 Pacific Avenue, Santa Cruz",
"phone": "831-600-7784"
}
],
"exactFields": [
"name",
"address"
],
"fuzzyFields": [
"name",
"address"
],
"vetoFields": [
"phone"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("broomwagon/dataset-deduper").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"items": [
{
"name": "Blue Bottle Coffee",
"address": "300 Webster St, Oakland, CA",
"phone": "+1 510-653-3394",
},
{
"name": "blue bottle coffee",
"address": "300 webster st, oakland ca",
"phone": "(510) 653-3394",
},
{
"name": "Verve Coffee",
"address": "2101 Pacific Ave, Santa Cruz, CA",
"phone": "+1 831-600-7784",
},
{
"name": "Verve Coffee Roasters",
"address": "2101 Pacific Avenue, Santa Cruz",
"phone": "831-600-7784",
},
],
"exactFields": [
"name",
"address",
],
"fuzzyFields": [
"name",
"address",
],
"vetoFields": ["phone"],
}
# Run the Actor and wait for it to finish
run = client.actor("broomwagon/dataset-deduper").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"items": [
{
"name": "Blue Bottle Coffee",
"address": "300 Webster St, Oakland, CA",
"phone": "+1 510-653-3394"
},
{
"name": "blue bottle coffee",
"address": "300 webster st, oakland ca",
"phone": "(510) 653-3394"
},
{
"name": "Verve Coffee",
"address": "2101 Pacific Ave, Santa Cruz, CA",
"phone": "+1 831-600-7784"
},
{
"name": "Verve Coffee Roasters",
"address": "2101 Pacific Avenue, Santa Cruz",
"phone": "831-600-7784"
}
],
"exactFields": [
"name",
"address"
],
"fuzzyFields": [
"name",
"address"
],
"vetoFields": [
"phone"
]
}' |
apify call broomwagon/dataset-deduper --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,broomwagon/dataset-deduper"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/5dnC0a2cKa8Gh1Dxp/builds/kQk48mU4WoyIQu5JB/openapi.json