Dataset Deduper

Exact and fuzzy dedupe for any dataset, with an audit trail for every decision and a cross-run ledger so tomorrow's run knows what today's already delivered.

Scraped data arrives with the same thing in it more than once: the same business under two spellings, the same product from two category pages, the same listing today and tomorrow. This actor collapses those, in two passes you control, and shows its work.

Part of the Broomwagon family: deterministic tools that watch scraped and AI data change. No LLM decides what matches, so the same input always produces the same output, and the AUDIT record explains every merge.

Two passes, both yours to configure

1. Exact, always on. Records match when the fields you name in exactFields agree after normalizing: case, punctuation, extra spaces, and word order are ignored. "The Blue Cafe" equals "blue cafe, the". Leave exactFields empty and the whole record has to match.
2. Fuzzy, opt in. For the near-duplicates normalization can't join, name fields in fuzzyFields and set a fuzzyThreshold. Similarity is averaged across the fields you named; a field missing on either side is skipped rather than counted as a difference. "Verve Coffee" at 2101 Pacific Ave and "Verve Coffee Roasters" at 2101 Pacific Avenue merge at 90. Two different cafés on the same street do not.
3. Cross-run, if you keep the memory. Anything delivered in an earlier run is dropped, so a daily scrape only ever hands you what you have not seen.

Guardrails worth knowing:

• vetoFields stop a fuzzy merge when two records disagree on something decisive, e.g. ["phone"] or ["sku"]. A record missing the field is never vetoed by it, and values that are the same number written differently (+1 831-600-7784 and 831-600-7784) count as agreeing.
• blockField tells the actor to only compare records sharing a value, e.g. "city". Without it, records are grouped by the distinctive words in their fuzzy fields, which works well until one word is everywhere ("Inc", "Ltd"); very large groups get capped and the run report says so.
• First occurrence wins, always. The record that arrives first is the one delivered, so the output does not depend on how the run was chunked.

Which deduper do I want

• This one when you want configurable matching over any shape of data, including fuzzy.
• Only New Items when you only care about "have I been given this before", identity is one exact field, and you want the cheapest possible per-record price.
• Google Maps Deduper when the data is places and you want the places rules (place IDs, phone identity, brand suffixes) already built in.

Stateless, with memory

The cross-run ledger lives in the MEMORY key-value record every run returns: a compressed Bloom filter of fingerprints, never your records. Pass it back next run (memory), or point the actor at a key-value store in your account (memoryStoreId) and scheduled runs chain themselves. Broomwagon stores nothing.

Aliases count. When a near-duplicate is merged away, its identity still goes into the ledger, so the same variant arriving tomorrow is recognized instead of sailing through.

Try it in two minutes (no scraper needed)

Leave the four sample records in Inline records and run. Two of them differ only in case and punctuation and merge on the exact pass; the Verve pair merges on the fuzzy pass. Read the AUDIT record to see which rule caught which, then set fuzzyThreshold to 99 and run again to watch the fuzzy merge stop happening.

Input

• datasetId | items | fileUrl: the records. A picked dataset wins.
• exactFields, fuzzyFields, fuzzyThreshold, vetoFields, blockField: matching, as above.
• ledgerCapacity: how many distinct records to size the cross-run ledger for.
• memory / memoryStoreId: the ledger.

Output

• Dataset: the surviving records, byte-for-byte as they arrived.
• AUDIT: the first 1,000 decisions, each with the action, the reason, what it matched, and the similarity score. This is the record to read when a merge surprises you.
• OUTPUT: run report. MEMORY: the ledger, to send back next run.

Pricing (pay-per-event)

Event	What you pay for	Price
record-processed	Per record examined	$0.001
apify-actor-start	Run start, per GB of run memory	$0.005

Worked example: deduping a 50,000-row scrape: 50,000 × $0.001 = $50 (plus a penny to start at the default 2 GB), once. Daily on a 5,000-row scrape: $5 per run, and every run after the first only hands downstream the records that are actually new. Cheaper per record than the places-specific deduper because it does not carry the places rules; cheaper still if all you need is "have I seen this", which is Only New Items at a fifth the price.

Set a max charge on any run (Maximum cost per run in Console, or ACTOR_MAX_TOTAL_CHARGE_USD via API) and the actor stops cleanly at your budget.

Integrations

• Apify Schedule (zero glue): create a key-value store once, put its ID in memoryStoreId, and schedule this actor right after your scraper.
• API: POST https://api.apify.com/v2/acts/broomwagon~dataset-deduper/runs with {"datasetId": "<scrape run's dataset>", "exactFields": ["name","address"], "memory": <last MEMORY record>}.
• n8n / Make: scraper node → this actor → your warehouse or CRM load. The load step stops re-upserting rows it already has.
• MCP / AI agents: callable as a tool via the Apify MCP server, so an agent can collapse a messy result set before reasoning over it, at a fixed cost per record instead of a token bill that scales with the mess.

Roadmap

Keep-richest merging (survivorship rules picking the most complete record rather than the first), and per-field merge strategies for combining duplicates instead of dropping them.

The Broomwagon family

This actor is one of nine deterministic post-processing tools from Broomwagon: the layer that follows your scrapers and agents, cleaning and watching what they produce. Same input, same output, every time.

• Scraper Output Monitor — catch the day your scrape silently breaks.
• Only New Items — deliver only records you have never delivered before.
• CSV Doctor — repair broken CSV files, with a report of every fix.
• LLM Output Guard — validate LLM and agent JSON against your schema.
• PII Redactor — strip emails, phones, SSNs, cards, and addresses, deterministically.
• Record Linker — fuzzy join two datasets that share no key.
• Google Maps Scraper Deduper — merge duplicate places across Google Maps scrapes.
• AI Search Visibility Monitor — track your brand's share of voice in AI answers.