Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places

Scheduled scrapes keep returning the same job postings, leads and places. This Actor takes any Apify dataset (or a CSV / JSON file, or pasted rows) and gives you back only the unique rows, the removed duplicates with the reason, and, if you switch it on, only the rows that are new since the last run. No login, no API key, no AI.

Why this one

• Understands scraped data. The same job with ?utm_source=... or refId=... tracking junk is one job; a LinkedIn job link is reduced to its job id; www., trailing slashes and URL case are ignored.
• Presets for the common cases. jobs (same URL, or same title + company + location), places (Google place id, or title + address + phone, or URL), products, leads, posts. Auto-detect picks the right one from your columns.
• Three ways to find duplicates. Key fields (exact, fast), whole-record, or fuzzy near-duplicate text matching (re-posts and lightly reworded copies) with MinHash + LSH, verified exactly.
• Only new items across runs. Turn on rememberAcrossRuns for a scheduled scrape and each run returns only rows it has never delivered before. The memory is a named key-value store you can inspect or delete.
• Built for size. Dataset pages are streamed; 1,000,000 rows were processed in about 10 seconds of CPU in testing. Memory is roughly 100 bytes per row for key modes and 260 bytes with fuzzy matching.
• Honest billing. You pay per unique row delivered. Removed duplicates and the report are free. Runs stop cleanly at your maximum charge.

Input

Field	Default	What it does
datasetIds	[]	Datasets (pick one or pass IDs) to clean. Several datasets are merged, which also removes duplicates between runs and sources.
records / sourceUrl / csvText	built-in job sample	Alternative sources: pasted JSON rows, a CSV / JSON / NDJSON URL, pasted CSV.
mode	keys	keys (key fields match), wholeRecord (identical rows), fuzzy (keys plus near-duplicate text).
preset	auto	jobs, places, products, leads, posts or auto-detect.
keyFields	[]	Your own keys. "url", "url|link" (first filled column) or "title+company+location" (combined). A row is a duplicate if any entry matches.
fuzzyFields / similarityThreshold	auto / 85	Fuzzy mode: which text columns, and how similar (0-100) counts as a duplicate.
keepPolicy	first	Keep the first, the last or the most complete row of each duplicate group.
outputMode	kept	kept unique rows, removed duplicates only, or all tagged with _dedupeStatus.
rememberAcrossRuns / memoryName	false / default	Only deliver rows not seen in earlier runs. Use one memory name per scheduled scrape.
normalizeUrls, caseSensitive, ignoreFields	true, false, []	Comparison details.

Started with an empty input, it cleans a 15-row job-board sample (6 duplicates: tracking-parameter copies, a LinkedIn slug variant and re-posts on another board) and returns 9 unique postings.

Output

Default dataset: the unique rows, unchanged, ready to use.

{ "title": "Senior Data Engineer", "company": "Northwind Analytics", "location": "Austin, TX",
  "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz",
  "source": "linkedin" }

With outputMode: all or removed, rows carry the audit trail:

{ "title": "Senior Data Engineer", "company": "Northwind Analytics", "url": "https://careers.northwind-analytics.com/jobs/4471",
  "_dedupeStatus": "duplicate", "_dedupeRow": 3, "_dedupeDuplicateOfRow": 1,
  "_dedupeReason": "same title+company+location", "_dedupeSimilarity": 100 }

Key-value store: SUMMARY (rows in, kept, removed by key / near-duplicate / identical, seen in earlier runs, the keys used, the biggest duplicate groups, warnings) and REMOVED_DUPLICATES (row number, which row it duplicates, why).

Typical setups

• Daily job scrape, new postings only: datasetIds: [<today's run>], preset: jobs, rememberAcrossRuns: true, memoryName: "linkedin-data-engineer-austin". Schedule it after the scraper and read the dataset.
• Google Maps leads from overlapping searches: datasetIds: [<run 1>, <run 2>], preset: places. For fuzzy name matching with phone and address evidence use Remove Duplicate Leads.
• Products from two stores: keyFields: ["sku", "title+brand"], keepPolicy: richest.
• Reworded duplicates: mode: fuzzy, fuzzyFields: ["title","description"], threshold 85-90.

From code:

curl -X POST "https://api.apify.com/v2/acts/that_red_bird~remove-duplicate-scraped-data/runs?token=$APIFY_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"datasetIds":["YOUR_DATASET_ID"],"preset":"jobs","rememberAcrossRuns":true,"memoryName":"my-jobs"}'

Works in Make, Zapier and n8n through the Apify integrations (trigger on "run succeeded", read the default dataset), and as a tool for AI agents over MCP.

Pricing

Pay per event: $0.002 per run + $0.001 per unique row delivered. 100,000 rows that shrink to 70,000 cost about $70; a scheduled run that finds 40 new postings costs about $0.04. Removed duplicates, reports and rows seen in earlier runs are not charged.

Limits and behaviour you should know

• Fuzzy similarity is checked exactly (word or character shingles) up to 400,000 rows; above that the MinHash estimate is used.
• Keys mode is transitive: if row A matches B on the URL and B matches C on title + company + location, all three are one group.
• Rows that have none of the key columns cannot be compared; they are kept and counted in rowsWithoutAnyKey.
• Nested values are compared as text; arrays are compared order-independently.
• Cross-run memory stores 53-bit hashes of keys (not your data). Two different keys sharing a hash is possible in theory; for a million remembered keys the chance of even one such collision is below 0.01%.

FAQ

Does it change my rows? No. Kept rows are returned exactly as they came in (in all mode they gain _dedupe* columns).

What if my duplicates have different IDs? IDs assigned by a scraper per run are not identity. Use url, title+company+location or fuzzy mode.

Can I rely on it for 1M+ rows? Give the run 2-4 GB memory. Tested with 1,000,000 synthetic job rows (30% duplicates, 300,000 removed, exactly as injected). Fuzzy mode on 200,000 rows finished in under 10 seconds and found 59,940 of the 60,000 injected duplicates.

Is anything sent to a third party? No. All matching runs inside the Actor.

Support

Something deduplicated wrongly? Open an issue with two example rows and the settings you used. If this saved you hours of cleanup, a short review helps others find it.