Merge & Dedup Datasets: Remove Duplicates, Only New Items

Merge & Deduplicate Datasets combines any number of Apify datasets into one clean dataset and removes duplicates by the fields you choose, such as url or id. Its only new items mode remembers what earlier runs returned and outputs just the items you haven't seen before, which turns any scheduled scraper into a monitor for new listings, leads, jobs or prices. It's for anyone who re-scrapes the same sources and wants each run's output without repeats, whether you work in Apify Console or call it through the API.

• Remove duplicates by the fields you choose (url, id, seller.id …), or by the whole item when you choose none.
• Keep the first or the last copy. With "last", the copy that comes last in input order wins (datasets in the order you list them, then item order). List older runs first and a re-scrape's values replace the old ones. There's no date comparison: to keep the newest by a date field, put the input in date order first.
• Only new items: remember what earlier runs returned, and output just the new ones.
• Just merge: combine datasets and keep everything.
• Clean up: drop items missing required fields, keep only some fields, remove others, ignore letter case or extra spaces in keys, and tag each item with the dataset it came from.

Try it in a minute: the input is already filled in with a small example. Click Try for free, then Start, and you'll see real results before you use your own data. Apify's free plan needs no credit card and gives you $5 to spend in Apify Store every month.

Why use Merge & Deduplicate Datasets?

• Fast and light on memory. With "keep first" it streams items and holds only a small fingerprint per unique key (about 70 bytes in memory), so a run at the default 512 MB handles about 3 million unique items. For more, or for "keep last", give the run 1–4 GB.
• Monitoring memory that survives crashes. In "only new" mode the memory is saved as the run goes (at least every 15 seconds, and right away when the cost limit is hit or the run is migrated or aborted), so after a timeout or crash the next run repeats at most the last few seconds of items.
• Works with scrapers whose dataset changes every run. A fixed Monitoring name ties the memory to your monitor, not to a dataset ID.
• Limited permissions. It can only read the datasets you select and write its own output and memory.
• Duplicates are free. You pay per 1,000 items written; items read and removed as duplicates cost nothing.

Use cases

• New-listing alerts: a scheduled real estate, marketplace or job-board scraper that should only report listings that weren't there yesterday.
• Lead generation without repeats: merge results from several searches or regions and remove duplicate companies by domain or phone.
• Combine runs: merge a week of runs into one dataset, keeping the version from the last-listed run (list runs oldest first).
• Clean before export: drop items missing an email or price and keep only the columns you need before sending data to Google Sheets, a CRM or a file.
• Quick dedupe from code: pass a JSON array in Items and get the unique items back through the API.

How to use Merge & Deduplicate Datasets

1. Open the Actor and pick one or more Datasets (processed in the order given), or paste a JSON array into Items.
2. Choose the Mode: merge and remove duplicates, only items not seen in earlier runs, or just merge.
3. Set Deduplicate by fields (for example url; dots for nested fields). Leave it empty to compare whole items.
4. Optionally choose Which duplicate to keep, required fields, and the fields to output or remove.
5. Click Start. The unique items are in the run's dataset, and the counts are in the key-value store OUTPUT.

Only new items after every scraper run (integration)

1. Open your scraper's saved task, go to the Integrations tab and click Add integration.
2. Search for Merge & Deduplicate Datasets, choose to start it when the run succeeded, and set the input. Apify fills in {{resource.defaultDatasetId}} with that run's dataset:

{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "mode": "onlyNew",
  "dedupFields": ["url"],
  "stateName": "my-listing-monitor"
}

3. Save, and test the integration on a past run.

Each scheduled run then outputs only listings that weren't in any earlier run. The fixed stateName matters because every scraper run creates a new dataset. To get the new items into a sheet or a Slack message, add a second integration on this Actor's runs: Google Sheets Sync or Send Email Report & Slack, Discord, Teams, Telegram Alerts, each with {{resource.defaultDatasetId}}.

Merge several runs: pick the datasets, set dedupFields to ["id"] and keep to last (list the datasets oldest first, so the newest run's copy is the one kept).

Input example

{
  "datasetIds": ["aBcDeFgHiJkLmNoPq", "QrStUvWxYzAbCdEfG"],
  "mode": "dedupe",
  "dedupFields": ["url"],
  "keep": "last",
  "trimWhitespace": true,
  "requiredFields": ["price"],
  "omitFields": ["debug"],
  "addSourceDataset": true
}

Field	Notes
Datasets	One or more datasets, processed in the order given.
Items	A JSON array to deduplicate instead of datasets (used when no dataset is picked).
Mode	Remove duplicates, only new since earlier runs, or just merge.
Deduplicate by fields	Dots for nested fields. Empty = compare whole items.
Which duplicate to keep	First (streams) or last (keeps one buffered item per key).
Ignore letter case / extra spaces	Applied to the key fields only.
Drop items missing these fields	Missing, empty string, empty list or empty object all count as missing. Works in every mode.
Output only these fields / Remove these fields	Shape the output.
Add _sourceDataset	Records where each item came from.
Max output items	Optional cap.
Monitoring name	For "only new": runs with the same name (and the same dedup settings) share one memory. Without a name, a saved task keeps its own memory; otherwise the memory belongs to that exact set of datasets.

Output example

The run's dataset holds the unique (or merged) items, unchanged apart from the fields you picked or removed, for example:

[
  { "url": "https://example.com/a", "price": 10, "_sourceDataset": "QrStUvWxYzAbCdEfG" },
  { "url": "https://example.com/b", "price": 12, "_sourceDataset": "aBcDeFgHiJkLmNoPq" }
]

The key-value store OUTPUT holds the counts:

{
  "read": 3,
  "duplicates": 1,
  "missingRequired": 0,
  "output": 2,
  "mode": "dedupe",
  "datasets": 2,
  "inlineItems": 0,
  "stoppedBy": null
}

How much does it cost to deduplicate datasets?

Merge & Deduplicate Datasets uses pay-per-event pricing. From November 5, 2026: $0.02 per 1,000 items written, plus a $0.005 run start fee (one start event per GB of run memory, minimum one; the default 512 MB run is charged one). Until then: $0.0019 per 1,000 items written plus $0.00005 per start. Each started block of 1,000 counts as one. Items that are read but removed as duplicates are free, and platform usage is included. The price is the same on every Apify plan. The run start applies to every run, including an "only new" run that finds nothing new.

Why the change: every output item is a dataset write, and the old price was below what those writes cost, so the Actor lost money on every large run.

Worked examples (prices from November 5):

• Merging three datasets with 120,000 items in total, of which 30,000 are duplicates, outputs 90,000 items: 90 × $0.02 + $0.005 = $1.805.
• A daily "only new" monitor that reads 5,000 scraped listings and finds 150 new ones writes one block: $0.02 + $0.005 = $0.025 per run, about $0.75 per month.
• Removing duplicate emails from a 3,000-row lead list that keeps 2,400 rows: 3 blocks × $0.02 + $0.005 = $0.065.

If you set a maximum cost per run, the Actor stops cleanly at that limit, and in "only new" mode it remembers only the items it actually returned.

FAQ

"Only new" returns every item again on each run. Why?

Every scraper run writes to a new dataset. Without a Monitoring name and outside a saved task, the memory belongs to the exact dataset IDs, so each run starts fresh. Set stateName (any name, like my-listing-monitor), or run the Actor from a saved task. Changing the dedup fields or the case/space options also starts a fresh memory.

How do I reset the memory of a monitor?

Use a new Monitoring name. The old memory is no longer used.

"Couldn't open dataset …"

Pick the dataset with the dataset picker, or pass a valid dataset ID. In an integration, use ["{{resource.defaultDatasetId}}"] (a list).

The run ran out of memory.

"Keep last" holds one copy of each unique item until the end. Use "keep first" if you can, or give the run more memory. Each remembered key takes about 70 bytes, so 512 MB is comfortable up to about 3 million unique keys; give a full 5-million-key monitor 1 GB.

Are 1 and "1" the same key?

No. Keys are compared by type, so a number and a string are different keys. Use Ignore letter case and Ignore extra spaces to match keys that differ only in case or spacing.

Can I deduplicate by a nested field?

Yes, with dots: seller.id, or images.0 for the first list element.

Limitations

• "Only new" remembers about 5 million keys per memory, then forgets the oldest (in blocks of 100,000).
• Don't run two "only new" runs with the same memory at the same time; the one that finishes last wins.
• Remove these fields works on top-level fields only.
• Keys are compared as compact 64-bit fingerprints. Two different keys colliding is astronomically unlikely, but not impossible.

Related Actors

Merge & Deduplicate Datasets is part of a small suite of Actors that work together after any scraper:

Actor	Use it to
Dataset Join & VLOOKUP	Add columns from another dataset or file by a shared key (email, SKU, URL), like VLOOKUP.
CSV & Excel to Dataset	Turn CSV, Excel or Google Sheets files into a dataset you can deduplicate here.
Google Sheets Sync	Append or upsert the new items into a Google Sheet.
Send Email Report & Slack, Discord, Teams, Telegram Alerts	Get told about new items by email, Slack, Discord, Teams or Telegram, only when there are some.
Dataset Delivery	Deliver the new items as a CSV, JSON or Excel file to SFTP, FTP, Dropbox or Google Drive.
Vector DB Sync	Load a clean dataset into Pinecone or Qdrant for RAG.
SEC Funding Rounds Tracker	Track newly funded companies from SEC Form D, Form C and Reg A filings.
Phone Number Validator & Formatter	Check and format the phone numbers in your unique leads (E.164, mobile or landline).
Dataset Filter & Transform	Filter rows, keep or rename columns, or flatten nested fields before exporting.
PDF & Image to Text	Turn PDFs and scanned documents into text, Markdown or tables (with OCR).
Website Contact Info Scraper	Add published emails, phones and social profiles to each lead's website.
Career Page Jobs Scraper	Collect open jobs from company career pages; it has its own only-new-jobs mode.