Merge & Deduplicate Datasets (only new items, fast)

Combine any number of Apify datasets into one clean dataset:

• Remove duplicates by the fields you choose (url, id, seller.id …), or by the whole item when you choose none.
• Keep the first or the last copy. With "last", the newer dataset wins, which is handy when re-scraping.
• Only new items: remember what earlier runs returned, and output just the new ones. This is the classic setup for a scheduled scraper that should only alert you about new listings, leads or prices.
• Clean up: drop items missing required fields, keep only some fields, remove others, ignore letter case or extra spaces in keys, and tag each item with the dataset it came from.

With "keep first" it streams items and holds only an 8-byte fingerprint per unique key, so a run at the default 1 GB memory handles several million unique items. "Keep last" has to hold one copy of each unique item until the end, so give it more memory for big datasets. It runs with limited permissions: it can only read the datasets you select and write its own output and memory.

Typical setups

After a scraper (Integrations): in your scraper's task, go to Integrations → Connect Actor or task, pick this Actor, and use:

{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "mode": "onlyNew",
  "dedupFields": ["url"],
  "stateName": "my-listing-monitor"
}

Each scheduled run then outputs only listings that weren't in any earlier run. The fixed stateName matters because every scraper run creates a new dataset. The memory is saved as the run goes (at least every 15 seconds, and right away when the cost limit is hit or the run is migrated or aborted), so after a timeout or crash the next run repeats at most the last few seconds of items.

Merge several runs: pick the datasets, set dedupFields to ["id"] and keep to last.

Input

Field	Notes
Datasets	One or more datasets, processed in the order given.
Items	A JSON array to deduplicate instead of datasets (used when no dataset is picked).
Mode	Remove duplicates, only new since earlier runs, or just merge.
Deduplicate by fields	Dots for nested fields. Empty = compare whole items.
Which duplicate to keep	First (streams) or last (keeps one buffered item per key).
Ignore letter case / extra spaces	Applied to the key fields only.
Drop items missing these fields	Missing, empty string, empty list or empty object all count as missing. Works in every mode.
Output only these fields / Remove these fields	Shape the output.
Add _sourceDataset	Records where each item came from.
Max output items	Optional cap.
Monitoring name	For "only new": runs with the same name (and the same dedup settings) share one memory. Without a name, a saved task keeps its own memory; otherwise the memory belongs to that exact set of datasets.

Output

• Dataset: the unique (or merged) items.
• Key-value store OUTPUT: counts of items read, duplicates removed, items missing required fields, and items output.

Pricing

You pay per 1,000 items written, and each started block of 1,000 counts as one. Items that are read but removed as duplicates are free. If you set a maximum cost per run, the Actor stops cleanly at that limit, and in "only new" mode it remembers only the items it actually returned.

Notes

• "Only new" remembers about 5 million keys per memory, then forgets the oldest (in blocks of 100,000).
• Don't run two "only new" runs with the same memory at the same time; the one that finishes last wins.
• Keys are compared by type: 1 and "1" are different keys. Changing the dedup fields or the case/space options starts a fresh memory.
• Keys are compared as compact 64-bit fingerprints. Two different keys colliding is astronomically unlikely, but not impossible.