Dataset Merge, Dedupe & Only New Items

Merge any number of Apify datasets, remove duplicates by URL or any field, and output only the items you haven't seen in earlier runs. Built for scheduled scrapers, integrations and AI agents: it runs with limited permissions, so it starts without an extra permission prompt and only reads the datasets you give it.

• 🔀 Merge many datasets into one, in the order you choose
• 🧹 Dedupe on one or more key fields, with smart matching: spaces, letter case, accents, and URL variants (http/https, www., trailing slashes, #fragments, parameter order, utm_*, fbclid, gclid...)
• 🆕 Only new items: remembers every key it has output under a Memory name, so a daily scraper run returns just today's new listings, jobs, products or posts. Scales to millions of keys (8 bytes each, loaded in well under a second for 100k keys)
• ✂️ Pick, remove and rename fields, including nested ones (price.value → price)
• 📤 CSV, Excel and JSON download links for the result
• 🔗 Integration-ready: add it as "Run another Actor when a run succeeds" with {"datasetId": "{{resource.defaultDatasetId}}"}
• 💸 Duplicates and already-seen items are never charged

Pricing

Pay per event. Compute is included.

Event	Free & Bronze	Silver	Gold & above
Output item (unique or new)	$0.10 / 1,000	$0.09 / 1,000	$0.08 / 1,000
Items read and compared	$0.002 / 1,000	$0.002 / 1,000	$0.002 / 1,000
Run start	$0.00005 per GB	same	same

Examples (Free/Bronze prices):

Job	Read	Output	Cost
Merge 3 scraper runs and remove duplicates	150,000	120,000	about $12.30
Daily "only new" check on a 30,000-item scrape with 500 new items	30,000	500	about $0.11 a day (about $3.30 a month)
Hourly check of 2,000 items, usually nothing new	2,000	0	about $0.004 per run
Dedupe 1,000,000 leads down to 400,000	1,000,000	400,000	about $42

Set Max cost per run in the run options: the Actor stops before it goes over, and items it couldn't output are not remembered, so the next run picks them up.

Running with no datasets selected starts a free 4-item demo (useful for Store health checks and trying the Actor); pick a dataset or pass datasetId for real data.

How to use it

Dedupe or merge datasets

1. Pick one or more datasets in Datasets (from your runs, or paste dataset IDs).
2. Choose Mode: Merge + remove duplicates or Merge only.
3. Set Key fields, for example url, or title + price. Leave empty to compare whole items.
4. Start. Download the result as CSV or Excel from the Output tab.

When two items share a key, the first one wins, so list your most trusted dataset first.

Only new items after every scraper run

1. Open your scraper (or its task) → Integrations → Run another Actor → pick this Actor.
2. Use this input:

{
  "datasetId": "{{resource.defaultDatasetId}}",
  "mode": "onlyNew",
  "memoryName": "amazon-laptops",
  "dedupeFields": ["url"]
}

3. Each scraper run now produces a second dataset with only the items that never appeared before. Chain it into Google Sheets, Slack, email or a webhook.

The memory lives in a key-value store in your account named dedupe-memory-<memory name>. Use a different memory name per scraper or per search. Reset memory starts over.

Input example

{
  "datasetIds": ["flob8zAqUIziccWR2", "dzm4JlQiUbnu3dA3C"],
  "mode": "dedupe",
  "dedupeFields": ["url"],
  "normalizeUrls": true,
  "caseInsensitive": true,
  "pickFields": ["url", "title", "price.value"],
  "renameFields": { "price.value": "price" }
}

Output

The default dataset holds your items with their original fields (after pick, remove and rename). The OUTPUT record summarizes the run:

{
  "mode": "onlyNew",
  "itemsRead": 30000,
  "itemsOutput": 15000,
  "duplicatesRemoved": 0,
  "seenInEarlierRuns": 15000,
  "memoryKeys": 135000,
  "exports": {
    "csv": "https://api.apify.com/v2/datasets/<id>/items?format=csv",
    "xlsx": "https://api.apify.com/v2/datasets/<id>/items?format=xlsx"
  }
}

Matching rules

Option	Default	Effect
Trim spaces	on	" Apple " = "Apple"
Collapse inner whitespace	on	"New York" = "New York"
Ignore letter case	on	"IPHONE" = "iPhone"
Normalize URLs	on	http://www.shop.com/p/1/?utm_source=x = https://shop.com/p/1
Ignore all URL parameters	off	shop.com/p?id=1 = shop.com/p?id=2
Ignore accents	off	"Café" = "Cafe"

Numbers, booleans and objects are compared by value. Items whose key fields are all empty are kept by default (choose Drop them to remove them).

Performance

Measured on Apify with 1 GB of memory: 150,000 items from 3 datasets deduped in 35 seconds; an only-new run over 30,000 items against 120,000 remembered keys took 4 seconds (memory loaded in 0.18 s). Works on 256 MB for most jobs.

Troubleshooting

• "No access to dataset": this Actor uses limited permissions. Select datasets in Datasets or pass them in datasetId/datasetIds. IDs placed elsewhere can't be read.
• "is not a dataset ID": use the 17-character ID (e.g. WkzbQMuFYuamGv3YF) or the picker, not a dataset name.
• "template that was not filled in": {{resource.defaultDatasetId}} only works inside an integration.
• Everything shows up as new again: the memory name or key fields changed. The log warns when key fields differ from the ones the memory was built with.
• Two runs at once with the same memory name: avoid it; the second run may not see the first run's items.

FAQ

How accurate is "only new"? Keys are stored as 64-bit fingerprints. The chance that any two different keys collide is about 1 in 37 million at 1 million remembered keys, and about 1 in 370,000 at 10 million.

Do I pay for duplicates? No. You pay for items written to the output and a small fee per 1,000 items read.

Where is my data stored? Only in your own Apify account: the output dataset and the memory key-value store.

Questions or ideas? Open an issue on the Issues tab.