Merge & Deduplicate Datasets - Reliable, Any Size

Merge any number of Apify datasets into one clean dataset and remove duplicates by the fields you choose. Use it to combine runs from several scrapers, tidy up a scheduled scraper's history, or find the items that appear in more than one dataset.

• Pick datasets straight from your storage. Input uses Apify's dataset picker, so the run gets read access to exactly the datasets you choose. No "insufficient permissions" failures.
• Dedup by any fields: url, email, productId + variant... or by whole item.
• Three outputs: unique items (first occurrence wins, original order kept), only the duplicates, or just the counts.
• Trim output: keep only the fields you need, and tag every item with its _sourceDataset.
• Streams data, so millions of items work in small memory. A STATS record shows items loaded, unique, duplicates and output.

Example

Dataset A has products 1–7 and dataset B has products 5–10. Deduplicating by url gives products 1–10, 10 unique items, with 3 duplicates removed.

{
  "datasetIds": ["5HVQsVrFp3bTQ06Yv", "xbRTmeYjFNWdbMs8e"],
  "dedupFields": ["url"],
  "output": "unique",
  "outputFields": [],
  "addSourceDataset": true
}

STATS record:

{"datasets": 2, "itemsLoaded": 13, "uniqueItems": 10, "duplicateItems": 3, "itemsOutput": 10}

Leave datasetIds empty to try it on built-in demo data.

Use cases

• Combine scraper runs: merge the datasets of many scheduled runs into one deduplicated list.
• Incremental scraping: find what's new compared with yesterday (output only the unique items of today's dataset after yesterday's).
• Lead lists: merge lead sources and drop repeated emails or domains.
• Data quality checks: count duplicates without outputting anything.

Pricing

$0.50 per 1,000 output items ($0.0005 per item). Counting duplicates with output "Nothing" only costs the start fee.

FAQ

Which dataset is kept when items are duplicated? The first one, in the order you list the datasets.

Can I use datasets from other accounts? Only datasets your account can read.

Nested fields? Use top-level field names. Whole-item deduplication compares everything, including nested values.