Merge & Deduplicate Datasets

Merge any number of Apify datasets into one clean dataset and remove duplicates by one or more fields. Built for large runs: items are streamed page by page, so memory stays flat even when you merge millions of items. Runs with limited permissions: it can only read the datasets you pick.

What it does

• Merge datasets from many runs (for example 30 daily runs of the same scraper) into one dataset, in the order you list them.
• Deduplicate by one field (url) or a combination (name + address.city). Nested values use dot paths such as address.city or images.0.url.
• Keep the first or the last duplicate. "Last" is handy when datasets are listed oldest first and you want the freshest copy of each item.
• Ignore case and extra spaces when comparing (on by default), so ACME Inc and acme inc are the same.
• Shape the output: keep only some fields and rename them (address.city to city).
• Clear errors before any work: unknown dataset, dataset you did not share with the run, or a dedupe field that does not exist (you get the list of fields that do exist).
• Run summary in the key-value store record SUMMARY: items read, unique items, duplicates removed, items without a dedupe key, items read per dataset.

How to use it

1. In Datasets to merge, pick your datasets with the picker or paste dataset IDs. A run's results live in its default dataset (run detail page, Storage tab).
2. In Deduplicate by fields, enter the field(s) that identify an item, e.g. url. Leave it empty to only merge. Want to try first? Use our public demo dataset CotvbQSQ340wBAwq3 (19 coffee shops with 9 duplicates) and dedupe by url.
3. Start the run. Download the merged dataset as JSON, CSV or Excel, or read it through the API.

Example input:

{
  "datasetIds": ["<datasetId1>", "<datasetId2>"],
  "dedupeFields": ["url"],
  "keep": "last",
  "fields": ["url", "title", "address.city", "rating"],
  "rename": { "address.city": "city" }
}

Input

Field	What it does
datasetIds	Datasets to read, in order. Dataset IDs or username~dataset-name. Required.
dedupeFields	Fields whose combined values must be unique. Dot paths for nested values. Empty = merge only.
keep	first (default) or last occurrence of each duplicate. last reads the input twice.
normalizeValues	Trim, collapse spaces and lowercase before comparing (default on). Objects and arrays are compared by content.
fields	Optional list of fields to output, in order. Missing values become null.
rename	Optional map from a field or dot path to a new name.
pageSize	Items per API call (default 1,000). Lower it for very large items.

Items that have none of the dedupe fields are never treated as duplicates. They are kept and counted in itemsWithoutDedupeKey.

Why it does not run out of memory

The Actor never loads a whole dataset. It reads one page (1,000 items by default) while the next page is being fetched, writes the unique items, and keeps only a 16-character hash of each unique key. That is about 55 MB per million unique keys (measured), so the default 512 MB handles several million unique items. In our tests with 256 MB of memory, a 105,000-item merge (keep first) peaked at 125 MB and finished in 48 seconds; the same data with keep last (two passes, 210,000 item reads) peaked at 88 MB.

If the platform migrates the run to another server, the Actor resumes without writing the same item twice.

Pricing

Pay per event: $0.05 per 1,000 items written to the output dataset, plus a tiny Actor start fee. Items read and duplicates removed are not charged. Platform usage is included. You can cap the cost of a run with the maximum charge setting; the Actor stops cleanly when it is reached.

Limits

• Datasets must be ones your account can read. Because the Actor runs with limited permissions, it cannot list your runs by itself: pass dataset IDs (every run has one).
• Transform functions (custom JavaScript) are not supported. Use fields and rename for shaping.
• The order of the output follows the input: dataset order, then item order (for keep: last, the position of the last occurrence).

FAQ

Can I merge the results of all runs of my scraper? Yes. Copy the default dataset ID of each run into datasetIds. You can also call this Actor from an integration or webhook after each run.

How are nested objects compared? By content with sorted keys, so {"a":1,"b":2} equals {"b":2,"a":1}.

Does it change my source datasets? No. It only reads them (read-only permission) and writes to the run's own dataset.

Made by Ctrio Lab. Questions or bugs: open an issue on the Actor's Issues tab.