Merge & Deduplicate Datasets: Remove Duplicates

Merge & Deduplicate Datasets combines any number of Apify datasets into one and removes duplicate items by one or more fields, such as url, id or product.id + shop. It reads datasets page by page, so it can handle large datasets. It runs with limited permissions: it reads only the datasets you pick. An optional only-new-items mode remembers what earlier runs already output, so scheduled scrapers deliver only new results. Pricing: $0.25 per 1,000 unique output items. Duplicates and skipped items are free.

What does Merge & Deduplicate Datasets do?

1. Merge datasets: reads all selected datasets in the order you list them.
2. Remove duplicates: keeps the first item for each unique combination of your dedupFields. Dot paths like product.id work. Comparison can be case-insensitive and can ignore leading or trailing whitespace. Leave dedupFields empty to remove only fully identical items.
3. Output only new items (optional): skips items that earlier runs with the same stateKey already output. The remembered keys live in a key-value store the Actor creates in your account (dedup-state-<stateKey>).
4. Filter by reference datasets (optional): also skips items whose key appears in other datasets, for example "already imported into the CRM".
5. Select fields (optional): outputs only the fields you choose and can add _sourceDatasetId to each item.
6. Saves a run summary: counts of read, output and duplicate items, plus the stop reason, are saved to the OUTPUT record of the run's key-value store.

Key features

• ✅ Pre-flight check: before anything is written or charged, the Actor checks that every dataset exists and is readable. If one isn't, the run fails with a clear message, e.g. Dataset "x" was not found… or No permission to read dataset….
• 🔐 Limited permissions: datasets are selected with Apify's dataset picker (READ only). The Actor writes only to its own default dataset and key-value store and, in only-new mode, to a key-value store it created itself. It does not list your runs, Actors or other storages.
• 🌊 Streaming: reads in pages (default 5,000 items) and writes in batches (default 1,000). Memory is used mainly for one compact 80-bit hash per unique key.
• 🔁 Retry-safe: every read uses exponential backoff with jitter for 429, 5xx and network errors. Permanent errors (401/403/404) are not retried.
• 🧭 Migration-safe: the Actor saves checkpoints at page boundaries, and a restarted run continues where it stopped. Checkpoints and the only-new state are stored as compact binary records of at most 4 MB each (10 bytes per key), tested with 5 million keys.
• 🧮 Collision-safe keys: field values are JSON-encoded and joined with a separator, so "ab"+"c" never equals "a"+"bc". Objects are compared regardless of key order.
• 💸 Budget-aware: stops cleanly at your Max total charge or maxOutputItems. Items that were not output are not remembered as seen, so a later run still treats them as new.

Use cases

1. Combine parallel scraper runs: merge the datasets of many scraper runs into one clean, deduplicated dataset.
2. Incremental scraping: schedule a scraper, then output only new listings, jobs, products or posts each day.
3. Skip already-processed records: use reference datasets to exclude items you have already imported or handled.
4. Shrink exports: remove duplicates and keep only the fields you need before exporting to Google Sheets, Airtable, a CRM or a vector database.
5. Clean up datasets: remove fully identical items (empty dedupFields) or normalize keys with case-insensitive and trimmed comparison.

How much does it cost to merge and deduplicate datasets?

The Actor uses pay-per-event pricing:

Event	Price	When it is charged
item-output	$0.00025 per unique item written ($0.25 per 1,000)	When a unique item is written to the output dataset
apify-actor-start	$0.005 per start event	Automatically by Apify at run start

Free: items read, in-run duplicates, items seen in previous runs, items filtered by reference datasets and dropped items.

Example calculation: merge 100,000 items, 60,000 of them unique (default 1 GB memory):

60,000 unique items × $0.00025 = $15.00
+ 1 Actor start × $0.005       = $0.005
------------------------------------------
Total                           = $15.005   (= $0.25 per 1,000 output items + $0.005 per run)

The 40,000 duplicates cost nothing. Apify charges the start event once for runs with up to 1 GB of memory and once more for each additional GB, so a run with 4 GB of memory costs 4 × $0.005 = $0.02 in start events.

How to merge and deduplicate Apify datasets

1. Open Merge & Deduplicate Datasets in Apify Store and click Try for free.
2. In Datasets to merge, select one or more datasets with the dataset picker.
3. In Deduplicate by fields, enter the field(s) that define a duplicate, e.g. url or product.id and shop.
4. Optional: turn on Only output items not seen in previous runs and choose a State key (one per pipeline).
5. Optional: add Reference datasets, choose Output only these fields, and set Max output items or a Max total charge.
6. Click Start. The unique items appear in the run's dataset, and the summary is in the OUTPUT record of the key-value store.
7. For scheduled scrapers, add this Actor as a follow-up step (for example with a schedule or an integration) and keep the same stateKey on every run.

Input example

{
    "datasetIds": ["aBcD1234EfGh5678", "IjKl9012MnOp3456"],
    "dedupFields": ["url"],
    "caseInsensitive": false,
    "trimStrings": true,
    "missingFieldMode": "keep-all",
    "onlyNewItems": true,
    "stateKey": "daily-products",
    "updateState": true,
    "referenceDatasetIds": [],
    "outputFields": ["url", "title", "price.value"],
    "addSourceDatasetId": true,
    "maxOutputItems": 0,
    "pageSize": 5000,
    "pushBatchSize": 1000
}

Input field	Type	Default	Description
datasetIds	array	–	Datasets to merge (dataset picker, READ access). Needed for real use: without it the Actor runs the small demo below.
demoItems	array	5 demo items	Used only when no dataset is selected: a tiny built-in example (deduplicated by url) so a run with empty input still shows the result. Ignored as soon as you select a dataset. Max 100 items; output items are charged like normal output items.
dedupFields	array	prefill ["url"]	Fields whose combined value must be unique. Dot paths allowed. Empty = whole-item comparison.
caseInsensitive	boolean	false	Treat "ABC" and "abc" as equal.
trimStrings	boolean	true	Ignore leading/trailing whitespace.
missingFieldMode	string	keep-all	Items missing all dedup fields: keep-all (always output), as-empty (collapse into one), drop.
onlyNewItems	boolean	false	Skip items output by earlier runs with the same stateKey.
stateKey	string	default	Name of the remembered state (store dedup-state-<key>).
updateState	boolean	true	false = dry run: compare with the state without changing it.
referenceDatasetIds	array	–	Items whose key appears here are not output.
outputFields	array	–	Output only these fields (dot paths are re-nested).
addSourceDatasetId	boolean	false	Add _sourceDatasetId to each item.
maxOutputItems	integer	0	0 = unlimited; also capped by Max total charge.
pageSize	integer	5000	Items read per API call (100–50,000).
pushBatchSize	integer	1000	Items written per API call (1–5,000).

Output example

Dataset: your original items (or only the selected fields), unique by your dedup key. With outputFields: ["url", "title", "price.value"] and addSourceDatasetId: true:

[
    { "url": "https://shop.example/p/0", "title": "Product 0", "price": { "value": 0 }, "_sourceDatasetId": "aBcD1234EfGh5678" },
    { "url": "https://shop.example/p/1", "title": "Product 1", "price": { "value": 1 }, "_sourceDatasetId": "aBcD1234EfGh5678" }
]

Run summary (OUTPUT record in the default key-value store; numbers from a local test run):

{
    "read": 20003,
    "output": 13002,
    "duplicates": 7001,
    "seenInPreviousRuns": 0,
    "droppedMissingKey": 0,
    "keptWithoutKey": 2,
    "referenceItems": 0,
    "filteredByReference": 0,
    "perDataset": {
        "src-a": { "itemCount": 12000, "read": 12000, "output": 10000 },
        "src-b": { "itemCount": 8003, "read": 8003, "output": 3002 }
    },
    "stoppedReason": null,
    "dedupFields": ["url"],
    "onlyNewItems": true,
    "stateStore": "dedup-state-e2e"
}

Output fields

Field	Where	Description
(your item fields)	Dataset	Original item, or only outputFields (dot paths re-nested, e.g. price.value → { "price": { "value": … } }).
_sourceDatasetId	Dataset	ID of the dataset the item came from (only with addSourceDatasetId).
read	OUTPUT	Items read from the source datasets.
output	OUTPUT	Unique items written (= charged item-output events).
duplicates	OUTPUT	Duplicates found within this run.
seenInPreviousRuns	OUTPUT	Items skipped by only-new mode.
filteredByReference / referenceItems	OUTPUT	Items skipped because of reference datasets / items read from reference datasets.
keptWithoutKey / droppedMissingKey	OUTPUT	Items missing all dedup fields that were kept / dropped.
perDataset	OUTPUT	itemCount, read, output per source dataset.
stoppedReason	OUTPUT	null, maxOutputItems reached, or a max total charge message.
dedupFields, onlyNewItems, stateStore	OUTPUT	Settings used for the run.

Integrations and API

The unique items are in the run's default dataset, and the summary is in the OUTPUT record. Both are available through the Apify API and the API clients, and you can trigger the Actor with Apify schedules, webhooks or integrations after your scraper finishes.

curl -X POST "https://api.apify.com/v2/acts/grace_hartwell~dataset-merge-dedup/runs?token=[YOUR_APIFY_TOKEN]" \
  -H "Content-Type: application/json" \
  -d '{"datasetIds":["aBcD1234EfGh5678","IjKl9012MnOp3456"],"dedupFields":["url"],"onlyNewItems":true,"stateKey":"daily-products"}'

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('grace_hartwell/dataset-merge-dedup').call({
    datasetIds: ['aBcD1234EfGh5678', 'IjKl9012MnOp3456'],
    dedupFields: ['url'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const summary = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT');

FAQ

Which item is kept when there are duplicates?

The first one found, in the order of the datasets you selected and the order of items within each dataset. Items are written in the order they are read.

How do I remove duplicates by more than one field?

List several fields in dedupFields, e.g. ["product.id", "shop"]. Two items count as duplicates only if all listed values match.

How does the only-new-items mode work?

The Actor stores a hash of each output item's key in the key-value store dedup-state-<stateKey>. The next run with the same stateKey skips those items and reports them as seenInPreviousRuns. Use different state keys for unrelated pipelines.

Can I do a dry run?

Yes. Set updateState: false to compare with the remembered state without changing it. Note that output items are still written and charged.

How large can the datasets be?

The Actor streams items, so the size of the source datasets is not the problem. What uses memory is the set of unique-key hashes. For very large numbers of unique keys, give the run more memory (up to 4 GB). The extra memory also increases the number of start events charged (see pricing).

Can it write into one of my existing datasets?

No. In this limited-permissions version, output always goes to the run's default dataset. Use the API or an integration to forward it.

Why did my run fail right at the start?

The pre-flight check found a dataset that doesn't exist or can't be read. Select the datasets with the dataset picker, and use the dataset ID, not its name. Nothing is written or charged in that case.

Limitations: what this Actor does not do

• Apify datasets only: it doesn't read CSV or JSON files, URLs, or other storage types.
• Exact matching only: no fuzzy or similarity matching. Only string values used directly as dedup fields are trimmed or lower-cased; nested values are compared exactly.
• Keeps the first item; does not merge fields from duplicates into one record.
• Items missing all dedup fields are never deduplicated in keep-all mode. They are also not remembered by only-new mode, so they are output (and charged) again on every run. Use as-empty or drop if that is not what you want.
• Reference keys are remembered: in only-new mode with updateState: true, keys loaded from reference datasets are saved to the state too, so later runs keep skipping them.
• Hidden fields and empty items are skipped: datasets are read with clean: true, which drops empty items and fields whose names start with #.
• Output only to the run's default dataset; no direct export to external systems.
• No custom sorting: items are output in read order.
• Each run is capped by maxOutputItems and your Max total charge.

---

Author: Grace M. Hartwell