Dataset Deduplication & Merge - Remove Duplicates

Dataset Deduplication & Merge - Remove Duplicates

Merge Apify datasets and remove duplicate rows by selected fields for developers, data analysts and scheduled-scraping workflows. Keep the first or last matching row and retain source provenance; source datasets are never modified.

Pricing: $0.02 per 1,000 source rows processed, not output rows. The first 50 source rows in each run are free. The Pricing tab is authoritative. Apify plan allowances and any applicable platform/storage charges are separate; no additional free allowance is promised.

Example output (illustrative abbreviated record, not a captured result):

{"item":{"id":"example-1","product":"Example product"},"occurrences":2,"sourceDatasetId":"example-dataset","sourceIndex":0}

Try it: with no sources selected, the example input uses four built-in synthetic product rows with one duplicate. Replace Source datasets with your own to process real data. The default example produces three unique rows without a billable source-row event. sourceDatasetId is built-in-demo, not a real dataset. Set demoMode=false to require a real source.

What does Apify Dataset Merge & Deduplicate do?

Merge one or more Apify datasets and remove duplicate rows by chosen fields using the Apify API. Use it to combine scheduled scrape results, clean repeated leads, or compare product feeds. It does not scrape websites or use proxies.

Why use this Actor?

• Disk-backed SQLite index instead of keeping millions of keys in JavaScript memory.
• Limited permissions: source datasets receive READ access only; outputs go to the run's own default dataset.
• First or last representative, case-insensitive comparison, whitespace trimming, composite keys and duplicate-group reporting.
• Batches, API retries and visible processing counts. No browser or proxy costs.

How to merge and deduplicate datasets

1. Select one or more Source datasets. Input order determines first/last priority.
2. Set keyFields, for example ["email"] or ["shopId", "product.sku"].
3. Choose first/last and run. Export the results or connect them to Make, Zapier or another Actor.

The input tab lists every option. Example (replace the illustrative dataset IDs with your own):

{"datasetIds":["WkzbQMuFYuamGv3YF"],"keyFields":["email"],"keep":"last","caseInsensitive":true,"trim":true,"ignoreEmpty":true,"batchSize":1000}

Output data and exports

Field	Type	Meaning
item	object	Kept original row, after optional top-level field projection
occurrences	integer	Count of matching source rows
sourceDatasetId	string	Dataset containing the representative
sourceIndex	integer	Zero-based original row index

[{"item":{"id":"example-1","email":"example@example.com"},"occurrences":2,"sourceDatasetId":"WkzbQMuFYuamGv3YF","sourceIndex":0}]

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. Use unwind=item or flatten item when a flat export is needed. SUMMARY in the default key-value store contains processed/output/duplicate counts and a completion flag.

Pricing

$0.02 per 1,000 source rows processed (processed-item, $0.00002 per row); the first 50 source rows in every run are free. The exact current price is always shown in the Pricing tab. A batch may be read before a charge limit is detected, but only the permitted source prefix is indexed. Platform storage/transfer charges, if applicable to your plan, are separate.

Matching semantics and advanced options

• All key fields together define equality. Nested paths use dots; literal field names containing dots cannot be addressed as keys.
• Trimming and lowercasing affect comparison only, not original output. Objects compare regardless of property order; arrays remain ordered. Numbers and strings remain distinct; null and missing remain distinct.
• ignoreEmpty=true means rows with any missing, null or empty-string key remain separate; they are not dropped. With trimming, whitespace-only keys count as empty. False and zero are valid values.
• onlyDuplicates=true outputs one selected representative per repeated key, not all discarded rows.
• pickFields and omitFields operate on top-level original fields after comparison. Omit wins. Empty pick means keep all.
• Output order follows the selected representative's position in the merged source sequence.
• Sources should be finished and immutable. The row count is snapshotted per dataset; new appends after that are excluded. Read batches are unfiltered so empty/hidden rows do not break offsets.
• Memory is bounded by API/output batches, a 16 MiB SQLite cache and runtime overhead, not total row count. Disk space grows with unique keys and payload sizes. Very large individual rows need smaller batches and possibly more memory. Millions of rows are a design goal, not a measured platform guarantee.
• All modes index before producing output. Spending/item limits yield valid results for the processed prefix, marked complete=false; last/duplicate decisions are not global if processing stops early.

FAQ

Can I supply run IDs with limited permissions?

Apify only grants storage access to explicitly referenced dataset fields. A run ID does NOT grant access to its private metadata or default dataset. For private run IDs use the included local helper:

$APIFY_TOKEN=your_token node tools/resolve-runs.mjs RUN_ID > input.json

The helper resolves using your local account token and writes input containing datasetIds. Then start this Actor with that input; the Actor never accepts an account token in its input. The optional runIds input works only if metadata and resulting storage are already accessible. Prefer datasetIds for reliable limited-permissions operation.

Does it resume after migration or abort?

No. The disk index is ephemeral. Abort writes diagnostic STATE and exits promptly. Migration fails explicitly rather than silently losing the index. Start a fresh run; do not resurrect. Tune timeout to allow the full merge.

Permissions and privacy

Only pass datasets you are authorized to process. The Actor preserves your original data, which may include personal information; you remain responsible for lawful processing. It never writes to or deletes source datasets. Report problems through the Issues tab and use the API tab for integrations.