Dataset Merge & Dedupe (limited permissions, dataset picker) avatar

Dataset Merge & Dedupe (limited permissions, dataset picker)

Pricing

Pay per event

Go to Apify Store
Dataset Merge & Dedupe (limited permissions, dataset picker)

Dataset Merge & Dedupe (limited permissions, dataset picker)

Merge the Apify datasets you pick, remove duplicates by any fields, filter, keep/drop/rename fields and keep only new items across runs. Works under limited permissions via the dataset picker. $0.25 per 1,000 output items.

Pricing

Pay per event

Rating

0.0

(0)

Developer

fecklesspleb

fecklesspleb

Maintained by Community

Actor stats

0

Bookmarked

3

Total users

2

Monthly active users

7 days ago

Last modified

Features

Limited permissions

Share

Dataset Merge & Dedupe (works under limited permissions)

Merge the Apify datasets you pick, remove duplicates by any fields, filter, keep/drop/rename fields, and keep only items that earlier runs have not written. Results go to this run's dataset or to a dataset you pick.

Why another merge Actor? Under Apify's default limited permissions, an Actor can only read your datasets if you hand them over through a dataset picker field. This Actor's inputs are pickers (resourceType: dataset), so the platform grants the run access to exactly the datasets you pick. Nothing else in your account is readable.

What it does

  1. Reads the picked datasets in the order you picked them (items in dataset order).
  2. Applies your filters (all must match).
  3. Builds a dedupe key from your dedupe fields (dotted paths like seller.id work) and drops repeated keys, keeping the first or the last copy.
  4. With only new items on, skips items whose key an earlier run already wrote.
  5. Keeps, drops and renames top-level fields, optionally adds _sourceDatasetId.
  6. Writes to the default dataset, or appends to the output dataset you pick.

Empty input runs a free built-in demo (two sample datasets, 8 items, 5 written), so you can see the output shape without picking anything.

Input

FieldDefaultWhat it does
datasetIdsempty (= demo)Datasets to merge. Dataset picker, READ access. You can also type a dataset ID or username~dataset-name. A bare dataset name is not looked up.
dedupeFieldsprefilled ["url"]Same values in all these fields = duplicate. Empty = the whole item must be identical. Items that have none of the fields are never treated as duplicates (counted in keptWithoutKey).
dedupetruefalse = plain merge, every copy kept.
keepfirstlast keeps the last copy. It reads every dataset twice, so read blocks are charged twice.
normalizeValuesfalseCompare strings ignoring case and extra spaces (output is unchanged).
filtersnoneList of {"field", "op", "value"}. Ops: eq ne contains notContains exists notExists gt gte lt lte in notIn. Strings ignore case unless "caseSensitive": true. Numbers in strings ("20") compare as numbers for gt/gte/lt/lte. Filters run before dedupe.
fieldsToKeep / fieldsToOmitnoneTop-level fields to keep (in that order) or drop.
renameFieldsnone{"old": "new"}, applied after keep/drop. Dedupe always uses original names.
addSourceFieldfalseAdds _sourceDatasetId.
onlyNewItemsfalseSkip items whose dedupe key an earlier run wrote. Keys are stored in the named key-value store below, separately for each dedupe setting (or stateKey). The newest 2,000,000 keys are kept.
stateStoreNamedataset-merge-dedupe-stateNamed key-value store for onlyNewItems.
stateKeyderivedName of a pipeline, so different pipelines with the same dedupe fields don't share state.
outputDatasetIdempty (= default dataset)Append results to this dataset instead. Picker with READ and WRITE. The default dataset then gets one free summary item.
maxInputItems1,000,000Read at most this many items (hard limit 5,000,000).
maxOutputItemsno capWrite at most this many items.

Output

Your own items, merged and transformed. Problems (a dataset that could not be opened, invalid input) appear as free items with "type": "status", and the run still ends successfully so schedules keep working. RUN_SUMMARY in the key-value store has counts per dataset: items read, filtered out, duplicates dropped, seen in earlier runs, written, and why the run stopped.

Examples (from real local runs, 2026-11-11)

These were run with the Apify SDK's local storage (no cloud). Two datasets: shop-a: {"sku":"1","name":"Lamp","price":"20"}, {"sku":"2","name":"Desk","price":150}, {"sku":"3","name":" desk ","price":99} (plus seller on the first two); shop-b: {"sku":"2","name":"Desk","price":140}, {"sku":"4","name":"Chair","price":45}, {"name":"No sku item","price":1}.

1. Empty input ({}) runs the demo: read 8, wrote 5, dropped 3 duplicates by url. First item:

{"url": "https://example.com/a", "title": "Alpha", "price": 10}

2. Keep the latest price, only items costing 20 or more, tidy fields:

{"datasetIds": ["<shop-a id>", "<shop-b id>"], "dedupeFields": ["sku"], "keep": "last",
"filters": [{"field": "price", "op": "gte", "value": 20}],
"fieldsToKeep": ["sku", "name", "price"], "renameFields": {"price": "priceUsd"}}

Output (read 6, filtered 1, duplicates dropped 1, wrote 4):

[{"sku": "1", "name": "Lamp", "priceUsd": "20"},
{"sku": "3", "name": " desk ", "priceUsd": 99},
{"sku": "2", "name": "Desk", "priceUsd": 140},
{"sku": "4", "name": "Chair", "priceUsd": 45}]

3. A dataset ID that does not exist ({"datasetIds": ["2x1LjUSyc4tESKlhw", "HP1VXZZljAqGEMZmf", "missingID123"], "dedupeFields": ["sku"], "addSourceField": true}; the first two are the local IDs of shop-a and shop-b): the two real datasets are merged (5 items written) and one free status item explains the third:

{"type": "status", "status": "error", "message": "Dataset 'missingID123' could not be opened, so it was skipped (nothing read or charged for it). Pick it with the dataset picker, or give its ID or username~dataset-name. Error: ValueError: Dataset with ID \"missingID123\" not found", "datasetId": "missingID123"}

Pricing

Pay per event:

EventPriceWhen
item-output$0.00025 ($0.25 per 1,000)each item written
input-items-1000$0.0015each started block of 1,000 items read
apify-actor-start$0.00005Apify's standard run start (one per GB of memory, so 1 at the default 1,024 MB)
demo run, status and summary itemsfree

Your maximum charge per run is respected: the run stops reading and writing when the next event would go over it and says so in RUN_SUMMARY.stoppedBecause.

Limits and notes

  • Dedupe keys are kept in memory (about 100 bytes each, ESTIMATED from a local benchmark: 500,000 unique keys used about 70 MB). The default 1,024 MB covers a few million unique keys.
  • Nested fields can be used for dedupe and filters, but keep/drop/rename work on top-level fields only.
  • 1 and 1.0 are the same number. true is not 1. "20" equals 20 only in gt/gte/lt/lte.
  • The Actor reads your own datasets through the Apify platform. It fetches nothing from the web.

Terms and permissions evidence

Read on 2026-11-11 from Apify's documentation. Copies are in evidence/.