Dataset Merge & Dedupe (limited permissions, dataset picker)
Pricing
Pay per event
Dataset Merge & Dedupe (limited permissions, dataset picker)
Merge the Apify datasets you pick, remove duplicates by any fields, filter, keep/drop/rename fields and keep only new items across runs. Works under limited permissions via the dataset picker. $0.25 per 1,000 output items.
Pricing
Pay per event
Rating
0.0
(0)
Developer
fecklesspleb
Maintained by CommunityActor stats
0
Bookmarked
3
Total users
2
Monthly active users
7 days ago
Last modified
Categories
Features
Share
Dataset Merge & Dedupe (works under limited permissions)
Merge the Apify datasets you pick, remove duplicates by any fields, filter, keep/drop/rename fields, and keep only items that earlier runs have not written. Results go to this run's dataset or to a dataset you pick.
Why another merge Actor? Under Apify's default limited permissions, an Actor can only read your datasets if you hand them over through a dataset picker field. This Actor's inputs are pickers (resourceType: dataset), so the platform grants the run access to exactly the datasets you pick. Nothing else in your account is readable.
What it does
- Reads the picked datasets in the order you picked them (items in dataset order).
- Applies your filters (all must match).
- Builds a dedupe key from your dedupe fields (dotted paths like
seller.idwork) and drops repeated keys, keeping the first or the last copy. - With only new items on, skips items whose key an earlier run already wrote.
- Keeps, drops and renames top-level fields, optionally adds
_sourceDatasetId. - Writes to the default dataset, or appends to the output dataset you pick.
Empty input runs a free built-in demo (two sample datasets, 8 items, 5 written), so you can see the output shape without picking anything.
Input
| Field | Default | What it does |
|---|---|---|
datasetIds | empty (= demo) | Datasets to merge. Dataset picker, READ access. You can also type a dataset ID or username~dataset-name. A bare dataset name is not looked up. |
dedupeFields | prefilled ["url"] | Same values in all these fields = duplicate. Empty = the whole item must be identical. Items that have none of the fields are never treated as duplicates (counted in keptWithoutKey). |
dedupe | true | false = plain merge, every copy kept. |
keep | first | last keeps the last copy. It reads every dataset twice, so read blocks are charged twice. |
normalizeValues | false | Compare strings ignoring case and extra spaces (output is unchanged). |
filters | none | List of {"field", "op", "value"}. Ops: eq ne contains notContains exists notExists gt gte lt lte in notIn. Strings ignore case unless "caseSensitive": true. Numbers in strings ("20") compare as numbers for gt/gte/lt/lte. Filters run before dedupe. |
fieldsToKeep / fieldsToOmit | none | Top-level fields to keep (in that order) or drop. |
renameFields | none | {"old": "new"}, applied after keep/drop. Dedupe always uses original names. |
addSourceField | false | Adds _sourceDatasetId. |
onlyNewItems | false | Skip items whose dedupe key an earlier run wrote. Keys are stored in the named key-value store below, separately for each dedupe setting (or stateKey). The newest 2,000,000 keys are kept. |
stateStoreName | dataset-merge-dedupe-state | Named key-value store for onlyNewItems. |
stateKey | derived | Name of a pipeline, so different pipelines with the same dedupe fields don't share state. |
outputDatasetId | empty (= default dataset) | Append results to this dataset instead. Picker with READ and WRITE. The default dataset then gets one free summary item. |
maxInputItems | 1,000,000 | Read at most this many items (hard limit 5,000,000). |
maxOutputItems | no cap | Write at most this many items. |
Output
Your own items, merged and transformed. Problems (a dataset that could not be opened, invalid input) appear as free items with "type": "status", and the run still ends successfully so schedules keep working. RUN_SUMMARY in the key-value store has counts per dataset: items read, filtered out, duplicates dropped, seen in earlier runs, written, and why the run stopped.
Examples (from real local runs, 2026-11-11)
These were run with the Apify SDK's local storage (no cloud). Two datasets:
shop-a: {"sku":"1","name":"Lamp","price":"20"}, {"sku":"2","name":"Desk","price":150}, {"sku":"3","name":" desk ","price":99} (plus seller on the first two);
shop-b: {"sku":"2","name":"Desk","price":140}, {"sku":"4","name":"Chair","price":45}, {"name":"No sku item","price":1}.
1. Empty input ({}) runs the demo: read 8, wrote 5, dropped 3 duplicates by url. First item:
{"url": "https://example.com/a", "title": "Alpha", "price": 10}
2. Keep the latest price, only items costing 20 or more, tidy fields:
{"datasetIds": ["<shop-a id>", "<shop-b id>"], "dedupeFields": ["sku"], "keep": "last","filters": [{"field": "price", "op": "gte", "value": 20}],"fieldsToKeep": ["sku", "name", "price"], "renameFields": {"price": "priceUsd"}}
Output (read 6, filtered 1, duplicates dropped 1, wrote 4):
[{"sku": "1", "name": "Lamp", "priceUsd": "20"},{"sku": "3", "name": " desk ", "priceUsd": 99},{"sku": "2", "name": "Desk", "priceUsd": 140},{"sku": "4", "name": "Chair", "priceUsd": 45}]
3. A dataset ID that does not exist ({"datasetIds": ["2x1LjUSyc4tESKlhw", "HP1VXZZljAqGEMZmf", "missingID123"], "dedupeFields": ["sku"], "addSourceField": true}; the first two are the local IDs of shop-a and shop-b): the two real datasets are merged (5 items written) and one free status item explains the third:
{"type": "status", "status": "error", "message": "Dataset 'missingID123' could not be opened, so it was skipped (nothing read or charged for it). Pick it with the dataset picker, or give its ID or username~dataset-name. Error: ValueError: Dataset with ID \"missingID123\" not found", "datasetId": "missingID123"}
Pricing
Pay per event:
| Event | Price | When |
|---|---|---|
item-output | $0.00025 ($0.25 per 1,000) | each item written |
input-items-1000 | $0.0015 | each started block of 1,000 items read |
apify-actor-start | $0.00005 | Apify's standard run start (one per GB of memory, so 1 at the default 1,024 MB) |
| demo run, status and summary items | free |
Your maximum charge per run is respected: the run stops reading and writing when the next event would go over it and says so in RUN_SUMMARY.stoppedBecause.
Limits and notes
- Dedupe keys are kept in memory (about 100 bytes each, ESTIMATED from a local benchmark: 500,000 unique keys used about 70 MB). The default 1,024 MB covers a few million unique keys.
- Nested fields can be used for dedupe and filters, but keep/drop/rename work on top-level fields only.
1and1.0are the same number.trueis not1."20"equals20only ingt/gte/lt/lte.- The Actor reads your own datasets through the Apify platform. It fetches nothing from the web.
Terms and permissions evidence
Read on 2026-11-11 from Apify's documentation. Copies are in evidence/.
- Limited permissions, https://docs.apify.com/platform/actors/development/permissions: "By default, limited-permissions Actors can't access user storages. However, they can access storages that users explicitly provide via the Actor input. To do so, use the input schema to add a storage picker and declare exactly which operations your Actor needs." and "Add a field with
editor: "resourcePicker". SetresourceTypeto one ofdataset,keyValueStore, orrequestQueue. SpecifyresourcePermissionswith the minimal required scope". Also: "The user's selection is injected into the run input, and the run token is expanded to allow only the requested operations on the selected storages." - Resource input type, https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1/resource.md: "the resource input value is a string representing either the resource ID or (unique) name. The type of the property is either
stringorarray." - Named stores under limited permissions, https://docs.apify.com/platform/actors/development/permissions/migration-guide: "With limited permissions, an Actor can create a named storage on its first run and will automatically retain access to it in all subsequent runs by the same user."
- Get dataset API, https://docs.apify.com/api/v2/dataset-get: "datasetId ... Dataset ID or
username~dataset-name."