Dataset Merge and JavaScript Deduplication

Combine Apify Datasets, keep the first occurrence or later duplicates, and run async JavaScript before and after selection. Read explicit Datasets, resolve a run to its default Dataset, or collect run Datasets from an Actor or Task history. Deliver to one Dataset or complete JSON arrays in key-value storage.

Use this Actor for Dataset cleanup, combining recurring exports, excluding previously seen records, and transformations written in JavaScript. It runs on Node 24 with the Apify SDK 2.3.2 API; the pinned SQLite compatibility override is 13.0.3.

{
  "datasetIds": [
    "peerless_columbine~dataset-merge-js-example-posts",
    "peerless_columbine~dataset-merge-js-example-posts"
  ],
  "fields": [
    "postId"
  ],
  "mode": "dedup-after-load",
  "output": "unique-items",
  "outputTo": "dataset"
}

The default example reads the same small demonstration Dataset twice and keeps one record per postId: eight input occurrences produce four distinct real post records. These records were obtained from an existing Apify Dataset on 2026-10-15; this Actor processes supplied Datasets and does not crawl the original websites. The small demonstration Dataset is publicly readable by name. For your own workflow, replace datasetIds and choose the identity fields present in your records. Repeated Dataset references remain repeated occurrences. A supplied run ID is resolved only after the Dataset lookup misses. A Dataset miss followed by a run miss is listed in SUMMARY.missingSources. Available sources still run, but any missing main/filter source makes the result PARTIAL, complete: false; an omitted filter is not counted as a successfully applied constraint. If all requested main sources are missing, SOURCE_NOT_FOUND stops before transforms or delivery. A real existing zero-row Dataset, or a successful history lookup with no date-window matches, can finish as a valid empty result. Permission or transport errors also stop incomplete.

The complete history workflow and unrestricted SDK helpers require Full permissions, subject to platform access controls for the actual resources. Other users approve that scope on their first Console run; owners running their own Actor do not need that additional approval step. Limited-permission, agentic-prepaid and MCP compatibility have not been demonstrated. See the official permissions documentation.

Selection and identity

unique-items keeps a first occurrence. duplicate-items emits later occurrences with a duplicationKey diagnostic property. nothing runs membership and transforms without managed output writes. Leaving fields empty merges rows without identity selection, including repeated input sources.

Fields are literal top-level names. For each component, scalar coercion is retained: numeric 1 and string "1" match; missing and empty string match unless null uniqueness is enabled. Objects and arrays use JSON.stringify, including object insertion order; property order is not sorted. Null normally matches the string "null". With nullAsUnique, any null or missing component receives an occurrence identity in a separate namespace, without a random-string collision.

Internal identities encode the complete sequence of field components. Consequently ("ab", "c") differs from ("a", "bc"). The displayed duplicationKey remains the old concatenated diagnostic string for post-transform compatibility; it is not the membership key and need not be globally unique. A pre-existing row property named duplicationKey is retained. Unique-null diagnostics use a deterministic occurrence label.

Sources, history and slicing

actorOrTaskId accepts an Actor/Task ID or owner/name. A Task lookup is tried first; a genuine miss falls back to the Actor lookup. History pages are requested in descending order, with inclusive startedAt bounds onlyRunsNewerThan and onlyRunsOlderThan. A date-only upper bound is midnight at the start of that date. All history run statuses are eligible when they have a default Dataset. Their Datasets follow explicit datasetIds; repeated IDs are not removed.

The resolved plan stores each Dataset's initial item count and its source/run identity. Saved source pages are reused on recovery. A history page anchor that moved, a short page, or a malformed source response stops with an explicit source gap. The frozen plan does not promise a transactional snapshot of a Dataset that another writer edits while it is being read. New appends beyond the initial item count are not included in that plan.

By default offset and limit apply separately to every main Dataset, matching the observed reference workflow. For two four-row Datasets, offset 1 / limit 2 yields four rows. Our optional slicingScope: "combined" counts across concatenated inputs and yields two rows in that example. This explicit option resolves a reference description/runtime discrepancy; it does not remove per-Dataset behavior. Filters are not sliced. Projection through fieldsToLoad happens before transforms; include properties required by your JavaScript.

JavaScript and processing modes

{
  "preDedupTransformFunction": "async (items, { customInputData, persistedSharedObject }) => { persistedSharedObject.batches = (persistedSharedObject.batches || 0) + 1; return items.map(row => ({ ...row, organization: row.organization.toLowerCase(), tag: customInputData.tag })); }",
  "postDedupTransformFunction": "async (items, { diff }) => items.map(row => ({ ...row, reviewed: true }))",
  "customInputData": { "tag": "reviewed" }
}

Both functions must return arrays of row objects for delivery. They receive the genuine Apify SDK 2.3.2, customInputData, persistedSharedObject, and the fast-diff helper in their context. SDK methods, async work and ordinary Node capabilities remain available. Ordinary eval is not a sandbox. Run only code you trust. Such code can make API calls, alter other storage, incur costs, expand output, or have side effects outside this Actor's journal. Browser binaries are not included.

In dedup-after-load, filter rows seed membership without transforms; main rows are loaded together and pre/post run once over the whole main array. This mode uses memory proportional to all loaded rows plus transformed output and membership.

In dedup-as-loading, pre/post run per main page with datasetId and datasetOffset. Filter pages also run pre, with those identifiers, before any main page. As in the reference, that filter pre context does not include diff. Membership and JSON checkpoints persist across pages. Source reads may be concurrent; transformation and durable unit commits are ordered. Memory still grows with all distinct identities, so this is not constant-memory deduplication.

appendDatasetIds attaches __datasetId__ before transforms. persistedSharedObject must be JSON-serializable; arbitrary closures/functions cannot be persisted. On a known prepared unit, recovery uses staged output rather than invoking its transforms again. Function factory expressions can be evaluated again when a later unit starts in a new process, so external side effects in factories are not restart-transactional.

Output and ordering

Omit outputDatasetId to append to the run default Dataset, or supply an existing ID/name or a new name. Existing rows are preserved. outputTo: "key-value-store" writes complete arrays and lists every record, count and hash in SUMMARY.outputRecords:

• After-load: OUTPUT-0, OUTPUT-1, …
• As-loading: OUTPUT-<datasetId>[-<nonzeroOffset>]-<chunkIndex>.

An actual key collision from a repeated source occurrence receives -source-<sourceOrdinal> rather than silently overwriting earlier rows. This narrow correction is explicit; ordinary keys remain compatible. Byte-size splitting can create more chunks than count-only batching. Dataset and KVS payloads are split at 8 MiB; an individual larger row fails explicitly and is not truncated. For output nothing, no managed row validation/write is required after the post function.

With parallelPushes: 1, append order follows the planned source and transform order. Higher concurrency permits backend append reordering. Upload groups use uploadBatchSize × parallelPushes; Dataset groups are divided among writers, while KVS groups remain complete arrays. Safe state commits do not imply that arbitrary user-code side effects are serialized or transactionally ordered.

Input reference

All 23 original inputs remain available; slicingScope is an additional optional input. No debugPlatform field is required.

Input	Meaning / default
datasetIds	Dataset IDs/names or run IDs; default empty array
fields	Literal identity names; omitted/empty means merge
output	unique-items, duplicate-items, nothing; default unique
mode	dedup-after-load or dedup-as-loading; default after-load
outputDatasetId	Optional append destination ID/name
fieldsToLoad	Optional source projection
preDedupTransformFunction	Async function expression; default identity
postDedupTransformFunction	Async function expression; default identity
actorOrTaskId	Optional history source ID/name
onlyRunsNewerThan	Optional inclusive history start lower bound
onlyRunsOlderThan	Optional inclusive history start upper bound
outputTo	dataset or key-value-store; default Dataset
parallelLoads	Default 10, maximum 100; nonpositive means all finite planned pages
parallelPushes	Default 5, range 1–50; see normalization below
uploadBatchSize	Schema/UI default 500, range 10–1000; raw omitted runtime default 1000
batchSizeLoad	Raw default 50000; positive page size required to make progress
offset	Default 0, applied using the chosen slicing scope
limit	Raw default 999999999, applied before transforms
verboseLog	Default false; source-range and commit progress without row bodies
nullAsUnique	Default false
datasetIdsOfFilterItems	Optional membership-only source Datasets
customInputData	Optional JSON object; omitted context value remains undefined
appendDatasetIds	Default false
slicingScope	Additional option: per-dataset default or combined

For as-loading, supplied parallelPushes > 1 becomes 1, and supplied batchSizeLoad is replaced by supplied uploadBatchSize, before runtime defaults. Therefore a UI-populated input (upload 500 / pushes 5) becomes page 500 / pushes 1. A raw input omitting those properties retains upload 1000 / page 50000 / pushes 5. These paths are distinct; a raw API input may differ from a form-populated input. Nonpositive page size is rejected instead of attempting a nonterminating page plan.

Delivery, migration and incomplete results

DEDUP-STATE freezes input identity, source/history plan, acknowledged unit progress and shared state. DEDUP-UNIT-* separates transform intent, prepared payloads and each output write's intent/ACK. Staged blobs have size/hash checks. Keys beginning DEDUP- are execution records; do not edit or delete them while a run can resume.

A known ACK is never appended again on recovery. A checkpoint with only UNSENT chunks resumes those chunks. A request that may have succeeded without a durable ACK is reported WRITE_UNKNOWN; it is not automatically sent again, even if rows appear in the destination. Likewise a user-JS invocation that never durably prepared its result is TRANSFORM_UNKNOWN and is not automatically re-invoked. A destination-open/create without an ACK stops uncertain rather than blindly creating another destination.

Managed API requests use zero automatic retries. There is no distributed exactly-once guarantee: platform storage, user JavaScript and the journal are separate systems. Parallel processes sharing the same default KVS are not a supported lock protocol. Use platform migration/restart of the same execution storage, not concurrent runs pointed at one checkpoint. A source-read failure can resume from already cached pages; changing input against existing state is rejected.

On migration/abort, started writes drain and state is saved. In the cloud the main callback then waits for platform termination rather than returning a successful completion. The local test control may return CHECKPOINTED. SUMMARY.complete remains false for checkpointed, source-error, transform-error and unknown-write cases. Inspect deliveredRowsAcked separately from unknownRowsUpperBound; neither is an invoice or a customer payment.

Managed service events and cost boundary

Managed output uses the run's effective PAY_PER_EVENT policy. The service fee is $0.0001 per delivered row ($0.10 per 1,000 rows), plus an automatic $0.005 start event at the default 512 MB. Default-Dataset item charging is disabled by leaving that optional synthetic event unconfigured; it is not a second row fee. A native fee cap and the complete configured-event/count maps are required. Missing pricing metadata and unexpected charges stop the run.

Before each output request, the Actor reserves fee capacity for its exact row count. After a durable output ACK it sends the custom charge for those delivered post-transform rows. Filter-only rows, removed duplicates and nothing do not trigger this event. The default Dataset, a specified Dataset and KVS output follow the same managed row price. A positive automatic default-Dataset fee is rejected to prevent double charging. Output ACK and charge ACK are separate records, and the source unit is not complete until both are durable.

If a whole prepared chunk cannot fit, it remains UNSENT with FEE_LIMIT; the Actor does not raise the cap, drop the remainder, or call the run complete. A charge without a durable 201 ACK stops CHARGE_UNKNOWN. It is not replayed after restart, even with the same idempotency key: the provider's short idempotency window is not a durable replay guarantee. Receipts contain request/response hashes and status, not authorization headers. Explicit official rejection is reported separately.

The start event still applies when no output rows are delivered. Arbitrary user JavaScript can bypass these managed operations; the service-event limiter is not a sandbox or a cap on SDK calls written by the user.

Platform compute, reads, writes, storage, retention and custom JavaScript consume resources outside the managed service-event cap. Check the Actor Pricing page for the current prices and platform-usage terms. Larger transforms can produce more rows than source limits.

The sample is small. Memory usage grows with loaded records and distinct identities; no throughput guarantee or automatic recovery of uncertain external side effects is implied.

The image uses Node 24 LTS and pinned dependencies. Browser automation and every optional SDK integration are outside the demonstrated workflow.