Dataset Merge & Dedupe

Merge one or more Apify datasets (and/or JSON items you paste in), remove duplicate rows by the key fields you choose, then optionally pick, omit, rename or cast fields. The result is written to the run's default dataset, which you can export from Apify as JSON, CSV, Excel and more.

Pricing

$0.30 per 1,000 output rows (event output-row, $0.0003 per row written) plus Apify's start fee of $0.005 per run per GB of memory (apify-actor-start), so $0.005 at the 1,024 MB default and up to $0.02 at the 4,096 MB maximum.

• The per-row charge applies only to rows written to your output dataset: duplicates removed, rows dropped by your settings and rows of the bundled sample have no per-row charge, and Apify's per-run start fee applies to every run, including sample runs and runs that stop on invalid input.
• An output row is one row in the final dataset after merging, dedupe and field changes; merging adds no rows, and rows kept by missingKeyPolicy: keep each count as one output row.
• If you set a maximum total charge on the run, the Actor writes only as many rows as that budget covers (after the start fee) and then stops with a clear message.

Your data, your responsibility

You are the controller of the data you merge; the developer acts only as your processor. Do not submit special-category personal data (for example health, biometric, religious or political data) without a lawful basis to process it.

• Reads only what you pick. The Actor is designed for Apify's limited permissions mode: the dataset field uses Apify's dataset picker with read-only access, so the run can read only the datasets you select. It never creates, changes or deletes your other storages.
• No outside calls. The Actor talks only to the Apify API. It does not fetch URLs found in your data and sends nothing to any third party.
• No row contents in logs. The log and the RUN_SUMMARY record contain counts, dataset IDs and the field names you chose, never your field values.
• Stays in your Apify account. Results are written only to this run's default storage in your Apify account; the developer does not copy or keep your rows and does not use them to train AI models.
• Formula escaping on by default. Text values starting with =, +, -, @, tab or carriage return are prefixed with ' so they are not run as formulas when you open a CSV/Excel export (turn off with escapeFormulas: false). Text such as "-5" is prefixed too; cast such fields to number with castFields if you want numbers.

Input

Field	Default	What it does
datasetIds	–	Datasets to merge, in order: pick them with the dataset picker or paste their IDs (read-only). Up to 20.
items	–	JSON array of objects to merge after the datasets. Up to 10,000 items.
dedupeFields	whole row	Field names (dot paths such as product.sku allowed) that make a row unique. Empty = compare the whole row. Up to 50 fields.
keep	first	Which copy of a duplicate to keep: first or last (in dataset order, then items).
trimWhitespace	true	Ignore leading/trailing spaces when comparing keys. Output values are not changed.
caseInsensitive	false	Compare text keys ignoring upper/lower case.
missingKeyPolicy	keep	Rows with all key fields missing: keep each one, drop them, or group them as one key.
emptyCountsAsMissing	true	Treat "" and null key values as missing.
pickFields / omitFields	–	Keep only / remove these fields in the output.
renameFields	–	Object of {"oldName": "newName"}.
castFields	–	Object of {"field": "string" | "number" | "integer" | "boolean"}. Values that cannot be cast are left unchanged and counted.
escapeFormulas	true	See "Formula escaping" above.
addSourceFields	false	Add _sourceType, _sourceDatasetId, _sourceIndex to each row.
maxOutputRows	no limit	Stop after writing this many rows.

If you give neither datasetIds nor items, the Actor runs on a small bundled sample (6 rows, 1 duplicate) so you can see the output. Sample rows have no per-row charge; the per-run start fee still applies.

Rules worth knowing:

• Keys are compared on the input field names, before pick/omit/rename/cast. pickFields, omitFields and renameFields use input names; castFields uses output (renamed) names.
• Values are compared by type: "1" and 1 are different keys.
• The output keeps the order in which kept rows appeared.
• A run reads at most 2,000,000 rows in total (see Limits).

Output

• Default dataset: the merged, de-duplicated rows.
• Key-value store record RUN_SUMMARY: counts for the run, e.g. rowsRead, duplicatesRemoved, outputRows, chargedEvents, missingKey, castFailures, formulaEscaped, budgetReached, maxOutputReached, rowsReadCapReached.

Limits

• Up to 20 datasets, 10,000 inline items and 2,000,000 rows read per run (limits set by this Actor for memory safety, not Apify limits).
• If a run reaches the 2,000,000-row read limit, it stops reading, says so in the log and RUN_SUMMARY, and removes duplicates only among the rows it read.
• The number of rows mostly affects run time; memory depends mainly on how large each row is (see Known limitations).
• Tested at 1,024 MB of memory with rows of about 500 KB each (76 rows, about 38 MB); because the first read page is 50 rows, much wider rows can run out of memory at that size, so give runs with wider rows more memory.
• If the platform restarts the run, rows already written are not written or charged again.

Known limitations

• Very large rows may need more memory. With small rows, 2,000,000 distinct keys peaked at about 491 MiB (515 MB) in a local test of the dedupe engine (development machine, not Apify; see CHANGELOG). In one Apify test, 76 rows of about 500 KB each peaked at about 174 MiB with 1,024 MB of memory; much wider rows may need 2,048 MB or more. After the first page of 50 rows, the Actor reads fewer rows per page when rows are large. If a run fails with an out-of-memory error, run it again with 2,048 MB; the start fee is per GB of memory (see Pricing). If a run runs out of memory, the rows it wrote are charged, except that the last batch (up to 500 rows) may be written without a charge. A new run starts over and writes and charges every row again.
• Text is compared as stored: no Unicode normalization (an accented letter typed two different ways counts as two values), and caseInsensitive uses simple lower-casing (ß and SS differ).
• Arrays inside key fields are order-sensitive ([1,2] and [2,1] differ).
• A field whose own name contains a dot ("a.b") cannot be used as a key, because dots separate nested fields.
• This Actor can't take over a run from another Actor (metamorph). Such runs stop at the start, and this Actor reads, writes and charges no rows. Start this Actor directly instead.
• If a run is restarted by the platform, this Actor checks its saved progress, including the content of rows it already wrote, before continuing, and stops with a message saying why if anything does not line up. In a rare case, where the run's dataset already holds rows identical to this Actor's own output and the platform's row count is delayed for more than 15 seconds, this check cannot tell those rows apart from its own, and some rows could be skipped without an error. For runs where that matters, compare the output row count with the expected count.

Support

Questions, bugs or feature requests: open an issue in the Issues tab on this Actor's page in Apify Store. Please do not paste personal or confidential row data into an issue.

Changelog

See CHANGELOG.md.