One or more Apify dataset IDs (e.g. dHZ7Xy9aBc...) or dataset names to merge and deduplicate. Items are concatenated in the order listed, which decides which duplicate is kept — the first occurrence wins. Leave this empty (and Actor or Task ID empty too) to run on built-in sample data for a quick first test.

Type:array

Deduplication fields

fields

Optional

The field(s) whose combined value makes an item unique — e.g. ["url"] or ["name", "id"]. Each value is JSON-stringified and concatenated into a single key, and the first item seen for each key is kept. Leave empty to merge datasets without removing any duplicates.

Type:array

Items to output

output

Optional

Choose what the run returns: the unique items, only the duplicate items that were removed, or nothing (just the unique/duplicate counts — handy for a quick duplicate audit).

Type:string

Default:unique-items

Options:

unique-itemsduplicate-itemsnothing

Deduplication mode

mode

Optional

Dedup after load loads everything into memory, then deduplicates — fastest for typical datasets. Dedup as loading deduplicates batch-by-batch and streams the result to the output dataset as it goes, keeping memory near-constant for very large (10M+) datasets. Streaming applies when the output destination is a dataset; the post-dedup transform is then applied per pushed batch.

Type:string

Default:dedup-after-load

Options:

dedup-after-loaddedup-as-loading

Treat null fields as unique

nullAsUnique

Optional

When on, items whose deduplication field(s) are null or missing are always kept as unique and never removed. When off, a null/missing value is treated like any other value when matching duplicates.

Type:boolean

Default:false

Actor or Task ID

actorOrTaskId

Optional

Optionally pull source datasets from every run of an Actor or Task. Enter an ID or full name (e.g. apify/web-scraper); the default dataset of each successful run is merged together with anything in Dataset IDs.

Type:string

Only runs newer than

onlyRunsNewerThan

Optional

When using Actor or Task ID, only include runs that finished on or after this date. Accepts a date (2024-01-31) or a full ISO datetime.

Type:string

Only runs older than

onlyRunsOlderThan

Optional

When using Actor or Task ID, only include runs that finished on or before this date. Accepts a date (2024-01-31) or a full ISO datetime.

Type:string

Dataset IDs of filter items

datasetIdsOfFilterItems

Optional

Datasets used only to pre-seed the 'already seen' keys. Items in Dataset IDs that match these keys are dropped as duplicates, but the filter items themselves are never output — perfect for outputting only records you haven't seen before.

Type:array

Pre-deduplication transform function

preDedupTransformFunction

Optional

Optional Python that receives items (a list of dicts) plus input_data/customInputData and returns a new list of dicts. Runs on each loaded batch before deduplication, so you can filter or add items. Example: [i for i in items if i.get('price')]. You can also provide a full function, e.g. def transform(items, input_data): .... JavaScript functions are not supported.

Type:string

Post-deduplication transform function

postDedupTransformFunction

Optional

Optional Python that receives items (a list of dicts) and returns a new list of dicts. Runs after deduplication, just before the result is written — ideal for trimming or renaming fields.

Type:string

Custom input data

customInputData

Optional

Arbitrary JSON passed straight to your transform functions as input_data / customInputData. Use it to parameterize the transforms without editing their code.

Type:object

Output destination

outputTo

Optional

Where to store the result: a dataset (downloadable as JSON, CSV, Excel, or HTML) or the run's key-value store under the OUTPUT key (results over the 9 MB record limit are split into OUTPUT, OUTPUT-2, ...).

Type:string

Default:dataset

Options:

datasetkey-value-store

Output dataset ID

outputDatasetId

Optional

Dataset ID or name to push the output to. If a name doesn't exist yet, a named dataset is created. Leave empty to use this run's default dataset.

Type:string

Append dataset IDs

appendDatasetIds

Optional

When on, every output item gets a datasetId field recording which source dataset it came from. When off, items are passed through unchanged.

Type:boolean

Default:false

Fields to load

fieldsToLoad

Optional

Load only these fields from the source datasets to save memory and speed up loading — for example, just your deduplication fields when you only need counts. Your deduplication fields are always loaded in addition to this list, so dedup keys can be computed. Leave empty to load every field.

Type:array

Offset

offset

Optional

Skip this many items from the start of the merged input before processing. Leave empty to start from the first item.

Type:integer

Minimum:0

Limit

limit

Optional

Process at most this many items from the merged input. Leave empty to process all of them.

Type:integer

Minimum:1

Load batch size

batchSizeLoad

Optional

How many items to fetch per load request. Larger batches load faster but use more memory.

Type:integer

Minimum:1000

Default:50000

Upload batch size

uploadBatchSize

Optional

How many items to send per push request when writing the output.

Type:integer

Minimum:10

Maximum:1000

Default:500

Parallel loads

parallelLoads

Optional

How many load batches to fetch at the same time. Higher values speed up large runs but use more memory and resources.

Type:integer

Minimum:1

Maximum:100

Default:10

Parallel pushes

parallelPushes

Optional

How many output batches to upload at the same time.

Type:integer

Minimum:1

Maximum:50

Default:5

Verbose logging

verboseLog

Optional

Turn on detailed debug logging to trace exactly what the run is doing. Leave off for normal, quieter logs.

Type:boolean

Default:false

Dataset Deduplicator

zentrafoundry/dataset-deduplicator

Deduplicate Apify datasets and create stable merge keys.

Zentra

Dataset Deduplicator

automation-lab/dataset-dedup

Merge and deduplicate Apify datasets by any field combination. Remove duplicates, keep first or last occurrence. Case-insensitive matching, whitespace trimming. Pay per 1K items processed.

Stas Persiianenko

Data Deduplicator

parsebird/dataset-deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows while keeping the first or last occurrence. Supports case-insensitive matching and whitespace trimming.

ParseBird

Lead List Deduplicator & Merger

jurassic_jove/lead-deduplicator-merger

Merge and deduplicate lead lists from multiple Apify datasets, CSV files and inline JSON into one clean, outreach-ready list. Pure data processor — no scraping, no proxies, no external APIs.

Data Runner

Dataset Toolbox

cyberfly/dataset-toolbox

Perform common actions on datasets - merge, unify, validate, transform, order fields etc.

Vasek Codey Vlcek

Merge Key-Value store pieces

pocesar/merge-key-value-store-pieces

Paulo Cesar

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Lukáš Křivka

5.1K

5.0

Lead Intelligence Scorer Pro

leadops_lab/lead-intelligence-scorer-pro

Clean, deduplicate, score, and prepare B2B leads from Apify datasets for CRM import and outreach workflows.

jiaxun mao

🔥 Power Data Transformer

wiseek/power-data-transformer

🔥 Unlock your scraped data—clean, merge, split, deduplicate, filter, standardize, validate, enrich and sync—using built-in transformations and powerful SQL pipelines for ETL/ELT workflows. Seamlessly integrate processed datasets with automation platforms like n8n, Make.com, and Zapier.

wiseek

42 Polars AI Data Transformer

salesmart-srl/polars-ai-data-transformer

Transform datasets using natural language. Upload CSV/Excel/JSON, describe your transformation in plain English, get results + reusable Python code. Powered by AI.

Salesmart Srl