Merge, Dedup & Transform Datasets

Pricing

Pay per usage

Try for free

Go to Apify Store

Merge, Dedup & Transform Datasets

Try for free

Developed by

Lukáš Křivka

Maintained by Community

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

0.0 (0)

Pricing

Pay per usage

4.4K

140

Last modified

7 months ago

Automation

Developer tools

Open source

Dataset IDs

datasetIdsarrayOptional

Datasets that should be deduplicated and merged

Fields for deduplication

fieldsarrayOptional

Fields whose combination should be unique for the item to be considered unique. If none are provided, the actor does not perform deduplication.

What to output

outputEnumOptional

What will be pushed to the dataset from this actor

Value options:

"unique-items": string"duplicate-items": string"nothing": string

Default value of this property is "unique-items"

Mode

modeEnumOptional

How the loading and deduplication process will work.

Value options:

"dedup-after-load": string"dedup-as-loading": string

Default value of this property is "dedup-after-load"

Output dataset ID or name (optional)

outputDatasetIdstringOptional

Optionally can push into dataset of your choice. If you provide a dataset name that doesn't exist, a new named dataset will be created.

Limit fields to load

fieldsToLoadarrayOptional

You can choose which fields to load only. Useful to speed up the loading and reduce memory needs.

Pre dedup transform function

preDedupTransformFunctionstringOptional

Function to transform items before deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

Post dedup transform function

postDedupTransformFunctionstringOptional

Function to transform items after deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

Actor or Task ID (or name)

actorOrTaskIdstringOptional

Use Actor or Task ID (e.g. nwua9Gu5YrADL7ZDj) or full name (e.g. apify/instagram-scraper).

Only runs newer than

onlyRunsNewerThanstringOptional

Use a date format of either YYYY-MM-DD or with time YYYY-MM-DDTHH:mm:ss.

Only runs older than

onlyRunsOlderThanstringOptional

Use a date format of either YYYY-MM-DD or with time YYYY-MM-DDTHH:mm:ss.

Where to output

outputToEnumOptional

Either can output to a single dataset or to split data into KV records depending on upload batch size. KV is upload is much faster but data end up in many files.

Value options:

"dataset": string"key-value-store": string

Default value of this property is "dataset"

Parallel loads

parallelLoadsintegerOptional

Datasets can be loaded in parallel batches to speed things up if needed.

Default value of this property is 10

Parallel pushes

parallelPushesintegerOptional

Deduped data can be pushed in parallel batches to speed things up if needed. If you want the data to be in the exact same order, you need to set this to 1.

Default value of this property is 5

Upload batch size

uploadBatchSizeintegerOptional

How many items it should upload in one pushData call. Useful to not overload Apify API. Only important for dataset upload.

Default value of this property is 500

Download batch size

batchSizeLoadintegerOptional

How many items it will load in a single batch.

Default value of this property is 50000

Offset (how many items to skip from start)

offsetintegerOptional

By default we don't skip any items which is the same as setting offset to 0. For multiple datasets, it takes offset into the sum of their item counts but that is not very useful.

Limit (how many items to load)

limitintegerOptional

By default we don't limit the number loaded items

verbose log

verboseLogbooleanOptional

Good for smaller runs. Large runs might run out of log space.

Default value of this property is false

Null fields are unique

nullAsUniquebooleanOptional

If you want to treat null (or missing) fields as always unique items.

Default value of this property is false

Dataset IDs for just deduping

datasetIdsOfFilterItemsarrayOptional

The items from these datasets will be just used as a dedup filter for the main datasets. These items are loaded first and then the main datasets are compared for uniqueness and pushed.

Custom input data

customInputDataobjectOptional

You can pass custom data as a JSON object to be accessible in the transform functions as part of the 2nd parameter object.

Append dataset IDs to items

appendDatasetIdsbooleanOptional

Useful for transform functions. Each item will get a field __datasetId__ with the dataset ID it came from.

Default value of this property is false

🔥 Power Data Transformer

wiseek/power-data-transformer

🔥 Unlock your scraped data—clean, merge, split, deduplicate, filter, standardize, validate, enrich and sync—using built-in transformations and powerful SQL pipelines for ETL/ELT workflows. Seamlessly integrate processed datasets with automation platforms like n8n, Make.com, and Zapier.

wiseek

Duplications Checker

lukaskrivka/duplications-checker

Check your dataset for duplications. Accept only the highest quality data!

Lukáš Křivka

231

LLM Dataset Processor

dusan.vystrcil/llm-dataset-processor

Allows you to process output of other actors or stored dataset with single LLM prompt. It's useful if you need to enrich data, summarize content, extract specific information, or manipulate data in a structured way using AI.

Dušan Vystrčil

Doctor Data Merger

muhammetakkurtt/doctor-data-merger

This actor is ideal for researchers, healthcare analysts, or anyone needing to combine doctor profiles with patient feedback for comprehensive analysis.