Merge, Dedup & Transform Datasets

Datasets that should be deduplicated and merged

Fields whose combination should be unique for the item to be considered unique. If none are provided, the actor does not perform deduplication.

What will be pushed to the dataset from this actor

How the loading and deduplication process will work.

Optionally can push into dataset of your choice. If you provide a dataset name that doesn't exist, a new named dataset will be created.

You can choose which fields to load only. Useful to speed up the loading and reduce memory needs.

Function to transform items before deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

Function to transform items after deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

Either can output to a single dataset or to split data into KV records depending on upload batch size. KV is upload is much faster but data end up in many files.

Datasets can be loaded in parallel batches to speed things up if needed.

Deduped data can be pushed in parallel batches to speed things up if needed. If you want the data to be in the exact same order, you need to set this to 1.

How many items it should upload in one pushData call. Useful to not overload Apify API. Only important for dataset upload.

How many items it will load in a single batch.

By default we don't skip any items which is the same as setting offset to 0. For multiple datasets, it takes offset into the sum of their item counts but that is not very useful.

By default we don't limit the number loaded items

Good for smaller runs. Large runs might run out of log space.

The items from these datasets will be just used as a dedup filter for the main datasets. These items are loaded first and then the main datasets are compared for uniqueness and pushed.

You can pass custom data as a JSON object to be accessible in the transform functions as part of the 2nd parameter object.