Merge, Dedup & Transform Datasets avatar
Merge, Dedup & Transform Datasets
Try for free

No credit card required

View all Actors
Merge, Dedup & Transform Datasets

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets
Try for free

No credit card required

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Dataset IDs

datasetIdsarrayRequired

Datasets that should be deduplicated and merged

Fields for deduplication

fieldsarrayOptional

Fields whose combination should be unique for the item to be considered unique. If none are provided, the actor does not perform deduplication.

What to output

outputEnumOptional

What will be pushed to the dataset from this actor

Value options:

"unique-items": string"duplicate-items": string"nothing": string

Default value of this property is "unique-items"

Mode

modeEnumOptional

How the loading and deduplication process will work.

Value options:

"dedup-after-load": string"dedup-as-loading": string

Default value of this property is "dedup-after-load"

Output dataset ID or name (optional)

outputDatasetIdstringOptional

Optionally can push into dataset of your choice. If you provide a dataset name that doesn't exist, a new named dataset will be created.

Limit fields to load

fieldsToLoadarrayOptional

You can choose which fields to load only. Useful to speed up the loading and reduce memory needs.

Pre dedup transform function

preDedupTransformFunctionstringOptional

Function to transform items before deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

Post dedup transform function

postDedupTransformFunctionstringOptional

Function to transform items after deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

Where to output

outputToEnumOptional

Either can output to a single dataset or to split data into KV records depending on upload batch size. KV is upload is much faster but data end up in many files.

Value options:

"dataset": string"key-value-store": string

Default value of this property is "dataset"

Parallel loads

parallelLoadsintegerOptional

Datasets can be loaded in parallel batches to speed things up if needed.

Default value of this property is 10

Parallel pushes

parallelPushesintegerOptional

Deduped data can be pushed in parallel batches to speed things up if needed. If you want the data to be in the exact same order, you need to set this to 1.

Default value of this property is 5

Upload batch size

uploadBatchSizeintegerOptional

How many items it should upload in one pushData call. Useful to not overload Apify API. Only important for dataset upload.

Default value of this property is 500

Download batch size

batchSizeLoadintegerOptional

How many items it will load in a single batch.

Default value of this property is 50000

Offset (how many items to skip from start)

offsetintegerOptional

By default we don't skip any items which is the same as setting offset to 0. For multiple datasets, it takes offset into the sum of their item counts but that is not very useful.

Limit (how many items to load)

limitintegerOptional

By default we don't limit the number loaded items

verbose log

verboseLogbooleanOptional

Good for smaller runs. Large runs might run out of log space.

Default value of this property is false

Dataset IDs for just deduping

datasetIdsOfFilterItemsarrayOptional

The items from these datasets will be just used as a dedup filter for the main datasets. These items are loaded first and then the main datasets are compared for uniqueness and pushed.

Custom input data

customInputDataobjectOptional

You can pass custom data as a JSON object to be accessible in the transform functions as part of the 2nd parameter object.

Developer
Maintained by Apify
Actor metrics
  • 265 monthly users
  • 97.8% runs succeeded
  • 0.4 days response time
  • Created in Apr 2020
  • Modified 2 months ago