Merge datasets and remove duplicates by a key field
Quiet Works Lab
Combines up to 10 of your Apify datasets into one and removes duplicates by the fields you choose, such as url or sku, keeping the first or the last occurrence. Output rows keep your own fields at the top level, ready for a spreadsheet or database. Runs with limited permissions and reads only the datasets you pick. Run it as it is to merge two small built-in sample datasets.
Input
Output fields
Sign up on Apify or log in
The Merge, Deduplicate & Join Datasets Actor is ready to run with the inputs already set for "Merge datasets and remove duplicates by a key field". Sign up or log in and you'll land in Apify Console with a copy of this exact task.
Start the "Merge datasets and remove duplicates by a key field" task
The Merge, Deduplicate & Join Datasets Actor will start running based on the input automatically. You can also change all the input fields, like remove duplicates and which duplicate to keep, to match your specific use case.
Receive the "Merge datasets and remove duplicates by a key field" output
Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as sku, name, maker, and more.
Integrate into your workflow
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "Merge datasets and remove duplicates by a key field" task via API.
Integrate "Merge datasets and remove duplicates by a key field" into your workflow
Connect "Merge datasets and remove duplicates by a key field" with 100+ integrations, or run it via API or MCP.
Webhook
n8n
Make
Zapier
Airbyte
Keboola
IFTTT
Hubspot
GDrive
Gmail
Apify MCP
GitHub
Slack
LangChain
LlamaIndex
Flowise
Pinecone
OpenAI
Mastra
Clay
The merge and deduplicate task compares the duplicate key fields you choose, for example url or sku (dots for nested fields). Without key fields, two items are duplicates only when all their fields are equal.
The merge and deduplicate task keeps the first occurrence by default, in the order the datasets are read. With keep set to last it keeps the item from the dataset read last, so put the newest data last.
The Merge, Deduplicate & Join Datasets Actor charges $0.0001 per item read ($0.10 per 1,000 items) and one item per started 20 items of a lookup dataset, plus Apify's Actor start event. The built-in sample is not charged per item.