Pricing

from $0.05 / 1,000 row outputs

Try for free

Go to Apify Store

Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX

Try for free

Merge Apify datasets, CSV, JSON, JSONL or Excel files and pasted rows into one clean dataset: remove duplicates by any fields (case and whitespace normalised), keep only items new since your last run, left/inner join on a key, flatten nested JSON. $0.05 per 1,000 rows out.

Pricing

from $0.05 / 1,000 row outputs

Rating

0.0

(0)

Developer

ActorForge

Actor stats

Bookmarked

Total users

Monthly active users

3 days ago

Last modified

Categories

Developer tools

Automation

Integrations

You can access the Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {
11    "rows": [
12        {
13            "id": 1,
14            "name": "a"
15        },
16        {
17            "id": 1,
18            "name": "a"
19        },
20        {
21            "id": 2,
22            "name": "b"
23        }
24    ],
25    "dedupeFields": [
26        "id"
27    ]
28};
29
30// Run the Actor and wait for it to finish
31const run = await client.actor("actor-forge/dataset-merge-dedupe").call(input);
32
33// Fetch and print Actor results from the run's dataset (if any)
34console.log('Results from dataset');
35console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
36const { items } = await client.dataset(run.defaultDatasetId).listItems();
37items.forEach((item) => {
38    console.dir(item);
39});
40
41// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Merge & Deduplicate Apify Datasets - Only New Items API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX API in Python

Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX API through CLI

Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX OpenAPI definition

Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX API

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Lukáš Křivka

5.4K

5.0

(2)

Dataset Join & Merge (VLOOKUP for Datasets)

nerolabs/dataset-join-merge

Returns two Apify datasets, files or Google Sheets by URL, or JSON arrays joined on key fields like SQL or VLOOKUP: left, inner, right, full and anti joins plus union, with CSV/Excel export, named-dataset append and webhook delivery. Agent-ready: pay per event (x402, MCP), per output row.

Adam Pearce

Merge & Deduplicate Datasets - Reliable, Any Size

geese_chief/dataset-merge-dedup

Merge any number of Apify datasets into one and remove duplicates by the fields you choose (url, id, email...). Keep unique items or only the duplicates, pick output fields, tag each item with its source. Streams data, so millions of items work. Pick datasets from your storage: no permission errors.

Ponytail Data

Merge Datasets & Remove Duplicates: Combine Runs, CSVs, Sheets

nerolabs/dataset-merge-dedupe

Returns one de-duplicated list from many Apify datasets, CSV, Excel or JSON files and Google Sheets, with blanks filled from the copies and each row's source labelled. Inputs: datasetIds, fileUrls, dedupKeys. Charged per merged row written ($0.001). Agent-ready via x402 and MCP.

Adam Pearce

Deduplicate, Merge & Transform Datasets

datacach/deduplicate-datasets

Merge multiple datasets, deduplicate items by a combination of fields, and apply custom transforms — powered by Polars.

DataCach

JSON Dataset Cleaner and Deduplicator

rodrgds/dataset-cleaner

Clean JSON datasets, remove empty rows, deduplicate by any field, validate emails, and prepare scraper output for CRMs, analysis, or AI workflows.

Rodrigo Dias

Dedup & Merge Datasets — Apify, JSON, CSV, $1.50/1K

kadro/dedup-merge-datasets

Merge Apify datasets, JSON and CSV files, then remove duplicate rows by the fields you choose.

Kadro Tools

Merge & Deduplicate Datasets: Only New Items, CSV/Excel Export

tinplatelabs/dataset-toolkit

Merge and deduplicate Apify datasets and CSV, JSON, Excel or Parquet files in one run. Remove duplicates (exact, normalized or fuzzy), output only new items since your last run, filter, sort, rename, then export to CSV, Excel, JSON or Parquet. $0.002 per 1,000 rows.

tinplatelabs

Line Deduplicator, Sorter & Shuffler

hipersoft/line-dedup-sort

Clean big text and list files: remove duplicate lines, sort alphabetically or numerically, reverse, shuffle and trim blanks — handles lists too large for a desktop editor.

hiper soft

Dataset Filter, Merge & Dedup – Clean & Export CSV, Excel, JSON

gazidev/dataset-filter-transform

No-code filter, merge and dedupe for Apify datasets, CSV, XLSX and JSON files: combine several datasets, remove duplicates by fields, filter rules (equals, contains, regex, ranges, dates, AND/OR), only-new-since-last-run, rename/select/drop, flatten, sort. Export CSV/XLSX/JSON. Pay per row.