Pricing

from $0.02 / 1,000 item processeds

Try for free

Go to Apify Store

Dedupe and Merge Datasets: Only New Items

Try for free

Merge Apify datasets, runs and JSON or CSV files, remove duplicates by key fields with URL, email, phone and fuzzy matching, and output only items not seen in earlier runs. Runs after any scraper as an integration.

Pricing

from $0.02 / 1,000 item processeds

Rating

0.0

(0)

Developer

Plainsource

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

Integrations

You can access the Dedupe and Merge Datasets: Only New Items programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1from apify_client import ApifyClient
2
3# Initialize the ApifyClient with your Apify API token
4# Replace '<YOUR_API_TOKEN>' with your token.
5client = ApifyClient("<YOUR_API_TOKEN>")
6
7# Prepare the Actor input
8run_input = {
9    "rawData": [
10        {
11            "company": "Northwind Traders",
12            "website": "https://www.northwind.example/?utm_source=directory",
13            "email": "Sales@Northwind.example",
14            "phone": "(415) 555-2671",
15            "city": "San Francisco",
16        },
17        {
18            "company": "Northwind Traders Inc.",
19            "website": "http://northwind.example/",
20            "email": "sales@northwind.example",
21            "phone": "+1 415 555 2671",
22            "employees": 120,
23        },
24        {
25            "company": "Contoso Ltd",
26            "website": "https://contoso.example/about",
27            "phone": "212-555-0187",
28            "city": "New York",
29        },
30        {
31            "company": "Contoso",
32            "website": "https://www.contoso.example/about/#team",
33            "email": "hello@contoso.example",
34            "city": "",
35        },
36        {
37            "company": "Fabrikam",
38            "website": "https://fabrikam.example",
39            "email": "info@fabrikam.example",
40            "city": "Austin",
41        },
42        {
43            "company": "Tailspin Toys",
44            "website": "https://tailspin.example/shop?gclid=abc123",
45            "city": "Denver",
46        },
47        {
48            "company": "Tailspin Toys",
49            "website": "https://tailspin.example/shop",
50            "phone": "720 555 0144",
51            "city": "Denver",
52        },
53    ],
54    "dedupeFields": ["website"],
55    "urlFields": ["website"],
56    "keep": "merge",
57}
58
59# Run the Actor and wait for it to finish
60run = client.actor("fernandosord/dataset-merge-dedupe").call(run_input=run_input)
61
62# Fetch and print Actor results from the run's dataset (if there are any)
63print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
64for item in client.dataset(run.default_dataset_id).iterate_items():
65    print(item)
66
67# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

Remove Duplicates from JSON and CSV Data API in Python

The Apify API client for Python is the official library that allows you to use Dedupe and Merge Datasets: Only New Items API in Python, providing convenience functions and automatic retries on errors.

Install the apify-client

$pip install apify-client

Other API clients include:

Dedupe and Merge Datasets: Only New Items API in JavaScript

Dedupe and Merge Datasets: Only New Items API through CLI

Dedupe and Merge Datasets: Only New Items OpenAPI definition

Dedupe and Merge Datasets: Only New Items API

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Lukáš Křivka

5.4K

5.0

(2)

Merge & Dedup Datasets: Remove Duplicates, Only New Items

charlesmarkette/merge-deduplicate-datasets-only-new-items

Merge any number of Apify datasets and remove duplicates by the fields you choose (or the whole item). Keep the first or last copy, or output only items not seen in earlier runs: ideal after a scheduled scraper. Streams millions of items. Duplicates are free; $0.0019 per 1,000 items output.

Charles Markette

Data Deduplicator

parsebird/dataset-deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows while keeping the first or last occurrence. Supports case-insensitive matching and whitespace trimming.

ParseBird

CSV & Excel Dataset Merger / Splitter [Bulk Optimizer]

thecfmarketing/csv-xlsx-dataset-merger-pattern-splitter

High-speed tool to merge multiple CSV & Excel files or split large datasets into smaller chunks by rows, file size, or specific patterns. Clean duplicates, optimize data formats, and structure columns. The ultimate utility for handling millions of B2B cold email leads and data lists.

sadique shaikh

Dataset Transformer - Filter, Dedupe, Merge & Export CSV

seemuapps/dataset-transformer

Filter, deduplicate, merge, sort, rename and flatten Apify datasets from any scraper, then export clean rows to a dataset, CSV or JSON file.

Seemu Scraping

Merge & Deduplicate Datasets - Reliable, Any Size

geese_chief/dataset-merge-dedup

Merge any number of Apify datasets into one and remove duplicates by the fields you choose (url, id, email...). Keep unique items or only the duplicates, pick output fields, tag each item with its source. Streams data, so millions of items work. Pick datasets from your storage: no permission errors.

Ponytail Data

Vendor security.txt Monitor

zentrafoundry/vendor-security-trust-center-monitor

Fetch public security.txt and trust-center pages. Try is Stripe’s file, not a pentest. $0.01 / vendor.

Zentra

Apify Smart Dataset Comparator

agenscrape/apify-smart-dataset-comparator

Compare 2-10 Apify datasets to detect changes, new/removed records, and duplicates. Features field-level diffs, smart merging, schema validation, data cleaning, and anomaly detection. Perfect for price monitoring, lead deduplication, and data quality tracking.

Agenscrape

CSV Doctor | Repair Broken CSV Files, With a Fix Report

broomwagon/csv-doctor

Repair the CSV that will not load: wrong delimiters, unquoted values, ragged rows, blank and duplicate headers, mojibake. Get back rectangular typed records plus a report of exactly what changed. Nothing is invented and nothing is dropped, so ragged rows are named, never truncated.

Brandon Mensing

Only New Items | Cross-Run Dedupe Ledger for Any Dataset

broomwagon/only-new-items

Run your scraper on a schedule and get back only what you have never seen before. Name the field that identifies an item, pipe any dataset in, and this actor holds back everything already delivered. Records pass through untouched. The ledger is yours, not a database you rent.

Brandon Mensing