Pricing

from $0.02 / 1,000 item processeds

Try for free

Go to Apify Store

Dedupe and Merge Datasets: Only New Items

Try for free

Merge Apify datasets, runs and JSON or CSV files, remove duplicates by key fields with URL, email, phone and fuzzy matching, and output only items not seen in earlier runs. Runs after any scraper as an integration.

Pricing

from $0.02 / 1,000 item processeds

Rating

0.0

(0)

Developer

Plainsource

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

Categories

Developer tools

Integrations

You can access the Dedupe and Merge Datasets: Only New Items programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

# Set API token
$API_TOKEN=<YOUR_API_TOKEN>

# Prepare Actor input
$cat > input.json << 'EOF'
<{
<  "rawData": [
<    {
<      "company": "Northwind Traders",
<      "website": "https://www.northwind.example/?utm_source=directory",
<      "email": "Sales@Northwind.example",
<      "phone": "(415) 555-2671",
<      "city": "San Francisco"
<    },
<    {
<      "company": "Northwind Traders Inc.",
<      "website": "http://northwind.example/",
<      "email": "sales@northwind.example",
<      "phone": "+1 415 555 2671",
<      "employees": 120
<    },
<    {
<      "company": "Contoso Ltd",
<      "website": "https://contoso.example/about",
<      "phone": "212-555-0187",
<      "city": "New York"
<    },
<    {
<      "company": "Contoso",
<      "website": "https://www.contoso.example/about/#team",
<      "email": "hello@contoso.example",
<      "city": ""
<    },
<    {
<      "company": "Fabrikam",
<      "website": "https://fabrikam.example",
<      "email": "info@fabrikam.example",
<      "city": "Austin"
<    },
<    {
<      "company": "Tailspin Toys",
<      "website": "https://tailspin.example/shop?gclid=abc123",
<      "city": "Denver"
<    },
<    {
<      "company": "Tailspin Toys",
<      "website": "https://tailspin.example/shop",
<      "phone": "720 555 0144",
<      "city": "Denver"
<    }
<  ],
<  "dedupeFields": [
<    "website"
<  ],
<  "urlFields": [
<    "website"
<  ],
<  "keep": "merge"
<}
<EOF

# Run the Actor using an HTTP API
# See the full API reference at https://docs.apify.com/api/v2
$curl "https://api.apify.com/v2/actors/fernandosord~dataset-merge-dedupe/runs?token=$API_TOKEN" \
<  -X POST \
<  -d @input.json \
<  -H 'Content-Type: application/json'

Remove Duplicates from JSON and CSV Data API

Below, you can find a list of relevant HTTP API endpoints for calling the Dedupe and Merge Datasets: Only New Items Actor. For this, you’ll need an Apify account. Replace <YOUR_API_TOKEN> in the URLs with your Apify API token, which you can find under API & Integrations in Apify Console. For details, see the API reference.

Run Actor

POST
https://api.apify.com/v2/actors/fernandosord~dataset-merge-dedupe/runs?token=<YOUR_API_TOKEN>

Note: By adding the method=POST query parameter, this API endpoint can be called using a GET request and thus used in third-party webhooks. Please refer to our Run Actor API documentation.

Run Actor synchronously and get dataset items

POST
https://api.apify.com/v2/actors/fernandosord~dataset-merge-dedupe/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>

Note: This endpoint supports both POST and GET request methods. However, only the POST method allows you to pass input data. For more information, please refer to our Run Actor synchronously and get dataset items API documentation.

Get Actor

GET
https://api.apify.com/v2/actors/fernandosord~dataset-merge-dedupe?token=<YOUR_API_TOKEN>

For more information, please refer to our Get Actor API documentation.

Actors can be used to scrape web pages, extract data, or automate browser tasks. Use the Dedupe and Merge Datasets: Only New Items API programmatically via the Apify API.

You can choose from:

Dedupe and Merge Datasets: Only New Items API in Python

Dedupe and Merge Datasets: Only New Items API in JavaScript

Dedupe and Merge Datasets: Only New Items API through CLI

Dedupe and Merge Datasets: Only New Items OpenAPI definition

You can start Dedupe and Merge Datasets: Only New Items with the Apify API by sending an HTTP POST request to the Run Actorendpoint. An Actor’s input and its content type can be passed as a payload of the POST request, and additional options can be specified using URL query parameters. The Dedupe and Merge Datasets: Only New Items is identified within the API by its ID, which is the creator’s username and the name of the Actor.

When the Dedupe and Merge Datasets: Only New Items run finishes you can list the data from its default dataset(storage) via the API or you can preview the data directly on Apify Console.

Merge, Dedup & Transform Datasets

lukaskrivka/dedup-datasets

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

Lukáš Křivka

5.4K

5.0

(2)

Merge & Dedup Datasets: Remove Duplicates, Only New Items

charlesmarkette/merge-deduplicate-datasets-only-new-items

Merge any number of Apify datasets and remove duplicates by the fields you choose (or the whole item). Keep the first or last copy, or output only items not seen in earlier runs: ideal after a scheduled scraper. Streams millions of items. Duplicates are free; $0.0019 per 1,000 items output.

Charles Markette

Data Deduplicator

parsebird/dataset-deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows while keeping the first or last occurrence. Supports case-insensitive matching and whitespace trimming.

ParseBird

CSV & Excel Dataset Merger / Splitter [Bulk Optimizer]

thecfmarketing/csv-xlsx-dataset-merger-pattern-splitter

High-speed tool to merge multiple CSV & Excel files or split large datasets into smaller chunks by rows, file size, or specific patterns. Clean duplicates, optimize data formats, and structure columns. The ultimate utility for handling millions of B2B cold email leads and data lists.

sadique shaikh

Dataset Transformer - Filter, Dedupe, Merge & Export CSV

seemuapps/dataset-transformer

Filter, deduplicate, merge, sort, rename and flatten Apify datasets from any scraper, then export clean rows to a dataset, CSV or JSON file.

Seemu Scraping

Merge & Deduplicate Datasets - Reliable, Any Size

geese_chief/dataset-merge-dedup

Merge any number of Apify datasets into one and remove duplicates by the fields you choose (url, id, email...). Keep unique items or only the duplicates, pick output fields, tag each item with its source. Streams data, so millions of items work. Pick datasets from your storage: no permission errors.

Ponytail Data

Vendor security.txt Monitor

zentrafoundry/vendor-security-trust-center-monitor

Fetch public security.txt and trust-center pages. Try is Stripe’s file, not a pentest. $0.01 / vendor.

Zentra

Apify Smart Dataset Comparator

agenscrape/apify-smart-dataset-comparator

Compare 2-10 Apify datasets to detect changes, new/removed records, and duplicates. Features field-level diffs, smart merging, schema validation, data cleaning, and anomaly detection. Perfect for price monitoring, lead deduplication, and data quality tracking.

Agenscrape

CSV Doctor | Repair Broken CSV Files, With a Fix Report

broomwagon/csv-doctor

Repair the CSV that will not load: wrong delimiters, unquoted values, ragged rows, blank and duplicate headers, mojibake. Get back rectangular typed records plus a report of exactly what changed. Nothing is invented and nothing is dropped, so ragged rows are named, never truncated.

Brandon Mensing

Only New Items | Cross-Run Dedupe Ledger for Any Dataset

broomwagon/only-new-items

Run your scraper on a schedule and get back only what you have never seen before. Name the field that identifies an item, pipe any dataset in, and this actor holds back everything already delivered. Records pass through untouched. The ledger is yours, not a database you rent.

Brandon Mensing