Pricing

from $0.64 / 1,000 row processeds

Try for free

Go to Apify Store

Remove Duplicates: Deduplicate Datasets, CSV & Excel

Try for free

Remove duplicates from Apify datasets, pasted JSON, CSV, Excel and JSONL files or text lines. Deduplicate by chosen fields or whole rows, exact or fuzzy, with email, URL, phone and domain normalisers. Keep first, last or most complete, or merge. Get clean rows and a duplicates report.

Pricing

from $0.64 / 1,000 row processeds

Rating

0.0

(0)

Developer

Data Done Right

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

Categories

Developer tools

Automation

Integrations

Security

You can access the Remove Duplicates: Deduplicate Datasets, CSV & Excel programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

# Set API token
$API_TOKEN=<YOUR_API_TOKEN>

# Prepare Actor input
$cat > input.json << 'EOF'
<{
<  "items": [
<    {
<      "id": 1,
<      "name": "Anna Smith",
<      "email": "anna@acme.com",
<      "company": "Acme Inc",
<      "phone": "",
<      "city": "Austin"
<    },
<    {
<      "id": 2,
<      "name": "Ben Jones",
<      "email": "ben@globex.com",
<      "company": "Globex",
<      "phone": "+1 555 010 2000",
<      "city": "Boston"
<    },
<    {
<      "id": 3,
<      "name": "Carla Diaz",
<      "email": "carla@initech.com",
<      "company": "Initech",
<      "phone": "",
<      "city": "Denver"
<    },
<    {
<      "id": 4,
<      "name": "  ANNA   SMITH ",
<      "email": "Anna@Acme.com",
<      "company": "Acme Inc",
<      "phone": "+1 555 010 1000",
<      "city": ""
<    },
<    {
<      "id": 5,
<      "name": "Ben Jones",
<      "email": "ben@globex.com",
<      "company": "Globex",
<      "phone": "+1 555 010 2000",
<      "city": "Boston"
<    },
<    {
<      "id": 6,
<      "name": "Carla Diaz",
<      "email": "CARLA@initech.com ",
<      "company": "Initech",
<      "phone": "",
<      "city": "Denver"
<    },
<    {
<      "id": 7,
<      "name": "Ana Smith",
<      "email": "ana@acme.com",
<      "company": "Acme Inc",
<      "phone": "",
<      "city": "Austin"
<    },
<    {
<      "id": 8,
<      "name": "Ben Jonse",
<      "email": "ben@globex.com",
<      "company": "Globex",
<      "phone": "",
<      "city": "Boston"
<    }
<  ],
<  "keyFields": [
<    "name",
<    "email"
<  ],
<  "matchMode": "fuzzy",
<  "fuzzyThreshold": 90,
<  "fieldTypes": {
<    "email": "email"
<  },
<  "keepStrategy": "first",
<  "mergeDuplicates": true,
<  "addDuplicateCount": true,
<  "addClusterId": true,
<  "exports": [
<    "csv"
<  ]
<}
<EOF

# Run the Actor using an HTTP API
# See the full API reference at https://docs.apify.com/api/v2
$curl "https://api.apify.com/v2/actors/data-done-right~remove-duplicates/runs?token=$API_TOKEN" \
<  -X POST \
<  -d @input.json \
<  -H 'Content-Type: application/json'

Remove Duplicates: Deduplicate Datasets, CSV & Excel API

Below, you can find a list of relevant HTTP API endpoints for calling the Remove Duplicates: Deduplicate Datasets, CSV & Excel Actor. For this, you’ll need an Apify account. Replace <YOUR_API_TOKEN> in the URLs with your Apify API token, which you can find under API & Integrations in Apify Console. For details, see the API reference.

Run Actor

POST
https://api.apify.com/v2/actors/data-done-right~remove-duplicates/runs?token=<YOUR_API_TOKEN>

Note: By adding the method=POST query parameter, this API endpoint can be called using a GET request and thus used in third-party webhooks. Please refer to our Run Actor API documentation.

Run Actor synchronously and get dataset items

POST
https://api.apify.com/v2/actors/data-done-right~remove-duplicates/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>

Note: This endpoint supports both POST and GET request methods. However, only the POST method allows you to pass input data. For more information, please refer to our Run Actor synchronously and get dataset items API documentation.

Get Actor

GET
https://api.apify.com/v2/actors/data-done-right~remove-duplicates?token=<YOUR_API_TOKEN>

For more information, please refer to our Get Actor API documentation.

Actors can be used to scrape web pages, extract data, or automate browser tasks. Use the Remove Duplicates: Deduplicate Datasets, CSV & Excel API programmatically via the Apify API.

You can choose from:

Remove Duplicates: Deduplicate Datasets, CSV & Excel API in Python

Remove Duplicates: Deduplicate Datasets, CSV & Excel API in JavaScript

Remove Duplicates: Deduplicate Datasets, CSV & Excel API through CLI

Remove Duplicates: Deduplicate Datasets, CSV & Excel OpenAPI definition

You can start Remove Duplicates: Deduplicate Datasets, CSV & Excel with the Apify API by sending an HTTP POST request to the Run Actorendpoint. An Actor’s input and its content type can be passed as a payload of the POST request, and additional options can be specified using URL query parameters. The Remove Duplicates: Deduplicate Datasets, CSV & Excel is identified within the API by its ID, which is the creator’s username and the name of the Actor.

When the Remove Duplicates: Deduplicate Datasets, CSV & Excel run finishes you can list the data from its default dataset(storage) via the API or you can preview the data directly on Apify Console.

Dataset Merge & Dedup Datasets — Filter, Join CSV & Excel

saliga/dataset-merge-dedupe

Merge Apify datasets, CSV, JSON and Excel files or Google Sheets into one table. Clean fields, normalise URLs, extract domains, filter, remove duplicates and join (VLOOKUP) in one run. Works with limited permissions. CSV and Excel exports free. $0.50 per 1,000 output rows.

Salih Küçükkayış

Instagram LIST / REMOVE Followers

saswave/instagram-list-remove-followers

List all your followers for data analysis. Then provide a list of followers to be removed. Clean your account bots followers at scale. Help you to boost reach and engagements on your posts by removing bot accounts and not active followers

SASWAVE

205

5.0

(1)

Merge Datasets & Remove Duplicates — Deduplicate, Filter, Excel

tidydata/merge-dedupe-datasets

Merge datasets and remove duplicates in one call: combine Apify datasets, CSV, Excel, JSON or Google Sheets, dedupe by any field (smart URL/email/phone matching), filter, sort, keep only new rows, export CSV/Excel/JSON. Works from AI agents via Apify MCP (Claude, ChatGPT), n8n and the API.

tidydata

Monitoring Checker Duplication

apify/monitoring-checker-duplication

The Monitoring checker duplication is a part of the Apify Monitoring Suite (apify/monitoring). See its readme for more information and how to use this.

Apify

5.0

(5)

Dataset Delta — Added, Removed & Changed Records

orangepinlabs/dataset-delta

Compare two Apify datasets or track one dataset between runs. Emit only added, removed, and changed records with exact changed fields.

Orange Pin Labs

Fast Dataset Cleaner & CSV Formatter

motivational_nickel/dataset-cleaner-and-formatter

Fast dataset cleaning for CSV and JSON files. Automatically removes duplicates, trims whitespace, fixes capitalization, and normalizes fields. Works with Apify datasets or uploaded files and prepares data for analytics, CRM imports, and automation pipelines.

Leoncio Coronado Jr

Deduplicate, Merge & Transform Datasets

datacach/deduplicate-datasets

Merge multiple datasets, deduplicate items by a combination of fields, and apply custom transforms — powered by Polars.

DataCach

Data Deduplicator

parsebird/dataset-deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows while keeping the first or last occurrence. Supports case-insensitive matching and whitespace trimming.

ParseBird

Merge & Deduplicate Datasets: Only New Items, CSV/Excel Export

tinplatelabs/dataset-toolkit

Merge and deduplicate Apify datasets and CSV, JSON, Excel or Parquet files in one run. Remove duplicates (exact, normalized or fuzzy), output only new items since your last run, join / VLOOKUP a second list by key, filter, sort, then export to CSV, Excel, JSON or Parquet. $0.002 per 1,000 rows.

tinplatelabs

Google Autocomplete & Keyword Suggestions: 10 Search Engines

data-done-right/keyword-suggestions-tool

Google autocomplete scraper and keyword suggestions tool: Google search suggestions plus YouTube, Amazon keywords, Bing, DuckDuckGo, eBay, Yahoo and App Store autocomplete in one run. Alphabet and question expansion, recursive depth, cross-engine agreement score. No search volume data.