Dedup & Merge Datasets: combine Apify datasets, JSON and CSV and drop duplicates

Scraped the same thing in several runs, or exported lists from different places? This Actor merges Apify datasets, JSON, JSON Lines and CSV files (by link) and pasted rows into one clean list, and removes duplicates using the fields you choose. It works with limited permissions, so you do not have to hand over full account access.

What you get

Field	What it tells you
(all your original fields)	Every column of the surviving row is kept exactly as it came.
_source	Where the row came from: inline, dataset:<id> or the file URL.

Example row:

{"id": 1, "name": "Ana ", "city": "Madrid", "_source": "inline"}

How to use it in 3 steps

1. Add your data: pick one or more Apify datasets, paste file links (JSON, JSON Lines, CSV, up to 20 files of 5 MB) or paste rows as a JSON list.
2. Choose the fields that define a duplicate (for example url or email), keep the first or last row, and decide whether to ignore case and extra spaces.
3. Run it and download the unique rows from the dataset as JSON, CSV or Excel. The run status shows how many rows came in and how many duplicates were removed.

Pricing

$1.50 per 1,000 unique rows (1,000 unique rows = $1.50). You pay only for rows that are kept; removed duplicates are free, and a run with no rows costs nothing. Default input: 4 rows in, 3 out.

Integrations

• Apify API: call the Actor from any language and read the dataset as JSON, CSV or Excel.
• Make, n8n and Zapier: use the official Apify modules, run this Actor and map the output rows.
• Google Sheets: export the dataset to Sheets from the run page, or schedule runs and append the rows.
• MCP (Claude, Cursor): add the Apify MCP server https://mcp.apify.com with actors=kadro/dedup-merge-datasets. Example prompt: "Merge these two Apify datasets and remove duplicates by the url field."

FAQ

How is a duplicate defined? Two rows are duplicates when all the fields you list match. If you list none, the whole row is compared (fields starting with _ are ignored).

Does it need full account permissions? No. It runs with limited permissions and only reads the datasets you select in the picker.

How big can the input be? Up to 200,000 rows per run, held in memory. Files: up to 20 links, 5 MB each.

Which files can it read? JSON (list of objects), JSON Lines and CSV/TSV with automatic delimiter detection. Excel is not read here; convert it first with the CSV, Excel & JSON Converter.

Is the data safe? Only public http(s) addresses are fetched (private and local addresses are blocked) and files are parsed as data, never executed.

Limitations

• Everything is processed in memory, so the limit is 200,000 rows per run.
• Matching is exact after trimming spaces and, optionally, lower-casing; there is no fuzzy matching.
• Nested objects are compared as JSON text.
• Files over 5 MB or on private networks are skipped with a warning in the log.

Other Kadro tools

• CSV, Excel & JSON Converter: get your files into CSV or JSON first.
• Bulk Email Validator: clean the emails you just merged.
• Bulk Phone Number Validator: validate phone columns in bulk.
• Website Tech Stack Detector: enrich merged site lists with their tech stack.

Found it useful? Please leave a review ⭐ — it helps a lot. Bugs or feature ideas: open an issue, I answer within 24 h.