Merge CSV Files & Datasets: Remove Duplicates, Excel, JSON

What is Merge CSV Files & Datasets?

Merge Apify datasets, CSV, JSON and Excel files into one table and remove duplicates, for $0.0215 per 1,000 rows, no login.

For anyone stitching scraper runs together. $0.02 per 1,000 unique rows written plus $0.0015 per 1,000 rows and $0.00005 per MB read; a broken source is named in the status, never a failed run. Your first run with the example input costs about $0.0084 (16 rows in our test run).

Sources	Apify datasets, CSV, TSV, JSON, JSON Lines, Excel links, Google Sheets links, pasted rows
Dedupe	By any fields or nested paths, or whole rows; keep first or last
Matching	URLs (scheme, www., trailing slash, utm/gclid/fbclid), emails, numbers, case and spaces
Size	About 5 million unique rows per run in the default 1 GB (2 million with keep last)
Output	Merged dataset plus a STATS record; JSON, CSV, Excel, API
Automate	Only-new mode, integrations after any scraper, schedules, MCP
Price	$0.0215 per 1,000 rows ($0.0165 per 1,000 rows on paid Apify plans), on full result pages of 1000 rows
Free to try	Runs on Apify's free plan: the $5 monthly credit covers about 230,000 rows
Login needed	No

Daily monitor: only what's new, every morning

Set it once, get only the new rows every morning. With Only rows new since the last run on, each run passes on only rows whose dedupe key no earlier run returned; set it as an integration after your scraper so it runs on each new dataset.

{
  "datasetIds": [
    "<your scraper run's dataset ID>"
  ],
  "dedupeFields": [
    "website"
  ],
  "onlyNewEntries": true,
  "stateKey": "daily-leads"
}

1. Paste this input into the Input tab (JSON view) and click Save as a new task; it appears under Saved tasks.
2. Open the task from Saved tasks and click Schedule.
3. Choose Daily, pick the hour, and save.

Cost example: a daily scraper run of 2,000 rows with about 300 new ≈ $0.43/month on the free plan ($0.38 on paid plans): one run a day for 30 days, start fee included. The first run returns everything that matches, so it costs more.

Why Merge CSV Files & Datasets users pick this one

• Never fails on bad input. Wrong IDs, empty files and dead links are reported, not fatal.
• Careful matching. URLs and emails compared as people mean them, nothing guessed.
• Big datasets. About 5 million unique rows per run in 1 GB.
• Pay for unique rows. $0.02 per 1,000 written; duplicates not charged as rows.
• Only new rows. Pass on just what earlier runs had not seen.

Get started in 3 steps

1. Pick the datasets to merge in 'Datasets', or add CSV, JSON or Excel links in 'File links', or paste rows.
2. List the fields that make a row unique in 'Dedupe by fields', e.g. website, or leave it empty to compare whole rows.
3. Click Start. Download the merged rows as JSON, CSV or Excel, and open the STATS record for the counts.

▶ Try it with the example input

The form opens with an example input already filled in: that run costs about $0.0084 on the free plan.

How much does it cost to merge and dedupe datasets?

Pay per event: you pay for each result page the run reads and each result it returns, plus a small start fee. Bronze, Silver, Gold and higher Apify plans pay less per result.

Rows	Free plan	Paid plans (Bronze, Silver, Gold)
100	$0.0085	$0.008
1,000	$0.0265	$0.0215
10,000	$0.22	$0.17

Totals include the $0.005 start fee per run and the $0.0015 rows-read-1000 fee for each result page of up to 1000 rows, rounded to the cent (amounts under $0.10 to four decimals). Apify's free plan includes $5 of credit a month, enough for about 230,000 rows.

Unique rows written cost $0.02 per 1,000 ($0.015 on paid plans). Reading costs $0.0015 per 1,000 rows (per page of up to 1,000 rows of a source) plus $0.00005 per MB of row data, so huge rows pay for their size; 'Keep last' reads everything twice and is billed twice for reading. Duplicates removed and rows an earlier run already returned are not charged as rows. Worked examples on the free plan, start fee included: the example input (2 small files, 16 rows written) = $0.0084; merging 10 scraper runs of 5,000 rows of about 2 KB each into 20,000 unique rows = $0.485; a daily only-new run over a 2,000-row dataset of 2 KB rows that passes on 300 new rows = $0.0142.

Billing events

Event	Free plan	Paid plans (Bronze, Silver, Gold)
row-result	$0.02 per 1,000 rows written	$0.015 per 1,000 rows written
rows-read-1000	$0.0015 per 1,000 rows read (or part of a page)	$0.0015 per 1,000 rows read (or part of a page)
data-read-mb	$0.00005 per MB of row data read	$0.00005 per MB of row data read
apify-actor-start	$0.005 per run	$0.005 per run

Benchmarks: how we compare

Test	Input	Result	Run time	Peak memory
First run (prefilled input)	2 sample files, dedupe by website	23 rows read, 16 written	3 s	45 MB
Big CSV, keep last	100,000-row CSV (12 MB), dedupe by website	48,695 unique rows	28 s	81 MB
Only new, nothing new	48,695-row dataset, second run	0 rows written	53 s	92 MB
Three datasets	3 run datasets by ID	1,081 of 1,124 rows	3 s	63 MB
Excel, 300,000 rows (6.6 MB)	dedupe by city	1 row written	109 s	91 MB

Measured on Apify; runs before build 0.3 at 512 MB, from build 0.3 at the default 1 GB.

What data you get

Merge CSV Files & Datasets merges rows from any number of Apify datasets, public CSV, TSV, JSON, JSON Lines and Excel files and pasted JSON rows into one dataset, and removes duplicates by the fields you choose (or by whole-row equality, ignoring timestamps). Matching is conservative: URLs match with or without http(s), www., one trailing slash or utm/gclid/fbclid tags; emails in any case; 5 and 5.0 as one number; everything else must be written the same way. It streams sources page by page, so about 5 million unique rows per run fit in the default 1 GB, and a source that cannot be read is named in the status instead of failing the run. A STATS record gives rows and megabytes read, rows written and removed per source.

• One output row per unique row, with all its fields (or only the 'Output fields' you list, nested values as their own columns)
• _source on each row: dataset:
• STATS record: rowsRead, rowsWritten, duplicatesRemoved, rowsWithEmptyDedupeFields, seenInEarlierRuns, unreadableRows, dedupeFields and per-source read/written/duplicates/error
• A status message with the same counts and every source that could not be read and why

name	website	city	category	rating	reviews	_source
Northwind Coffee Roasters	https://www.northwindcoffeeroasters.example/	Toronto	cafe	4.0		file:https://api.apify.com/v2/key-value-stores/twkCCtTWzs...
URBAN SPROUT GROCERS	http://urbansproutgrocers.example?utm_source=newsletter	Victoria			25	file:https://api.apify.com/v2/key-value-stores/twkCCtTWzs...

One full row record, exactly as a real run returned it (long text and lists shortened here):

{
  "name": "Northwind Coffee Roasters",
  "website": "https://www.northwindcoffeeroasters.example/",
  "city": "Toronto",
  "category": "cafe",
  "rating": "4.0",
  "_source": "file:https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv"
}

Use cases

• Combining the results of several scraper runs into one lead list without repeats
• Passing on only new rows from a daily scraper to Google Sheets, a CRM or Slack
• Cleaning a CSV or Excel export: one row per website, email or phone
• Joining files from different tools that spell the same URL or email differently

Use-case recipes

Only new leads after a daily scraper

Add this Actor as an integration after your scraper's task with datasetIds set to the run's default dataset, dedupe by website, and turn on 'Only rows new since the last run' with a memory name. Each day only leads you have not seen go on to Sheets or your CRM.

One clean list from several runs

Pick all the run datasets in 'Datasets', dedupe by website (or email), keep 'Last' so the newest copy of each business wins, and set 'Output fields' to the columns your CRM imports.

Clean a CSV export

Paste the CSV link in 'File links', dedupe by phone or email, and download the result as Excel; the STATS record tells you how many rows were removed.

Input

Field	Type	What it does	Default
fileUrls (File links)	array	Public links to CSV, TSV, JSON, JSON Lines or Excel (.xlsx) files, e.g. https://example.com/leads.csv: up to 200 MB each (Excel...
dedupeFields (Dedupe by fields)	array	Rows with the same values in these fields are duplicates, e.g. 'website', or 'name' plus 'city'.
rows (Rows)	array	Rows pasted as a JSON array of objects, e.g. [{"name": "Acme", "website": "acme.com"}].
smartMatch (Smart matching)	boolean	Compare values the way people mean them: URLs (with or without http(s)://) ignoring www., the host's case, one trailing slash...	true
ignoreCase (Ignore upper/lower case)	boolean	'ACME Inc' and 'acme inc' count as the same value.	true
keep (Which duplicate to keep)	string	Which row of each duplicate group stays.	"first"

All 14 fields, with examples: see the Input tab.

Use with AI agents (MCP)

MCP (Model Context Protocol) is the open standard that lets AI assistants such as Claude, ChatGPT and Cursor call outside tools. Apify's hosted MCP server at https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup gives your assistant this Actor only: it fills in the input from your request, runs it and reads the rows back, at the same per-result price as a run started by hand.

Example prompts

• "Merge these three Apify datasets and remove duplicates by website:
• "Dedupe this CSV by email and give me the unique rows: "
• "From today's scraper dataset, return only the rows we have not seen before"

Claude Desktop and claude.ai

Settings > Connectors > Add custom connector, name it Apify, paste https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup, keep the detected OAuth settings, select Add and sign in to Apify.

Claude Code

$claude mcp add --transport http apify "https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup"

Cursor and VS Code

Add this to ~/.cursor/mcp.json (Cursor); in VS Code's .vscode/mcp.json put the same entry under "servers" with "type": "http". Sign in to Apify when asked.

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup"
    }
  }
}

ChatGPT

Turn on Developer mode, open Settings > Apps & Connectors > Create, paste the URL above as the MCP server URL, choose OAuth and authorize Apify.

Clients without sign-in send Authorization: Bearer <YOUR_APIFY_TOKEN> instead. No Apify account? This Actor supports Apify's agentic payments: an agent can pay per run with a prepaid, spend-capped token, bought with x402 stablecoin payments.

n8n AI agent

Give an AI Agent node an MCP Client Tool with the endpoint https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup and a Bearer credential holding your Apify token.

Use it from AI agents (MCP), the API or no-code

Copy this to your AI assistant

Use the Apify Actor aurelix/dataset-merge-dedup (Merge CSV Files & Datasets) through Apify's MCP server or API. Main input: fileUrls (File links); dedupeFields (Dedupe by fields). Example input: {"fileUrls": ["https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv", "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json"], "dedupeFields": ["website"]}. Each result is one row with fields such as name, website, city, category, rating, _source. It costs $0.0215 per 1,000 rows on Apify's free plan on full result pages ($0.00002 per row plus $0.0015 per result page of 1000) plus a $0.005 start fee per run, and needs no login.

MCP server URL for this Actor: https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup (setup for each client in the section above).

API

Start a run and read the dataset with Apify's REST API or the Python and JavaScript clients:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("aurelix/dataset-merge-dedup").call(run_input={'fileUrls': ['https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv', 'https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json'], 'dedupeFields': ['website']})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

No-code: schedule it in 3 clicks

Send the results to Google Sheets, Zapier, Make, n8n, Slack or any webhook URL.

1. Fill in the input, then click Save as a new task.
2. In the task, click Schedule and pick daily or weekly (Apify Schedules).
3. Turn on Only rows new since the last run so each run returns only what is new.

Prefer n8n or Make? Ask in the Issues tab and we will share a ready-made workflow for it.

FAQ

Can I merge Apify datasets without login?

Yes. It never logs in, so you need no extra account, password or cookies; only a free Apify account to run it.

Is there an API to merge and dedupe datasets?

You can use this Actor as one. Start it with Apify's REST API or the Python and JavaScript clients and read the results as JSON (see the code above), or call it from an AI agent through MCP.

How much does it cost to merge and dedupe datasets?

About $0.0265 for 1,000 rows on Apify's free plan and $0.0215 on paid plans, start fee and page fees included. The free $5 monthly credit covers about 230,000 rows.

Can I export merged, deduplicated data to Excel or Google Sheets?

Yes. Download the results as Excel, CSV, JSON, XML or HTML from the run's Storage tab, or send them to Google Sheets with Apify's Google Sheets integration or the API.

Is it legal to merge and dedupe datasets?

It reads only public data any logged-out visitor can see: public file links, plus the datasets, files and rows you give it, and it changes nothing at the source. You are responsible for the data you load and how you use it. This is not legal advice; check the rules that apply to your use.

Does it work with limited-permission runs?

Yes. This Actor reads datasets given through Apify's dataset picker, which limited-permission Actors need, and names a source it cannot read in the status instead of failing.

Can I send the merged rows to Google Sheets, n8n or an AI agent?

Yes. Use Apify's Google Sheets integration on the run, the n8n Apify node, or call it from an AI agent through Apify's MCP server; pass rows directly in 'Rows' from an agent.

How are duplicates found?

Each row's values in 'Dedupe by fields' are normalised and compared. Always: spaces trimmed and, with 'Ignore upper/lower case' on, case ignored. With Smart matching on, also: URLs match with or without http(s)://, www., the host's case, one trailing slash and utm_*, gclid and fbclid tags (other parameters, the path's case and #fragments still count); emails match in any case; 5 and 5.0 are one number. Signs, decimals, exponents and leading zeros are kept, so -12 and 12, 1.5 and 15 or 02111 and 2111 stay different. Phone numbers, dates and IDs must be written the same way to match. With no fields listed, whole rows are compared, ignoring timestamp fields such as scraped_at.

What does a run cost in total?

Rows written cost $0.02 per 1,000. Reading costs $0.0015 per 1,000 rows plus $0.00005 per MB of row data, plus a $0.005 start fee per run. The table above assumes every row is read once and written once and ignores the per-MB fee, which adds about $0.0001 per 1,000 rows of 2 KB; rows of 50 KB add about $0.0025 per 1,000.

What happens to rows without the dedupe fields?

By default they are kept and never merged with rows that have a value. With 'Only rows new since the last run' on, such a row is remembered by its whole content, so it is not sent (or charged) again. Choose 'Drop them' to remove them. The STATS record counts them either way. Empty rows (null or {}) are skipped and counted.

Why does it say it may not read my dataset?

Apify lets this Actor read only the datasets you give it in the 'Datasets' field. Pick the dataset there (or paste its ID there, not in 'File links'), and Apify grants read access for that run.

Can it merge datasets from another Apify account?

Only if the dataset is readable by anyone with its link; put that link in 'File links'. Datasets of your own account go in 'Datasets'.

How big can the input be?

Sources are read 1,000 rows at a time and only an 8-byte fingerprint of each unique key is kept, so about 5 million unique rows per run fit in the default 1 GB (about 2 million with 'Which duplicate to keep: Last'), with at least 20% of the memory left free. At the limit the run stops cleanly, keeps what it wrote and says so; give the run more memory to go higher. Files up to 200 MB each; Excel up to 30 MB with up to 40 MB of text in its shared-string table (about 400,000 rows of short text).

Limits (stated honestly)

• No JavaScript transform functions; use 'Output fields' to choose and flatten columns.
• About 5 million unique rows per run at the default 1 GB (2 million with 'Last'); the run stops cleanly at that point and says so.
• Excel files up to 30 MB with up to 40 MB of text in the shared-string table (streamed); CSV, JSON Lines and JSON lists up to 200 MB (streamed); other JSON shapes up to 20 MB.
• Google Sheets links work only for sheets shared 'anyone with the link'.
• Only-new memory keeps the newest 5 million keys (8 bytes each), saved after every batch of written rows.
• No phone-number matching: write phones the same way (or dedupe by another field).
• 'Which duplicate to keep: Last' reads every source twice and bills reading twice.
• File links must download at 2.5 MB/s or faster (after the first 10 s); a slower host is stopped and named in the status.

Works great with

• Before this: Google Maps Phone Scraper: pick the datasets of several runs (different cities or keywords) to get one list without repeated businesses
• Before this: LinkedIn Jobs Country Filter Scraper: merge daily runs and dedupe by job_url to keep one row per job

Related Aurelix tools

Public Aurelix Actors, grouped by the data they collect. All are pay-per-result and need no login.

More data tools

• Agoda API (Unofficial): Scrape Agoda guest reviews at $0.50 per 1,000 with this unofficial Agoda API, cheaper than the leading tools: Agoda and...
• Tennis Betting Odds Scraper: Tennis results and upcoming matches with opening and closing odds per bookmaker and implied probabilities.

Jobs and hiring data

• Job Bank Canada Scraper API: Official Job Bank Canada postings by month: NOC and NAICS codes, wages, hours and location.
• Greenhouse Jobs Scraper: Live jobs from a company's Greenhouse, Lever, Ashby or SmartRecruiters board, with salary and Canadian province tags.
• Career Page Monitor: New, removed and reposted roles plus hiring velocity per company, from seven job-board systems including Workday.
• LinkedIn Jobs Country Filter Scraper: Public LinkedIn job postings by keyword and country: title, company, location, seniority, job type, salary text, description.
• Multiple Job Boards Scraper: LinkedIn, about 50 company career sites, Remote OK and We Work Remotely jobs in one deduplicated list.
• Remote Job Board Scraper API: Remote OK and We Work Remotely jobs in one deduplicated list with salary, regions and tags.

E-commerce and product data

• AliExpress Scraper API: AliExpress search results by keyword: price, discount, rating, units sold, ship-from country and sponsored flag.
• Amazon Best Sellers & Amazon New Releases: Amazon Best Sellers and New Releases top 100 on 10 marketplaces: rank, ASIN, price, rating, rank change.
• Best Buy Canada Scraper API: Best Buy Canada search results: CAD price, sale savings, rating, reviews and seller, plus optional UPC and specs.
• Shopify Product Export Scraper: Every public product of any Shopify store with variants, SKUs, prices, barcodes, stock counts and collections.
• Shopify Best Sellers Tracker: Competitor Shopify stores' best-sellers, new products, price changes, stock-outs and restocks since your last run, plus catalogue summary.

Leads and company data

• Companies House Scraper: Newly incorporated UK companies from Companies House: SIC codes, region, registered office, directors with roles.
• Contact Page Scraper: Published company emails, phones, addresses, social pages and team members from a list of company websites.
• Decision Maker Finder: Founders, CEOs, VPs and department heads from company team pages, with title, seniority and function.
• Newly Funded Companies Hiring: Companies that just raised money (SEC Form D), with their open roles and business contacts, as a new-only monitor.
• Google Maps Phone Scraper: Local business lists from Google Maps: phones, websites, ratings, hours, coordinates; optional website emails and socials.
• Recently Funded Companies API: US companies that just filed an SEC Form D: amount raised, industry, state, executives, confirmed website.
• Tech Stack API: CMS, e-commerce platform, analytics, ad pixels, chat, CDN and email provider for each domain you enter.
• Yellow Pages Canada Scraper: YellowPages.ca businesses by search term and city: phone, website, address, postal code, hours and rating.

Ad intelligence

• Competitor Watch: One report per competitor with only what is new: Meta and TikTok ads, page changes, Shopify products and prices, jobs.
• Facebook Ad Library Search & API: Facebook and Instagram business ads from the Meta Ad Library: copy, creatives, landing pages, dates, optional EU reach.
• TikTok Ads Library API: TikTok Ads Library ads for Europe, UK, Switzerland and Turkey with targeting, reach, ad text and landing pages.

Canada and government data

• Canada Building Permits: Building permits and development applications from 7 Canadian cities: address, type, status, dates, value, applicant, contractor and Toronto planner contact.
• Government Tenders Scraper: Open tenders from SAM.gov, CanadaBuys and EU TED in one table: buyer, codes, value, deadline, contact and link.
• USPTO Trademark API: US trademark search and watch from USPTO: mark, owner or number lookups, new filings, owner, attorney, goods.

AI, content and SEO

• Google Trends by Region: Google Trends interest over time, by region and rising queries for any number of keywords, plus trending searches.
• Google Autocomplete API: Expands seed keywords into hundreds of real Google and YouTube autocomplete phrases, tagged by search intent.
• PDF Table Extraction & PDF to Markdown OCR: Turns linked PDF and Word files into Markdown per page, with tables as JSON rows, RAG chunks and OCR.
• Broken Link Crawler & SEO Audit: Audits a whole site: one row per page with SEO checks and broken links, plus a ranked issues summary.
• Website to Markdown API: Crawls a site and returns one row of clean main-content Markdown per page, with title, headings and links.
• YouTube Transcript Scraper: Returns YouTube video transcripts as text, timed segments and AI chunks, from videos, channels, playlists or searches.

Recent changes

• 0.3: 1 GB default memory; conservative URL and number matching (fragments and other parameters kept, no phone guessing); per-MB read fee; only-new memory saved after every batch; Excel capped by its text table; keep last recovers when a source breaks in the second pass.
• 0.2: Stricter smart matching (signs, decimals and leading zeros kept; schemeless URLs normalised); only-new also remembers rows without dedupe values; keep last stays correct when a source breaks; compact only-new memory; Excel streamed; bare dataset names accepted.
• 0.1: First release: datasets, CSV, TSV, JSON, JSON Lines, Excel and Google Sheets links, smart matching, keep first or last, only-new mode, STATS record.

The full history is on the Changelog tab.

Tested daily

An automated health check runs this Actor every day on a fixed input and checks that at least 16 rows come back with name, website, _source filled in; a failed check alerts the maintainer straight away.

Your feedback

Missing a field, found a bug or need another filter? Open an issue on the Issues tab; we aim to answer within a day. If this Actor saved you time, a short review helps other people find it.