# Merge CSV Files & Datasets: Remove Duplicates, Excel, JSON (`aurelix/dataset-merge-dedup`) Actor Extract one clean table from many CSV files, Excel files, JSON files or Apify datasets at $0.02 per 1,000 unique rows: merge, remove duplicates by any fields with URL, email and number matching, keep first or last, only-new mode. A bad source never fails the run. - **URL**: https://apify.com/aurelix/dataset-merge-dedup.md - **Developed by:** [Aurelix](https://apify.com/aurelix) (community) - **Categories:** Developer tools, Automation, Lead generation - **Features:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.02 / 1,000 row results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What is Merge CSV Files & Datasets? **Merge Apify datasets, CSV, JSON and Excel files into one table and remove duplicates, for $0.0215 per 1,000 rows, no login.** For anyone stitching scraper runs together. $0.02 per 1,000 unique rows written plus $0.0015 per 1,000 rows and $0.00005 per MB read; a broken source is named in the status, never a failed run. Your first run with the example input costs about $0.0084 (16 rows in our test run). [![Merge CSV Files & Datasets: example output from a real run](https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/dataset-merge-dedup-card-v1.png)](https://apify.com/aurelix/dataset-merge-dedup) | | | |---|---| | **Sources** | Apify datasets, CSV, TSV, JSON, JSON Lines, Excel links, Google Sheets links, pasted rows | | **Dedupe** | By any fields or nested paths, or whole rows; keep first or last | | **Matching** | URLs (scheme, www., trailing slash, utm/gclid/fbclid), emails, numbers, case and spaces | | **Size** | About 5 million unique rows per run in the default 1 GB (2 million with keep last) | | **Output** | Merged dataset plus a STATS record; JSON, CSV, Excel, API | | **Automate** | Only-new mode, integrations after any scraper, schedules, MCP | | **Price** | $0.0215 per 1,000 rows ($0.0165 per 1,000 rows on paid Apify plans), on full result pages of 1000 rows | | **Free to try** | Runs on Apify's free plan: the $5 monthly credit covers about 230,000 rows | | **Login needed** | No | ### Daily monitor: only what's new, every morning Set it once, get only the new rows every morning. With **Only rows new since the last run** on, each run passes on only rows whose dedupe key no earlier run returned; set it as an integration after your scraper so it runs on each new dataset. ```json { "datasetIds": [ "" ], "dedupeFields": [ "website" ], "onlyNewEntries": true, "stateKey": "daily-leads" } ``` 1. Paste this input into the **Input** tab (JSON view) and click **Save as a new task**; it appears under **Saved tasks**. 2. Open the task from **Saved tasks** and click **Schedule**. 3. Choose **Daily**, pick the hour, and save. **Cost example:** a daily scraper run of 2,000 rows with about 300 new ≈ $0.43/month on the free plan ($0.38 on paid plans): one run a day for 30 days, start fee included. The first run returns everything that matches, so it costs more. ### Why Merge CSV Files & Datasets users pick this one - **Never fails on bad input.** Wrong IDs, empty files and dead links are reported, not fatal. - **Careful matching.** URLs and emails compared as people mean them, nothing guessed. - **Big datasets.** About 5 million unique rows per run in 1 GB. - **Pay for unique rows.** $0.02 per 1,000 written; duplicates not charged as rows. - **Only new rows.** Pass on just what earlier runs had not seen. ### Get started in 3 steps 1. Pick the datasets to merge in 'Datasets', or add CSV, JSON or Excel links in 'File links', or paste rows. 2. List the fields that make a row unique in 'Dedupe by fields', e.g. website, or leave it empty to compare whole rows. 3. Click Start. Download the merged rows as JSON, CSV or Excel, and open the STATS record for the counts. **[▶ Try it with the example input](https://apify.com/aurelix/dataset-merge-dedup)** The form opens with an example input already filled in: that run costs about $0.0084 on the free plan. ### How much does it cost to merge and dedupe datasets? Pay per event: you pay for each result page the run reads and each result it returns, plus a small start fee. Bronze, Silver, Gold and higher Apify plans pay less per result. | Rows | Free plan | Paid plans (Bronze, Silver, Gold) | |---|---|---| | 100 | $0.0085 | $0.008 | | 1,000 | $0.0265 | $0.0215 | | 10,000 | $0.22 | $0.17 | Totals include the $0.005 start fee per run and the $0.0015 `rows-read-1000` fee for each result page of up to 1000 rows, rounded to the cent (amounts under $0.10 to four decimals). Apify's free plan includes $5 of credit a month, enough for about 230,000 rows. Unique rows written cost $0.02 per 1,000 ($0.015 on paid plans). Reading costs $0.0015 per 1,000 rows (per page of up to 1,000 rows of a source) plus $0.00005 per MB of row data, so huge rows pay for their size; 'Keep last' reads everything twice and is billed twice for reading. Duplicates removed and rows an earlier run already returned are not charged as rows. Worked examples on the free plan, start fee included: the example input (2 small files, 16 rows written) = $0.0084; merging 10 scraper runs of 5,000 rows of about 2 KB each into 20,000 unique rows = $0.485; a daily only-new run over a 2,000-row dataset of 2 KB rows that passes on 300 new rows = $0.0142. #### Billing events | Event | Free plan | Paid plans (Bronze, Silver, Gold) | |---|---|---| | `row-result` | $0.02 per 1,000 rows written | $0.015 per 1,000 rows written | | `rows-read-1000` | $0.0015 per 1,000 rows read (or part of a page) | $0.0015 per 1,000 rows read (or part of a page) | | `data-read-mb` | $0.00005 per MB of row data read | $0.00005 per MB of row data read | | `apify-actor-start` | $0.005 per run | $0.005 per run | ### Benchmarks: how we compare | Test | Input | Result | Run time | Peak memory | |---|---|---|---|---| | First run (prefilled input) | 2 sample files, dedupe by website | 23 rows read, 16 written | 3 s | 45 MB | | Big CSV, keep last | 100,000-row CSV (12 MB), dedupe by website | 48,695 unique rows | 28 s | 81 MB | | Only new, nothing new | 48,695-row dataset, second run | 0 rows written | 53 s | 92 MB | | Three datasets | 3 run datasets by ID | 1,081 of 1,124 rows | 3 s | 63 MB | | Excel, 300,000 rows (6.6 MB) | dedupe by city | 1 row written | 109 s | 91 MB | Measured on Apify; runs before build 0.3 at 512 MB, from build 0.3 at the default 1 GB. ### What data you get **Merge CSV Files & Datasets** merges rows from any number of Apify datasets, public CSV, TSV, JSON, JSON Lines and Excel files and pasted JSON rows into one dataset, and removes duplicates by the fields you choose (or by whole-row equality, ignoring timestamps). Matching is conservative: URLs match with or without http(s), www., one trailing slash or utm/gclid/fbclid tags; emails in any case; 5 and 5.0 as one number; everything else must be written the same way. It streams sources page by page, so about 5 million unique rows per run fit in the default 1 GB, and a source that cannot be read is named in the status instead of failing the run. A STATS record gives rows and megabytes read, rows written and removed per source. - One output row per unique row, with all its fields (or only the 'Output fields' you list, nested values as their own columns) - \_source on each row: dataset:, file: or rows - STATS record: rowsRead, rowsWritten, duplicatesRemoved, rowsWithEmptyDedupeFields, seenInEarlierRuns, unreadableRows, dedupeFields and per-source read/written/duplicates/error - A status message with the same counts and every source that could not be read and why | name | website | city | category | rating | reviews | \_source | |---|---|---|---|---|---|---| | Northwind Coffee Roasters | https://www.northwindcoffeeroasters.example/ | Toronto | cafe | 4.0 | | file:https://api.apify.com/v2/key-value-stores/twkCCtTWzs... | | URBAN SPROUT GROCERS | http://urbansproutgrocers.example?utm_source=newsletter | Victoria | | | 25 | file:https://api.apify.com/v2/key-value-stores/twkCCtTWzs... | One full row record, exactly as a real run returned it (long text and lists shortened here): ```json { "name": "Northwind Coffee Roasters", "website": "https://www.northwindcoffeeroasters.example/", "city": "Toronto", "category": "cafe", "rating": "4.0", "_source": "file:https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv" } ``` ### Use cases - Combining the results of several scraper runs into one lead list without repeats - Passing on only new rows from a daily scraper to Google Sheets, a CRM or Slack - Cleaning a CSV or Excel export: one row per website, email or phone - Joining files from different tools that spell the same URL or email differently ### Use-case recipes #### Only new leads after a daily scraper Add this Actor as an integration after your scraper's task with datasetIds set to the run's default dataset, dedupe by website, and turn on 'Only rows new since the last run' with a memory name. Each day only leads you have not seen go on to Sheets or your CRM. #### One clean list from several runs Pick all the run datasets in 'Datasets', dedupe by website (or email), keep 'Last' so the newest copy of each business wins, and set 'Output fields' to the columns your CRM imports. #### Clean a CSV export Paste the CSV link in 'File links', dedupe by phone or email, and download the result as Excel; the STATS record tells you how many rows were removed. ### Input | Field | Type | What it does | Default | |---|---|---|---| | `fileUrls` (File links) | array | Public links to CSV, TSV, JSON, JSON Lines or Excel (.xlsx) files, e.g. https://example.com/leads.csv: up to 200 MB each (Excel... | | | `dedupeFields` (Dedupe by fields) | array | Rows with the same values in these fields are duplicates, e.g. 'website', or 'name' plus 'city'. | | | `rows` (Rows) | array | Rows pasted as a JSON array of objects, e.g. \[{"name": "Acme", "website": "acme.com"}]. | | | `smartMatch` (Smart matching) | boolean | Compare values the way people mean them: URLs (with or without http(s)://) ignoring www., the host's case, one trailing slash... | true | | `ignoreCase` (Ignore upper/lower case) | boolean | 'ACME Inc' and 'acme inc' count as the same value. | true | | `keep` (Which duplicate to keep) | string | Which row of each duplicate group stays. | "first" | All 14 fields, with examples: see the **Input** tab. ### Use with AI agents (MCP) MCP (Model Context Protocol) is the open standard that lets AI assistants such as Claude, ChatGPT and Cursor call outside tools. Apify's hosted MCP server at `https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup` gives your assistant this Actor only: it fills in the input from your request, runs it and reads the rows back, at the same per-result price as a run started by hand. #### Example prompts - "Merge these three Apify datasets and remove duplicates by website: " - "Dedupe this CSV by email and give me the unique rows: " - "From today's scraper dataset, return only the rows we have not seen before" #### Claude Desktop and claude.ai **Settings > Connectors > Add custom connector**, name it Apify, paste `https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup`, keep the detected OAuth settings, select **Add** and sign in to Apify. #### Claude Code ```bash claude mcp add --transport http apify "https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup" ``` #### Cursor and VS Code Add this to `~/.cursor/mcp.json` (Cursor); in VS Code's `.vscode/mcp.json` put the same entry under `"servers"` with `"type": "http"`. Sign in to Apify when asked. ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup" } } } ``` #### ChatGPT Turn on **Developer mode**, open **Settings > Apps & Connectors > Create**, paste the URL above as the MCP server URL, choose **OAuth** and authorize Apify. Clients without sign-in send `Authorization: Bearer ` instead. No Apify account? This Actor supports Apify's agentic payments: an agent can pay per run with a prepaid, spend-capped token, bought with x402 stablecoin payments. #### n8n AI agent Give an **AI Agent** node an **MCP Client Tool** with the endpoint `https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup` and a Bearer credential holding your Apify token. ### Use it from AI agents (MCP), the API or no-code #### Copy this to your AI assistant ```text Use the Apify Actor aurelix/dataset-merge-dedup (Merge CSV Files & Datasets) through Apify's MCP server or API. Main input: fileUrls (File links); dedupeFields (Dedupe by fields). Example input: {"fileUrls": ["https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv", "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json"], "dedupeFields": ["website"]}. Each result is one row with fields such as name, website, city, category, rating, _source. It costs $0.0215 per 1,000 rows on Apify's free plan on full result pages ($0.00002 per row plus $0.0015 per result page of 1000) plus a $0.005 start fee per run, and needs no login. ``` MCP server URL for this Actor: `https://mcp.apify.com/?tools=aurelix/dataset-merge-dedup` (setup for each client in the section above). #### API Start a run and read the dataset with Apify's REST API or the Python and JavaScript clients: ```python from apify_client import ApifyClient client = ApifyClient("") run = client.actor("aurelix/dataset-merge-dedup").call(run_input={'fileUrls': ['https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv', 'https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json'], 'dedupeFields': ['website']}) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` #### No-code: schedule it in 3 clicks Send the results to Google Sheets, Zapier, Make, n8n, Slack or any webhook URL. 1. Fill in the input, then click **Save as a new task**. 2. In the task, click **Schedule** and pick daily or weekly (Apify Schedules). 3. Turn on **Only rows new since the last run** so each run returns only what is new. Prefer n8n or Make? Ask in the **Issues** tab and we will share a ready-made workflow for it. ### FAQ #### Can I merge Apify datasets without login? Yes. It never logs in, so you need no extra account, password or cookies; only a free Apify account to run it. #### Is there an API to merge and dedupe datasets? You can use this Actor as one. Start it with Apify's REST API or the Python and JavaScript clients and read the results as JSON (see the code above), or call it from an AI agent through MCP. #### How much does it cost to merge and dedupe datasets? About $0.0265 for 1,000 rows on Apify's free plan and $0.0215 on paid plans, start fee and page fees included. The free $5 monthly credit covers about 230,000 rows. #### Can I export merged, deduplicated data to Excel or Google Sheets? Yes. Download the results as Excel, CSV, JSON, XML or HTML from the run's **Storage** tab, or send them to Google Sheets with Apify's Google Sheets integration or the API. #### Is it legal to merge and dedupe datasets? It reads only public data any logged-out visitor can see: public file links, plus the datasets, files and rows you give it, and it changes nothing at the source. You are responsible for the data you load and how you use it. This is not legal advice; check the rules that apply to your use. #### Does it work with limited-permission runs? Yes. This Actor reads datasets given through Apify's dataset picker, which limited-permission Actors need, and names a source it cannot read in the status instead of failing. #### Can I send the merged rows to Google Sheets, n8n or an AI agent? Yes. Use Apify's Google Sheets integration on the run, the n8n Apify node, or call it from an AI agent through Apify's MCP server; pass rows directly in 'Rows' from an agent. #### How are duplicates found? Each row's values in 'Dedupe by fields' are normalised and compared. Always: spaces trimmed and, with 'Ignore upper/lower case' on, case ignored. With Smart matching on, also: URLs match with or without http(s)://, www., the host's case, one trailing slash and utm\_\*, gclid and fbclid tags (other parameters, the path's case and #fragments still count); emails match in any case; 5 and 5.0 are one number. Signs, decimals, exponents and leading zeros are kept, so -12 and 12, 1.5 and 15 or 02111 and 2111 stay different. Phone numbers, dates and IDs must be written the same way to match. With no fields listed, whole rows are compared, ignoring timestamp fields such as scraped_at. #### What does a run cost in total? Rows written cost $0.02 per 1,000. Reading costs $0.0015 per 1,000 rows plus $0.00005 per MB of row data, plus a $0.005 start fee per run. The table above assumes every row is read once and written once and ignores the per-MB fee, which adds about $0.0001 per 1,000 rows of 2 KB; rows of 50 KB add about $0.0025 per 1,000. #### What happens to rows without the dedupe fields? By default they are kept and never merged with rows that have a value. With 'Only rows new since the last run' on, such a row is remembered by its whole content, so it is not sent (or charged) again. Choose 'Drop them' to remove them. The STATS record counts them either way. Empty rows (null or {}) are skipped and counted. #### Why does it say it may not read my dataset? Apify lets this Actor read only the datasets you give it in the 'Datasets' field. Pick the dataset there (or paste its ID there, not in 'File links'), and Apify grants read access for that run. #### Can it merge datasets from another Apify account? Only if the dataset is readable by anyone with its link; put that link in 'File links'. Datasets of your own account go in 'Datasets'. #### How big can the input be? Sources are read 1,000 rows at a time and only an 8-byte fingerprint of each unique key is kept, so about 5 million unique rows per run fit in the default 1 GB (about 2 million with 'Which duplicate to keep: Last'), with at least 20% of the memory left free. At the limit the run stops cleanly, keeps what it wrote and says so; give the run more memory to go higher. Files up to 200 MB each; Excel up to 30 MB with up to 40 MB of text in its shared-string table (about 400,000 rows of short text). ### Limits (stated honestly) - No JavaScript transform functions; use 'Output fields' to choose and flatten columns. - About 5 million unique rows per run at the default 1 GB (2 million with 'Last'); the run stops cleanly at that point and says so. - Excel files up to 30 MB with up to 40 MB of text in the shared-string table (streamed); CSV, JSON Lines and JSON lists up to 200 MB (streamed); other JSON shapes up to 20 MB. - Google Sheets links work only for sheets shared 'anyone with the link'. - Only-new memory keeps the newest 5 million keys (8 bytes each), saved after every batch of written rows. - No phone-number matching: write phones the same way (or dedupe by another field). - 'Which duplicate to keep: Last' reads every source twice and bills reading twice. - File links must download at 2.5 MB/s or faster (after the first 10 s); a slower host is stopped and named in the status. ### Works great with - **Before this: [Google Maps Phone Scraper](https://apify.com/aurelix/google-maps-scraper)**: pick the datasets of several runs (different cities or keywords) to get one list without repeated businesses - **Before this: [LinkedIn Jobs Country Filter Scraper](https://apify.com/aurelix/linkedin-jobs)**: merge daily runs and dedupe by `job_url` to keep one row per job ### Related Aurelix tools Public Aurelix Actors, grouped by the data they collect. All are pay-per-result and need no login. #### More data tools - **[Agoda API (Unofficial)](https://apify.com/aurelix/agoda-reviews-scraper)**: Scrape Agoda guest reviews at $0.50 per 1,000 with this unofficial Agoda API, cheaper than the leading tools: Agoda and... - **[Tennis Betting Odds Scraper](https://apify.com/aurelix/tennis-odds-scraper)**: Tennis results and upcoming matches with opening and closing odds per bookmaker and implied probabilities. #### Jobs and hiring data - **[Job Bank Canada Scraper API](https://apify.com/aurelix/job-bank-canada-scraper)**: Official Job Bank Canada postings by month: NOC and NAICS codes, wages, hours and location. - **[Greenhouse Jobs Scraper](https://apify.com/aurelix/greenhouse-jobs-scraper)**: Live jobs from a company's Greenhouse, Lever, Ashby or SmartRecruiters board, with salary and Canadian province tags. - **[Career Page Monitor](https://apify.com/aurelix/hiring-signal-monitor)**: New, removed and reposted roles plus hiring velocity per company, from seven job-board systems including Workday. - **[LinkedIn Jobs Country Filter Scraper](https://apify.com/aurelix/linkedin-jobs)**: Public LinkedIn job postings by keyword and country: title, company, location, seniority, job type, salary text, description. - **[Multiple Job Boards Scraper](https://apify.com/aurelix/multi-job-board-scraper)**: LinkedIn, about 50 company career sites, Remote OK and We Work Remotely jobs in one deduplicated list. - **[Remote Job Board Scraper API](https://apify.com/aurelix/remote-jobs-scraper)**: Remote OK and We Work Remotely jobs in one deduplicated list with salary, regions and tags. #### E-commerce and product data - **[AliExpress Scraper API](https://apify.com/aurelix/aliexpress-product-search)**: AliExpress search results by keyword: price, discount, rating, units sold, ship-from country and sponsored flag. - **[Amazon Best Sellers & Amazon New Releases](https://apify.com/aurelix/amazon-best-sellers)**: Amazon Best Sellers and New Releases top 100 on 10 marketplaces: rank, ASIN, price, rating, rank change. - **[Best Buy Canada Scraper API](https://apify.com/aurelix/best-buy-canada-scraper)**: Best Buy Canada search results: CAD price, sale savings, rating, reviews and seller, plus optional UPC and specs. - **[Shopify Product Export Scraper](https://apify.com/aurelix/shopify-products-scraper)**: Every public product of any Shopify store with variants, SKUs, prices, barcodes, stock counts and collections. - **[Shopify Best Sellers Tracker](https://apify.com/aurelix/shopify-best-sellers-tracker)**: Competitor Shopify stores' best-sellers, new products, price changes, stock-outs and restocks since your last run, plus catalogue summary. #### Leads and company data - **[Companies House Scraper](https://apify.com/aurelix/companies-house-scraper)**: Newly incorporated UK companies from Companies House: SIC codes, region, registered office, directors with roles. - **[Contact Page Scraper](https://apify.com/aurelix/contact-details-scraper)**: Published company emails, phones, addresses, social pages and team members from a list of company websites. - **[Decision Maker Finder](https://apify.com/aurelix/decision-maker-finder)**: Founders, CEOs, VPs and department heads from company team pages, with title, seniority and function. - **[Newly Funded Companies Hiring](https://apify.com/aurelix/funded-companies-hiring)**: Companies that just raised money (SEC Form D), with their open roles and business contacts, as a new-only monitor. - **[Google Maps Phone Scraper](https://apify.com/aurelix/google-maps-scraper)**: Local business lists from Google Maps: phones, websites, ratings, hours, coordinates; optional website emails and socials. - **[Recently Funded Companies API](https://apify.com/aurelix/recently-funded-companies)**: US companies that just filed an SEC Form D: amount raised, industry, state, executives, confirmed website. - **[Tech Stack API](https://apify.com/aurelix/tech-stack-detector)**: CMS, e-commerce platform, analytics, ad pixels, chat, CDN and email provider for each domain you enter. - **[Yellow Pages Canada Scraper](https://apify.com/aurelix/yellow-pages-canada-scraper)**: YellowPages.ca businesses by search term and city: phone, website, address, postal code, hours and rating. #### Ad intelligence - **[Competitor Watch](https://apify.com/aurelix/competitor-watch)**: One report per competitor with only what is new: Meta and TikTok ads, page changes, Shopify products and prices, jobs. - **[Facebook Ad Library Search & API](https://apify.com/aurelix/facebook-ad-library-scraper)**: Facebook and Instagram business ads from the Meta Ad Library: copy, creatives, landing pages, dates, optional EU reach. - **[TikTok Ads Library API](https://apify.com/aurelix/tiktok-ads-library-scraper)**: TikTok Ads Library ads for Europe, UK, Switzerland and Turkey with targeting, reach, ad text and landing pages. #### Canada and government data - **[Canada Building Permits](https://apify.com/aurelix/canada-building-permits-scraper)**: Building permits and development applications from 7 Canadian cities: address, type, status, dates, value, applicant, contractor and Toronto planner contact. - **[Government Tenders Scraper](https://apify.com/aurelix/government-tenders-scraper)**: Open tenders from SAM.gov, CanadaBuys and EU TED in one table: buyer, codes, value, deadline, contact and link. - **[USPTO Trademark API](https://apify.com/aurelix/uspto-trademark-search)**: US trademark search and watch from USPTO: mark, owner or number lookups, new filings, owner, attorney, goods. #### AI, content and SEO - **[Google Trends by Region](https://apify.com/aurelix/google-trends-scraper)**: Google Trends interest over time, by region and rising queries for any number of keywords, plus trending searches. - **[Google Autocomplete API](https://apify.com/aurelix/google-autocomplete-scraper)**: Expands seed keywords into hundreds of real Google and YouTube autocomplete phrases, tagged by search intent. - **[PDF Table Extraction & PDF to Markdown OCR](https://apify.com/aurelix/pdf-to-markdown)**: Turns linked PDF and Word files into Markdown per page, with tables as JSON rows, RAG chunks and OCR. - **[Broken Link Crawler & SEO Audit](https://apify.com/aurelix/seo-audit-crawler)**: Audits a whole site: one row per page with SEO checks and broken links, plus a ranked issues summary. - **[Website to Markdown API](https://apify.com/aurelix/website-to-markdown)**: Crawls a site and returns one row of clean main-content Markdown per page, with title, headings and links. - **[YouTube Transcript Scraper](https://apify.com/aurelix/youtube-transcript-scraper)**: Returns YouTube video transcripts as text, timed segments and AI chunks, from videos, channels, playlists or searches. ### Recent changes - **0.3**: 1 GB default memory; conservative URL and number matching (fragments and other parameters kept, no phone guessing); per-MB read fee; only-new memory saved after every batch; Excel capped by its text table; keep last recovers when a source breaks in the second pass. - **0.2**: Stricter smart matching (signs, decimals and leading zeros kept; schemeless URLs normalised); only-new also remembers rows without dedupe values; keep last stays correct when a source breaks; compact only-new memory; Excel streamed; bare dataset names accepted. - **0.1**: First release: datasets, CSV, TSV, JSON, JSON Lines, Excel and Google Sheets links, smart matching, keep first or last, only-new mode, STATS record. The full history is on the **Changelog** tab. ### Tested daily An automated health check runs this Actor every day on a fixed input and checks that at least 16 rows come back with `name`, `website`, `_source` filled in; a failed check alerts the maintainer straight away. ### Your feedback Missing a field, found a bug or need another filter? Open an issue on the **Issues** tab; we aim to answer within a day. If this Actor saved you time, a short review helps other people find it. # Changelog This Actor's version history is a separate document: https://apify.com/aurelix/dataset-merge-dedup/changelog.md # Actor input Schema ## `datasetIds` (type: `array`): Apify datasets to merge, e.g. the results of several scraper runs. Pick them here or paste dataset IDs or names; Apify then lets this Actor read them. In an integration use the run's defaultDatasetId. ## `fileUrls` (type: `array`): Public links to CSV, TSV, JSON, JSON Lines or Excel (.xlsx) files, e.g. https://example.com/leads.csv: up to 200 MB each (Excel up to 30 MB with up to 40 MB of text). Google Sheets links work when the sheet is shared 'anyone with the link'. Your own Apify datasets go in 'Datasets'; a dataset link here works only if anyone with the link may read it. ## `rows` (type: `array`): Rows pasted as a JSON array of objects, e.g. \[{"name": "Acme", "website": "acme.com"}]. Handy for tests and AI agents. ## `dedupeFields` (type: `array`): Rows with the same values in these fields are duplicates, e.g. 'website', or 'name' plus 'city'. Dots reach nested values ('company.domain'). Field names are matched forgiving case and spaces. Empty: rows are duplicates when every field is equal, ignoring timestamps such as scraped_at. ## `smartMatch` (type: `boolean`): Compare values the way people mean them: URLs (with or without http(s)://) ignoring www., the host's case, one trailing slash and utm\_\*, gclid and fbclid tags (every other parameter, the path's case and #fragments still count); emails in any case; numbers such as 5 and 5.0 as one. Nothing else is guessed: phone numbers, dates and IDs must be written the same way to match. ## `ignoreCase` (type: `boolean`): 'ACME Inc' and 'acme inc' count as the same value. ## `keep` (type: `string`): Which row of each duplicate group stays. 'Last' reads the sources twice, so it takes about twice as long. ## `emptyKeyRows` (type: `string`): What to do with rows that have no value in any dedupe field. Kept rows are never merged with each other, so missing data does not wipe rows out. ## `onlyNewEntries` (type: `boolean`): Also skip rows whose dedupe key an earlier run with the same memory name already returned. Use after a scheduled scraper to pass on only new rows. ## `stateKey` (type: `string`): Name for the 'only new' memory, e.g. 'daily-leads'. Runs with different names keep separate memories. ## `outputFields` (type: `array`): Keep only these fields, in this order, e.g. 'name', 'website', 'company.domain' (a nested value becomes its own column). Empty: every field. ## `addSource` (type: `boolean`): Tag each row with where it came from: dataset:, file: or rows. ## `maxRows` (type: `integer`): Stop after writing this many rows, e.g. 1000. 0 = no limit. ## `excelSheet` (type: `string`): Worksheet to read in Excel files, e.g. 'Leads'. Empty: the first sheet. ## Actor input object example ```json { "fileUrls": [ "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv", "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json" ], "rows": [], "dedupeFields": [ "website" ], "smartMatch": true, "ignoreCase": true, "keep": "first", "emptyKeyRows": "keep", "onlyNewEntries": false, "outputFields": [], "addSource": true, "maxRows": 0 } ``` # Actor output Schema ## `rows` (type: `string`): The merged rows with duplicates removed, in source order. ## `stats` (type: `string`): Rows read, written and removed, per source, with any source that could not be read and why. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "fileUrls": [ "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv", "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json" ], "dedupeFields": [ "website" ] }; // Run the Actor and wait for it to finish const run = await client.actor("aurelix/dataset-merge-dedup").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "fileUrls": [ "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv", "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json", ], "dedupeFields": ["website"], } # Run the Actor and wait for it to finish run = client.actor("aurelix/dataset-merge-dedup").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "fileUrls": [ "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-a.csv", "https://api.apify.com/v2/key-value-stores/twkCCtTWzsi8fVbx5/records/merge-sample-b.json" ], "dedupeFields": [ "website" ] }' | apify call aurelix/dataset-merge-dedup --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,aurelix/dataset-merge-dedup" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/c2XwdVaAANEKi02LW/builds/Fpk7wT5vOmNljiskr/openapi.json