# Dataset Merge, Dedupe & Clean: Combine Apify Datasets (`stellalytics/dataset-merge-dedupe`) Actor Merge Apify datasets and remove duplicates by website, email, phone or any field, with smart matching (www, https, tracking tags, case). Drop rows missing a field, keep or rename columns, trim text. Pay per row written. Works as an integration after any scraper. - **URL**: https://apify.com/stellalytics/dataset-merge-dedupe.md - **Developed by:** [Stellalytics](https://apify.com/stellalytics) (community) - **Categories:** Developer tools, Automation, Lead generation - **Features:** Limited permissions - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.30 / 1,000 row writtens This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Dataset Merge, Dedupe & Clean: Combine Apify Datasets Merge several Apify datasets into one, **remove duplicates by website, email, phone or any field**, and clean the rows. You get one tidy dataset, ready for a CRM, a spreadsheet or the next Actor. Smart matching knows that `https://www.acme.com/`, `acme.com` and `http://ACME.com/?utm_source=maps` are the same website. It also matches `Info@Acme.com` with `info@acme.com`, and `+1 (415) 555-0100` with `001 415 555 0100`. ### What you get - **Merge** up to 20 datasets per run, in the order you pick them, plus pasted JSON rows. - **Remove duplicates**: - by one field (`website`), by several (`name` + `city`), or by the whole row, ignoring timestamps such as `scrapedAt`; - keeping the first or the last copy (for example, the newest run). - **Smart matching per field.** URLs ignore `https`, `www.`, a trailing slash, `#anchors` and tracking tags (`utm_*`, `gclid`, `fbclid`). Emails ignore case and `mailto:`. Phones compare by digits. Other text ignores case and extra spaces. Choose `exact` to compare byte for byte. - **Filter:** drop rows missing a field you need, such as `website` or `email`. - **Shape:** keep only some fields (in your order), remove fields, rename fields, trim spaces, and remove empty values. - **Trace:** an optional `_source` column says which dataset and row each result came from. - **SUMMARY record:** rows read, duplicates removed, rows dropped and rows written, per source. Sample: these 7 pasted rows (the prefilled input) become 5. | name | website | kept? | |---|---|---| | Python Software Foundation | https://www.python.org/ | yes | | Mozilla | https://www.mozilla.org/en-US/ | yes | | Wikimedia Foundation | https://wikimediafoundation.org | yes | | Python Software Foundation | python.org | duplicate of row 1 | | Mozilla Foundation | http://mozilla.org/en-US?utm_source=maps | duplicate of row 2 | | Apache Software Foundation | https://www.apache.org/ | yes | | No website listed | *(empty)* | yes: rows with an empty key are never treated as duplicates | SUMMARY for that run: ```json { "rowsRead": 7, "duplicatesRemoved": 2, "missingRequiredFields": 0, "rowsWithEmptyKey": 1, "rowsWritten": 5, "uniqueKeys": 4, "dedupeBy": ["website"], "matchMode": "auto", "keep": "first", "sources": [{"source": "rows", "rowsRead": 7, "rowsWritten": 5}] } ``` ### Use cases - **Lead lists from several scraper runs:** merge Google Maps Scraper runs for different cities or keywords, and keep one row per business website. - **Weekly refresh:** merge last week's and this week's results, and keep the newest copy of each item (`keep: last`). - **Clean before export:** drop rows without an email or website, keep only the columns your CRM needs, and rename them to match it. - **Chain Actors:** run it as an integration after any scraper, and pass the clean dataset to an enrichment Actor. ### How to use it 1. In **Datasets**, pick the datasets to merge (from your runs or saved datasets). When you pick one, the sample rows are ignored. 2. In **Duplicate key fields**, enter the field or fields that identify a duplicate, for example `website`. Leave it empty to compare whole rows. 3. Optionally set rows to drop, fields to keep, remove or rename, and **Maximum rows to write**. 4. Run it. The merged rows are in the Output tab and the counts are in the SUMMARY record. Field names are the ones in your dataset. Use dots for nested fields, for example `address.city`. With **How to compare values** set to `auto`, fields named like `website`, `url` or `domain` are compared as URLs, `email` as emails, and `phone` as phones. In **Fields to keep**, a nested field such as `address.city` becomes a column with that exact name, and a field missing from a row is written as empty. When a renamed field gets the name of a field that already exists, the renamed value replaces it. ### Pricing You pay **per row written** to the output dataset, plus Apify's small start fee; the current price is on the Pricing tab. Duplicates removed, rows dropped for a missing field and rows over your maximum are never charged. Set **Maximum rows to write**, or a maximum cost per run, to cap the cost. ### Automate it: schedules and integrations - **After another Actor:** on the scraper's **Integrations** tab, add this Actor and set its input to `{"datasetIds": ["{{resource.defaultDatasetId}}"], "dedupeBy": ["website"]}` (plus your other settings). Every finished run is then cleaned automatically. Naming the dataset in **Datasets** is what allows the Actor, which runs with limited permissions, to read it. - **Schedules:** run it after your weekly scrapes with `keep: last`, to keep a current list. - **Export and connect:** download as CSV, Excel or JSON, or send the result to Google Sheets, Slack, Make, Zapier or n8n through Apify's integrations. You can also call it from the Apify API or the Apify MCP server. ### Related Actors - [Tech Stack & Domain Lookup](https://apify.com/stellalytics/website-domain-intelligence): tech stack, email provider, DMARC and domain age for the websites in your merged dataset. - [Google Maps Contact Details Enricher](https://apify.com/stellalytics/business-contact-finder): company emails, phones and social links from those websites. ### Limitations - Duplicates must match on the key after normalization. It does not do fuzzy matching ("Acme Inc" and "Acme Incorporated" are different). - Lists are compared with their order ignored (repeated items still count). Nested objects are compared field by field. - With `keep: last` every row is read twice. If the run hits its timeout, it stops early and says so; a key whose last copy came after the stop is not written. Raise the timeout for very large jobs. - It reads at most 10 million rows per run. Memory grows by about 85 bytes per distinct key (about 180 with `keep: last`), so 1 GB holds about 10 million keys (about 5 million with `keep: last`). Raise the run's memory for bigger jobs. - It reads only datasets you pick or pass in: it runs with limited permissions and cannot see your other storage. - It does not join rows (fill one row's empty fields from another). It keeps one whole row per key. ### FAQ **Will it change my original datasets?** No. It only reads them, and writes the result to this run's own dataset. **Which copy of a duplicate is kept?** The first one by default: datasets in the order you picked them, then pasted rows. Choose `last` to keep the latest copy instead. **Are rows with an empty website removed as duplicates of each other?** No. A row whose key fields are all empty is always kept, and counted as `rowsWithEmptyKey`. To drop such rows, list the field in **Drop rows missing these fields**. **Can I merge datasets from other people's runs?** Only datasets your account can read. Pick them in the Datasets field so the Actor is allowed to read them. ### Support Found a bug or need a feature? Open an issue on the Issues tab, where fixes for broken runs are a priority. For a custom version or a question you'd rather not post publicly, email . Never send API keys or passwords by email or in an issue. # Actor input Schema ## `datasetIds` (type: `array`): Datasets to merge, in order: for example two Google Maps Scraper runs. When you add this Actor as an integration after another Actor, set this field to {{resource.defaultDatasetId}}. When you pick a dataset, the sample rows below are ignored. ## `rows` (type: `array`): Or paste up to 10,000 rows as a JSON array of objects. The prefilled sample has duplicate websites written in different ways, so you can see the result. ## `dedupe` (type: `boolean`): Keep one row per duplicate key. ## `dedupeBy` (type: `array`): Rows are duplicates when these fields match, for example website, or name + city. Use dots for nested fields (address.city). Leave empty to compare whole rows (timestamps such as scrapedAt are ignored). ## `matchMode` (type: `string`): auto: URLs, emails and phones by their own rules (https://www.site.com/ = site.com; tracking parameters dropped; emails lowercased; phones by digits), other text ignoring case and spacing. exact: byte for byte. ## `keep` (type: `string`): first: the earliest row (datasets in the order given). last: the latest, for example the newest run. ## `ignoreFields` (type: `array`): Only used when 'Duplicate key fields' is empty. Default: scrapedAt, scraped_at, crawledAt, checkedAt, fetchedAt, retrievedAt, timestamp, #debug, #error. ## `requireFields` (type: `array`): Rows where any of these fields is empty are dropped, for example website or email. ## `fields` (type: `array`): Output only these fields, in this order. Leave empty to keep all. ## `omitFields` (type: `array`): For example #debug or imageUrls. ## `renameFields` (type: `object`): For example {"title": "company", "phoneUnformatted": "phone"}. ## `trimStrings` (type: `boolean`): Remove leading, trailing and repeated spaces in every text value. ## `removeEmptyFields` (type: `boolean`): Drop fields that are null, empty text, empty lists or empty objects (0 and false stay). ## `addSource` (type: `boolean`): Add \_source with the dataset and row number each row came from. ## `maxRows` (type: `integer`): 0 means no limit. You pay per row written. ## `useSampleRowsWithDatasets` (type: `boolean`): Off by default, so the sample rows never mix with your data. ## Actor input object example ```json { "rows": [ { "name": "Python Software Foundation", "website": "https://www.python.org/", "city": "Beaverton", "source": "maps run 1" }, { "name": "Mozilla", "website": "https://www.mozilla.org/en-US/", "city": "San Francisco", "source": "maps run 1" }, { "name": "Wikimedia Foundation", "website": "https://wikimediafoundation.org", "city": "San Francisco", "source": "maps run 1" }, { "name": "Python Software Foundation ", "website": "python.org", "city": "Beaverton", "source": "maps run 2" }, { "name": "Mozilla Foundation", "website": "http://mozilla.org/en-US?utm_source=maps", "city": "San Francisco", "source": "maps run 2" }, { "name": "Apache Software Foundation", "website": "https://www.apache.org/", "city": "Wilmington", "source": "maps run 2" }, { "name": "No website listed", "website": "", "city": "Unknown", "source": "maps run 2" } ], "dedupe": true, "dedupeBy": [ "website" ], "matchMode": "auto", "keep": "first", "trimStrings": true, "removeEmptyFields": false, "addSource": false, "maxRows": 0, "useSampleRowsWithDatasets": false } ``` # Actor output Schema ## `results` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "rows": [ { "name": "Python Software Foundation", "website": "https://www.python.org/", "city": "Beaverton", "source": "maps run 1" }, { "name": "Mozilla", "website": "https://www.mozilla.org/en-US/", "city": "San Francisco", "source": "maps run 1" }, { "name": "Wikimedia Foundation", "website": "https://wikimediafoundation.org", "city": "San Francisco", "source": "maps run 1" }, { "name": "Python Software Foundation ", "website": "python.org", "city": "Beaverton", "source": "maps run 2" }, { "name": "Mozilla Foundation", "website": "http://mozilla.org/en-US?utm_source=maps", "city": "San Francisco", "source": "maps run 2" }, { "name": "Apache Software Foundation", "website": "https://www.apache.org/", "city": "Wilmington", "source": "maps run 2" }, { "name": "No website listed", "website": "", "city": "Unknown", "source": "maps run 2" } ], "dedupeBy": [ "website" ] }; // Run the Actor and wait for it to finish const run = await client.actor("stellalytics/dataset-merge-dedupe").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "rows": [ { "name": "Python Software Foundation", "website": "https://www.python.org/", "city": "Beaverton", "source": "maps run 1", }, { "name": "Mozilla", "website": "https://www.mozilla.org/en-US/", "city": "San Francisco", "source": "maps run 1", }, { "name": "Wikimedia Foundation", "website": "https://wikimediafoundation.org", "city": "San Francisco", "source": "maps run 1", }, { "name": "Python Software Foundation ", "website": "python.org", "city": "Beaverton", "source": "maps run 2", }, { "name": "Mozilla Foundation", "website": "http://mozilla.org/en-US?utm_source=maps", "city": "San Francisco", "source": "maps run 2", }, { "name": "Apache Software Foundation", "website": "https://www.apache.org/", "city": "Wilmington", "source": "maps run 2", }, { "name": "No website listed", "website": "", "city": "Unknown", "source": "maps run 2", }, ], "dedupeBy": ["website"], } # Run the Actor and wait for it to finish run = client.actor("stellalytics/dataset-merge-dedupe").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "rows": [ { "name": "Python Software Foundation", "website": "https://www.python.org/", "city": "Beaverton", "source": "maps run 1" }, { "name": "Mozilla", "website": "https://www.mozilla.org/en-US/", "city": "San Francisco", "source": "maps run 1" }, { "name": "Wikimedia Foundation", "website": "https://wikimediafoundation.org", "city": "San Francisco", "source": "maps run 1" }, { "name": "Python Software Foundation ", "website": "python.org", "city": "Beaverton", "source": "maps run 2" }, { "name": "Mozilla Foundation", "website": "http://mozilla.org/en-US?utm_source=maps", "city": "San Francisco", "source": "maps run 2" }, { "name": "Apache Software Foundation", "website": "https://www.apache.org/", "city": "Wilmington", "source": "maps run 2" }, { "name": "No website listed", "website": "", "city": "Unknown", "source": "maps run 2" } ], "dedupeBy": [ "website" ] }' | apify call stellalytics/dataset-merge-dedupe --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,stellalytics/dataset-merge-dedupe" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/iVWelPPom78rpl3wU/builds/GsyZdBaONb8okNx2p/openapi.json