Dataset Merge, Dedupe & Clean: Combine Apify Datasets

Merge several Apify datasets into one, remove duplicates by website, email, phone or any field, and clean the rows. You get one tidy dataset, ready for a CRM, a spreadsheet or the next Actor.

Smart matching knows that https://www.acme.com/, acme.com and http://ACME.com/?utm_source=maps are the same website. It also matches Info@Acme.com with info@acme.com, and +1 (415) 555-0100 with 001 415 555 0100.

What you get

• Merge up to 20 datasets per run, in the order you pick them, plus pasted JSON rows.
• Remove duplicates:
  • by one field (website), by several (name + city), or by the whole row, ignoring timestamps such as scrapedAt;
  • keeping the first or the last copy (for example, the newest run).
• Smart matching per field. URLs ignore https, www., a trailing slash, #anchors and tracking tags (utm_*, gclid, fbclid). Emails ignore case and mailto:. Phones compare by digits. Other text ignores case and extra spaces. Choose exact to compare byte for byte.
• Filter: drop rows missing a field you need, such as website or email.
• Shape: keep only some fields (in your order), remove fields, rename fields, trim spaces, and remove empty values.
• Trace: an optional _source column says which dataset and row each result came from.
• SUMMARY record: rows read, duplicates removed, rows dropped and rows written, per source.

Sample: these 7 pasted rows (the prefilled input) become 5.

name	website	kept?
Python Software Foundation	https://www.python.org/	yes
Mozilla	https://www.mozilla.org/en-US/	yes
Wikimedia Foundation	https://wikimediafoundation.org	yes
Python Software Foundation	python.org	duplicate of row 1
Mozilla Foundation	http://mozilla.org/en-US?utm_source=maps	duplicate of row 2
Apache Software Foundation	https://www.apache.org/	yes
No website listed	(empty)	yes: rows with an empty key are never treated as duplicates

SUMMARY for that run:

{
  "rowsRead": 7,
  "duplicatesRemoved": 2,
  "missingRequiredFields": 0,
  "rowsWithEmptyKey": 1,
  "rowsWritten": 5,
  "uniqueKeys": 4,
  "dedupeBy": ["website"],
  "matchMode": "auto",
  "keep": "first",
  "sources": [{"source": "rows", "rowsRead": 7, "rowsWritten": 5}]
}

Use cases

• Lead lists from several scraper runs: merge Google Maps Scraper runs for different cities or keywords, and keep one row per business website.
• Weekly refresh: merge last week's and this week's results, and keep the newest copy of each item (keep: last).
• Clean before export: drop rows without an email or website, keep only the columns your CRM needs, and rename them to match it.
• Chain Actors: run it as an integration after any scraper, and pass the clean dataset to an enrichment Actor.

How to use it

1. In Datasets, pick the datasets to merge (from your runs or saved datasets). When you pick one, the sample rows are ignored.
2. In Duplicate key fields, enter the field or fields that identify a duplicate, for example website. Leave it empty to compare whole rows.
3. Optionally set rows to drop, fields to keep, remove or rename, and Maximum rows to write.
4. Run it. The merged rows are in the Output tab and the counts are in the SUMMARY record.

Field names are the ones in your dataset. Use dots for nested fields, for example address.city. With How to compare values set to auto, fields named like website, url or domain are compared as URLs, email as emails, and phone as phones.

In Fields to keep, a nested field such as address.city becomes a column with that exact name, and a field missing from a row is written as empty. When a renamed field gets the name of a field that already exists, the renamed value replaces it.

Pricing

You pay per row written to the output dataset, plus Apify's small start fee; the current price is on the Pricing tab. Duplicates removed, rows dropped for a missing field and rows over your maximum are never charged. Set Maximum rows to write, or a maximum cost per run, to cap the cost.

Automate it: schedules and integrations

• After another Actor: on the scraper's Integrations tab, add this Actor and set its input to {"datasetIds": ["{{resource.defaultDatasetId}}"], "dedupeBy": ["website"]} (plus your other settings). Every finished run is then cleaned automatically. Naming the dataset in Datasets is what allows the Actor, which runs with limited permissions, to read it.
• Schedules: run it after your weekly scrapes with keep: last, to keep a current list.
• Export and connect: download as CSV, Excel or JSON, or send the result to Google Sheets, Slack, Make, Zapier or n8n through Apify's integrations. You can also call it from the Apify API or the Apify MCP server.

Related Actors

• Tech Stack & Domain Lookup: tech stack, email provider, DMARC and domain age for the websites in your merged dataset.
• Google Maps Contact Details Enricher: company emails, phones and social links from those websites.

Limitations

• Duplicates must match on the key after normalization. It does not do fuzzy matching ("Acme Inc" and "Acme Incorporated" are different).
• Lists are compared with their order ignored (repeated items still count). Nested objects are compared field by field.
• With keep: last every row is read twice. If the run hits its timeout, it stops early and says so; a key whose last copy came after the stop is not written. Raise the timeout for very large jobs.
• It reads at most 10 million rows per run. Memory grows by about 85 bytes per distinct key (about 180 with keep: last), so 1 GB holds about 10 million keys (about 5 million with keep: last). Raise the run's memory for bigger jobs.
• It reads only datasets you pick or pass in: it runs with limited permissions and cannot see your other storage.
• It does not join rows (fill one row's empty fields from another). It keeps one whole row per key.

FAQ

Will it change my original datasets? No. It only reads them, and writes the result to this run's own dataset.

Which copy of a duplicate is kept? The first one by default: datasets in the order you picked them, then pasted rows. Choose last to keep the latest copy instead.

Are rows with an empty website removed as duplicates of each other? No. A row whose key fields are all empty is always kept, and counted as rowsWithEmptyKey. To drop such rows, list the field in Drop rows missing these fields.

Can I merge datasets from other people's runs? Only datasets your account can read. Pick them in the Datasets field so the Actor is allowed to read them.

Support

Found a bug or need a feature? Open an issue on the Issues tab, where fixes for broken runs are a priority. For a custom version or a question you'd rather not post publicly, email stellalytics@gmail.com. Never send API keys or passwords by email or in an issue.