Merge Datasets & Remove Duplicates: Combine Runs, CSVs, Sheets

Merge datasets into one list and remove the duplicates

Ran the same scraper once per city, keyword or week and now have ten separate datasets full of overlap? Merge Datasets & Remove Duplicates takes any number of Apify datasets, CSV or Excel files and Google Sheets, stacks them into one list, removes the records that appear in more than one of them, fills in the blanks from the copies it removed, and tells you which list every row came from. No code, no spreadsheet formulas.

It will:

• Merge many datasets, files and Google Sheets in one run (up to 100 sources and 300,000 rows read per run)
• Remove duplicates across them by the fields you choose (phone, website, placeId, url...) or by the whole row
• Match sensibly: 0117 496 0012 matches 01174960012, and https://www.acme.com/ matches acme.com
• Fill blanks from the duplicates: an email found in one run and a rating found in another end up on the same row
• Label every row with the list it came from, every list it appeared in, and how many copies were merged
• Export a ready-to-open CSV and Excel file, append to a named dataset, or POST the result to a webhook

Why use this?

Splitting a big scrape into many smaller runs is normal (one run per city, per search term, per day), but it leaves you with many datasets and a lot of overlap: the same business shows up in two neighbouring towns, the same product in two categories, the same job in two searches. Merging those by hand means exporting each one, pasting them together and removing duplicates in a spreadsheet.

• Lead generation: combine Google Maps or directory runs from many areas into one de-duplicated lead list, keeping the most complete record for each business
• Ecommerce and price tracking: merge category or keyword runs into one product list without double counting
• Job and listing data: combine searches into one feed with each listing once
• Combining sources: merge an Apify dataset with a CSV export from your CRM and a shared Google Sheet in one step
• AI agents and pipelines: one deterministic call that turns several result sets into one clean list

How to use it

1. Run your scraper as many times as you need. Each run's results are a dataset.
2. Open this Actor and pick those datasets in Datasets to merge, and/or paste links in Files or Google Sheets to merge.
3. In Match duplicates on these fields, enter the field that identifies the same record, for example phone, website, placeId or url. Leave it empty to compare whole rows.
4. Run it. Download the CSV or Excel file from the Output tab, or read the merged dataset through the API.

Try it with the example input: two small sample lead lists with three businesses in both, written in different phone formats. The result is 9 unique businesses from 12 rows.

Input

Field	What it does
Datasets to merge	Any number of Apify datasets, picked with the dataset picker
Files or Google Sheets to merge	CSV, TSV, Excel, JSON or JSON Lines links, or Google Sheet links, one per line
Inline data	Optional JSON array, merged in as one more list
Match duplicates on these fields	Field names that identify the same record; empty means compare whole rows
How strictly to match	Normalized (default), Exact, Fuzzy (near matches, up to 5,000 rows) or None (just stack the lists)
Which duplicate to keep	The most complete row (default), the first one seen or the last one seen
Fill blanks from the duplicates	On by default; never overwrites a value that is already there
Add source columns	On by default: _source, _foundInSources, _duplicatesMerged
Max output rows, named dataset, download files, webhook	Optional extras

Example input:

{
    "datasetIds": ["aBcD1234efGh5678i", "jKlM9012noPq3456r"],
    "fileUrls": ["https://docs.google.com/spreadsheets/d/your-sheet-id/edit"],
    "dedupKeys": ["phone"],
    "dedupMode": "normalized"
}

Lists are read in the order given: datasets first, then files, then inline data. Every column from every list is kept; a row simply has no value in a column its list did not have.

Output

One row per unique record, with all of its original fields plus three source columns:

{
    "name": "Harbour Plumbing Ltd",
    "phone": "01174960012",
    "website": "harbour-plumbing.example.com",
    "email": "jobs@harbour-plumbing.example.com",
    "city": "Bristol",
    "rating": "4.8",
    "_source": "sample-leads-run-2.csv",
    "_foundInSources": "sample-leads-run-1.csv | sample-leads-run-2.csv",
    "_duplicatesMerged": 1
}

Column	Meaning
_source	The list the kept row came from (a dataset's name or ID, or a file's name)
_foundInSources	Every list the record appeared in, separated by |
_duplicatesMerged	How many copies were merged into this row (0 means it was unique)

The run also saves:

• merged-data.csv and merged-data.xlsx, ready-to-open files of the merged list
• MERGE_REPORT: rows read from each source, any source that could not be read and why, duplicates removed, blanks filled, the full column list and any warnings

You can also download the dataset in JSON, CSV, Excel, XML or HTML from the Output tab.

How much does it cost to merge datasets?

You pay per merged row written, plus a small fee per download file:

Event	Price (Free plan)	Bronze	Silver	Gold and above
Merged row written	$0.001	$0.0009	$0.0008	$0.0007
Download file (CSV or Excel)	$0.01	$0.009	$0.008	$0.007
Webhook delivered (2xx only)	$0.02	$0.018	$0.016	$0.014

Rows read, duplicates removed and file downloads are never charged. Example: merging 10 city runs of 2,000 rows each, where 20% of rows are duplicates, writes 16,000 unique rows: $16.00 on the Free plan or $11.20 on Gold, plus $0.02 for the CSV and Excel files. Set Max output rows or a spending limit on the run to cap the cost; the Actor stops writing at your limit and says so.

Tips

• Pick the field that really identifies a record. For Google Maps use placeId (or phone), for web pages url, for products an ID or SKU. Whole-row matching misses duplicates whose ratings or review counts changed between runs.
• Use several fields together when one is not unique, for example name and city.
• Normalized matching ignores case, spaces, punctuation, accents, http(s):// and www.. It does not convert country codes, so +44 117... and 0117... are different values; use Fuzzy matching for lists under 5,000 rows if that matters.
• Only new records? If you want just the records that were not in last week's run, use Dataset Diff & Change Detector. To merge two lists side by side on a shared key (like a VLOOKUP), use Dataset Join & Merge. To clean up one list (emails, phones, nested fields), use Dataset Cleaner & Exporter.
• Pipelines: call it from a webhook or the API after your last scraper run finishes, with outputDatasetName to keep one growing master list.

For AI agents

Pass datasetIds (array of dataset IDs) and/or fileUrls (array of links), plus dedupKeys (array of field names). The merged rows are the run's default dataset; MERGE_REPORT in the key-value store has the counts. Charged per merged row written ($0.001), per download file ($0.01) and per delivered webhook ($0.02). Works with limited permissions, through the Apify MCP server and with x402 payments.

FAQ

Does it change my original datasets or files? No. It only reads them and writes a new dataset.

What if one of the datasets or links cannot be read? It is skipped, the reason is written to MERGE_REPORT and the run log, and the rest are still merged. The run fails only if no source can be read.

Can it merge every run of a scraper automatically? Pick the datasets you want, or send their IDs through the API. Reading all of an Actor's runs on its own would need full access to your account, which this Actor deliberately does not ask for.

Is there a size limit? Up to 300,000 rows read per run across all sources, and 100 MB per file. Both the CSV and the Excel file cover the whole merged list.

Something not right? Open an issue on the Issues tab and I will look at it the same day. If this saved you stitching runs together by hand, a review on the Store page helps a lot.