Merge & Deduplicate Datasets - Only New Items, Join, CSV/XLSX

Merge Apify datasets, CSV, JSON, JSONL or Excel files and pasted rows into one clean dataset: remove duplicates by any fields (case and whitespace normalised), keep only items new since your last run, left/inner join on a key, flatten nested JSON. $0.05 per 1,000 rows out.

Combine scraper outputs into one clean, duplicate-free dataset, reliably. Pick any number of Apify datasets, add CSV, TSV, JSON, JSON Lines or Excel (XLSX) file URLs or paste rows as JSON, and get one merged dataset with duplicates removed by the fields you choose (email, URL, product ID, company name + city, or the whole row). Values are normalised before comparing, so " ACME Ltd" and "acme ltd", or 42 and "42", count as the same, while your rows stay exactly as they were. Turn on only new items and the actor remembers what it delivered (in a store on your own account), so a scheduled scrape outputs only the listings, jobs, leads or prices that are new since the last run, perfect for alerts, CRMs and Slack/Make/n8n/Zapier pipelines. Add a left or inner join on a key (e.g. enrich leads with a company lookup table by domain), flatten nested JSON into spreadsheet columns, keep or drop fields, and tag each row with its source. It streams datasets page by page, so millions of rows work, and it runs with limited permissions: it can read only the datasets you pick. Export the result as CSV, Excel, JSON, JSONL or XML from the dataset. $0.05 per 1,000 rows delivered; duplicates and skipped rows are free.

What data does it return?

Every run outputs clean, schema-validated rows like these:

id	name	_source	url	title	price
1	a	input:rows
2	b	input:rows
		dataset:Xk3...R9	https://shop.example.com/p/42	Trail Shoe X2	129

How much does it cost?

Transparent pay-per-event pricing — you only pay for what you get, and the actor always respects your run budget: set a maximum total charge and it stops cleanly at the cap instead of overrunning.

Event	Price	Charged	Worked example
row-output	$0.00005	per row delivered (duplicates removed, rows already seen and failed sources are free)	Merge 3 datasets of 40,000 rows into 25,000 unique rows = $1.25
apify-actor-start	$0.00005	per run per GB of memory (platform start fee)	One run at 1 GB = $0.00005

Input options

Field	Type	Required	Default	Description
datasetIds	array	no	—	Apify datasets to merge (pick them, or pass dataset IDs via the API). Picking a dataset here gives this run read access to it; nothing else on your account is readable. Read in the order given, after any pasted rows.
fileUrls	array	no	—	Public URLs of CSV/TSV (delimiter auto-detected, first row = column names), JSON (an array, or an object with an items/data/results array), JSON Lines or XLSX (first visible sheet) files, up to 100 MB each. Read after the datasets.
rows	array	no	—	Rows pasted as a JSON array of objects. Read first.
dedupeFields	array	no	—	Rows with the same values in all of these fields are duplicates; the first one is kept. Dot paths reach nested values (e.g. address.postcode). Leave empty to compare whole rows. Rows where all these fields are empty are always kept.
normalizeValues	boolean	no	true	Compare text case-insensitively with leading/trailing and repeated whitespace ignored (" ACME Ltd" = "acme ltd"), and a number equal to its text form (42 = "42"). The rows themselves are not changed.
onlyNewItems	boolean	no	false	Remember the dedupe keys of every delivered row (in a key-value store on your own account) and skip rows already delivered by an earlier run with the same history name. Ideal for scheduled scrapes: each run outputs only what is new.
historyKey	string	no	"default"	Separate 'only new items' memories for different pipelines, e.g. jobs-london or competitor-prices. Letters, digits, - and _.
joinDatasetId	string	no	—	Optional. A dataset whose rows are joined onto the merged rows (e.g. company details by domain). Up to 500,000 lookup rows; the first row per key is used.
joinFileUrl	string	no	—	Optional, instead of a lookup dataset: a CSV, JSON, JSONL or XLSX file URL with the lookup rows.
joinOn	string	no	—	Field of the merged rows to match on (dot paths allowed). Required for a join.
joinLookupField	string	no	—	Field of the lookup rows to match on. Defaults to the same name as 'Join on field'.
joinPrefix	string	no	""	Optional prefix for the fields copied from the lookup row, e.g. company_. Lookup fields only fill fields that are missing or empty in the row, so use a prefix to keep both versions.
joinType	string	no	"left"	Left: keep every row (unmatched rows unchanged). Inner: keep only rows that matched a lookup row.
flatten	boolean	no	false	Turn nested objects and arrays into flat columns (address.city, tags.0), handy for spreadsheets and CRMs.
fields	array	no	—	Optional. Output only these fields, in this order (dot paths allowed; the path becomes the column name).
omitFields	array	no	—	Optional. Drop these fields from every row (after flattening, use the flat names).
addSourceField	boolean	no	false	Tag each row with where it came from (input:rows, dataset:<id> or file:<url>).
maxItems	integer	no	1000000	Stop after delivering this many rows.

Limits & honest notes

• Sources are read in order (pasted rows, then datasets, then files) and the first occurrence of each key is kept. Rows whose dedupe fields are all empty are never treated as duplicates.
• Dedupe keys are kept in memory as 53-bit hashes: about 5–10 million unique keys per run at 1 GB (raise the memory for more). The chance of two different keys colliding is negligible (about 1 in 10,000 at 1 million keys).
• Only new items: the remembered keys live in the key-value store "dataset-merge-dedupe-history" on your own account (one record per history name, newest 5 million keys). Delete the record to start over. Rows not delivered (for example after hitting the spend limit) are not remembered.
• Datasets: with limited permissions the run can read only the datasets selected in the Datasets / lookup dataset fields (or passed by ID in the API input). A dataset that cannot be read is reported in the run summary and the other sources are still merged.
• Files are downloaded once (up to 100 MB each) and parsed in memory. CSV values stay text (no type guessing, so IDs, postcodes and phone numbers keep their leading zeros); XLSX reads the first visible sheet with typed values and ISO dates. Private and internal network addresses are refused at every redirect hop.
• Joins hold the lookup side in memory (up to 500,000 rows; the first row per key wins). Lookup fields fill only fields that are missing or empty in the row; use a prefix to keep both.
• When a run delivers nothing (no sources, empty sources, all duplicates or nothing new), it adds one free row with _status and _message explaining why and still succeeds.

Related ActorForge actors

• Word, Excel & PowerPoint to Markdown — turn spreadsheets and documents into JSON or Markdown
• Website Change Monitor — alerts when a page changes

Changelog

• 2026-10-23 — Initial release: merge Apify datasets, CSV/TSV/JSON/JSONL/XLSX URLs and pasted rows; normalised multi-field dedupe; only new items across runs; left/inner join; flatten; field selection; source tag.

---

Built and maintained by ActorForge. Reliability first: every actor in the portfolio is monitored by an automated nightly self-test, and breakages are typically fixed the same day.