Merge & Dedup Datasets, CSV, JSON & Excel Files

Combine the results of several scraper runs, lead lists or exports into one clean dataset. Give it your Apify datasets, links to CSV, JSON, JSON Lines or Excel files, or rows pasted as JSON; it merges them, removes the duplicates on the fields you choose, filters, keeps only the fields you want under the names you want, and writes the result to the run's dataset, from which Apify exports CSV, Excel, JSON or XML.

No code to write: every step is an input field. You pay for the rows written; duplicates, filtered rows and failed sources are not charged as written rows. Rows read from Apify datasets you pick cost $0.001 per started 1,000 (see Pricing).

What you get

• Merge any mix of sources in one run: up to 100 Apify datasets, up to 100 file URLs, and inline rows. Duplicates are found across all of them.
• Deduplicate on one or more fields (dotted paths such as address.city work), or on whole rows (["*"]):
  • forgiving matching, each switchable: surrounding spaces (on by default), upper and lower case (on by default), runs of spaces, and URL variants (http/https, www., trailing slash, #fragment, utm_*, gclid, fbclid and other tracking parameters, query order). 5, "5" and 5.0 are the same value;
  • keep the first, the last, or the most complete duplicate (the one with the most non-empty fields);
  • rows whose dedup fields are all empty are kept, treated as one duplicate group, or dropped: your choice.
• Filter with 16 operators (equals, contains, starts/ends with, in a list, greater/less than, regular expression, empty/not empty, and their negatives), combined with AND or OR. Numbers are read from text such as "$1,234.50".
• Shape the output: write only some fields (in your order), drop fields, rename fields, and add a field saying which source each row came from.
• A SUMMARY for every run: per source, the rows read, filtered out, removed as duplicates and written, and why a source failed.
• Flat memory on large inputs: rows are streamed and written in batches of 1,000; deduplication keeps only a 16-byte fingerprint per distinct key, not the rows. Datasets are read 5,000 rows at a time, and when you pick output fields only those fields are downloaded.

Failed sources are reported in the SUMMARY with a reason and write nothing: dataset-not-found, dataset-not-readable, dataset-read-failed, fetch-failed, robots-disallowed, file-too-large, unsupported-file, invalid-json, invalid-csv, invalid-xlsx. A file is parsed completely before any of its rows is used, so a damaged file costs nothing.

Respectful by design

• Your datasets are read only if you pick them. The Actor runs with Apify's limited permissions: it can read exactly the datasets you choose in the input, for reading, and nothing else in your account.
• Only the files you link are downloaded, one request each. The site's robots.txt is read first and obeyed, and downloads from one site are spaced (500 ms by default). No proxy, no browser, no login.
• Only public internet addresses are fetched: a URL or redirect into a private, loopback or link-local network is refused.
• Use files and data you have the right to process.

Input

Field	Type	Default	Notes
datasetIds	dataset[]		your Apify datasets (picker)
fileUrls	string[]		links to .csv/.tsv, JSON array, JSON Lines, .xlsx (first sheet) files
items	array		rows as JSON objects; other values become {"value": ...}
dedupFields	string[]		fields that together identify a duplicate; ["*"] = whole row; empty = merge only
keep	first | last | mostComplete	first	which duplicate survives; kept rows stay in reading order
trimValues	boolean	true	ignore leading and trailing spaces
ignoreCase	boolean	true	ignore upper and lower case
collapseWhitespace	boolean	false	treat runs of spaces, tabs and line breaks as one space
normalizeUrls	boolean	false	compare URL values loosely (above)
emptyKeyRows	keep | dedup | drop	keep	rows whose dedup fields are all empty
filters	array		[{"field": "country", "op": "in", "value": ["US", "CA"]}, {"field": "score", "op": "gte", "value": 50}]
filterMode	all | any	all	AND or OR
filtersCaseSensitive	boolean	false
fields	string[]		write only these fields, in this order; a missing one is written as null
omitFields	string[]		top-level fields to leave out
renameFields	object		{"old name": "new name"}, applied last
addSourceField	string		add a field with each row's source (dataset:<id>, the file URL, or items)
maxOutputRows	integer	0	stop after this many rows (0 = no limit)
fileFormat	auto | csv | json | jsonl | xlsx	auto	detected from the bytes, file name and content type
csvDelimiter	auto | comma | semicolon | tab | pipe	auto	detected from the header row
maxFileSizeMb	integer	50	1-200
requestDelayMs	integer	500	pause between downloads from one site

The order of work is: read (datasets, then files, then items, each in input order), filter, deduplicate, then pick, drop and rename. Filters and dedup fields use the original field names. CSV values stay text; text in UTF-8, UTF-16 (with a BOM) and Windows-1252 is read. Excel cells give their saved values, dates as ISO 8601 text.

Output

Each row written is one dataset item, exactly as read (plus your shaping); nothing is wrapped around it. The run's key-value store holds SUMMARY:

{
  "rowsRead": 9, "rowsFiltered": 0, "duplicatesRemoved": 3, "emptyKeyRowsDropped": 0,
  "rowsWritten": 6, "rowsNotWritten": 0, "distinctKeys": 6, "sourcesRequested": 2, "sourcesFailed": 0,
  "truncated": false, "stoppedReason": null, "dedupFields": ["email"], "keep": "mostComplete",
  "sources": [
    {"source": "https://.../leads-crm.csv", "type": "file", "status": "ok", "format": "csv", "encoding": "utf-8",
     "rowsRead": 5, "rowsFiltered": 0, "duplicatesRemoved": 1, "emptyKeyRowsDropped": 0, "rowsWritten": 4,
     "rowsNotWritten": 0, "error": null}
  ]
}

A source's status is ok, failed (nothing read), partial (a dataset whose reading failed part way; the rows before it were used), stopped (the run stopped while reading it) or skipped (the run stopped before it).

Pricing

Pay per event, no start fee:

Event	Price	Per 1,000
row-written (one row written to the output dataset)	$0.0002	$0.20
dataset-rows-read (each started 1,000 rows read from Apify datasets you picked, counted over the run)	$0.001	$0.001 ($1 per million)

Rows filtered out, removed as duplicates, or from a source that failed are not charged as written. Rows read from your Apify datasets carry the small read charge whether or not they are kept, because the platform bills every dataset row read; rows from file URLs and pasted items carry none. Merging 50,000 rows from file URLs into 35,000 unique ones costs $7.00; reading the same 50,000 rows from datasets adds $0.05 (50 blocks of 1,000). When you set a maximum charge for a run, the Actor reads and writes only what it covers and stops; the SUMMARY says "truncated": true and why.

Limits

• With keep set to last or mostComplete, every source is read before the first row is written (the rows wait in a temporary file on the run's disk), so very large inputs need a longer run timeout. The run stops 30 seconds before its timeout (90 s while still reading in those modes) and saves the SUMMARY; rows it could not write are counted in rowsNotWritten and not charged.
• A file is held in memory while it is read: allow about 4 times its size (a 50 MB CSV needs about 200 MB on top of the default 1 GB run's other needs). Datasets have no such limit.
• Memory grows with the number of distinct keys: about 80 bytes each when keeping the first, 120 when keeping the last or most complete (measured on 1 million keys), so 1 million distinct keys need about 80-120 MB.
• Legacy .xls, Google Sheets links and compressed (.gz, .zip) files are not read.
• Numbers in text are read with , as a thousands separator.

Licences

Python's standard library, the Apify SDK (Apache-2.0), and openpyxl with et-xmlfile (both MIT) for Excel files. No third-party API: nothing leaves the Actor except the downloads you asked for and the reads of the datasets you picked.