Dataset Merge, Dedup and Convert (CSV, JSON, XLSX)

Dataset Merge, Dedup and Convert (CSV, JSON, XLSX)

Merges CSV, JSON, JSONL and XLSX files (and Apify datasets) into one table, removes duplicate rows on the columns you choose, and saves the result as one CSV, JSON or XLSX file. It does not scrape anything, does not clean or reformat values, and does not fix files that are not tables.

Why use this actor

• Clear errors instead of silent failures. A file that cannot be downloaded or parsed is skipped, named in the run summary with the reason (HTTP status, wrong encoding, sheet not found, not an array of objects) and the rest of the run carries on. A run where nothing can be read, or where no data rows remain, fails with a message saying what to fix. A run never ends as a "success" with an empty result.
• Real dedup controls. Choose the columns that identify a duplicate (or leave empty to compare whole rows), ignore upper/lower case, ignore leading and trailing spaces, and keep the first or the last occurrence. Values are compared as text, so 24.7 in a CSV matches 24.7 in a JSON number or an Excel cell. Rows whose dedup columns are all empty are kept.
• Mixed formats in one run. Comma, semicolon, tab or pipe separated CSV (detected automatically), JSON arrays, JSONL, and XLSX (first sheet or a named one). Encodings UTF-8 (with or without BOM) and Windows-1252 are handled; anything else gets a clear error.
• Predictable cost. One event per output row. Removed duplicates are not billed, and the run's maximum cost is respected before any work starts.

Input

Field	Type	Description
sources	array of URLs	CSV, JSON (array of objects), JSONL or XLSX files, merged in the order given. Up to 50 files, 50 MB each (XLSX 20 MB each), 60 MB in total. The type comes from the file extension, or from the content when there is none. Links must be reachable without a login.
datasetIds	array of datasets	Optional. Apify datasets picked from the account that starts the run, merged in after the files. Up to 20. The actor gets read-only access to the datasets picked here and to nothing else.
dedupFields	array of text	Column names that identify a duplicate, for example email. Empty means a row is a duplicate only when every column matches.
caseInsensitive	boolean	Compare dedup values ignoring letter case. The output keeps the original text. Default false.
trim	boolean	Compare dedup values after trimming spaces. The output keeps the original text. Default true.
keep	first or last	Which row survives among duplicates. Default first.
columns	union or intersection	Union keeps every column found in any source (missing values stay empty). Intersection keeps only the columns present in every source. Default union.
outputFormat	csv, json or xlsx	Format of the file saved as OUTPUT.csv, OUTPUT.json or OUTPUT.xlsx. Default csv.
maxRows	integer	Upper bound on output rows, 1 to 500,000. Default 500,000.
sheetName	text	Sheet to read from XLSX files. Empty means the first sheet.

Column names are matched exactly (after trimming), so Email and email are two different columns. Nested JSON values (objects and arrays) are stored as JSON text in a single cell.

Output

The full merged result is the output file in the run's key-value store: OUTPUT.csv, OUTPUT.json or OUTPUT.xlsx, depending on outputFormat. It holds every output row. The run's dataset holds a preview only: the first 100 merged rows (one item per row, one field per column), for a quick look in the Console. A second view, Example columns, shows only the id, name, email, city and amount columns of the sample files, so it stays mostly empty for other data. For more than 100 rows, download the OUTPUT file; the run's output page links to it. The SUMMARY record in the key-value store holds:

Field	Description
sources	Per source: source (file name without the query string), status (ok or skipped), rows, and detail with the reason when skipped.
inputRows	Rows read from all readable sources.
duplicatesRemoved	Rows dropped as duplicates.
outputRows	Rows in the output file, which is what is billed. The dataset preview is not billed separately.
previewRows	Rows in the dataset preview: outputRows, at most 100.
uniqueRows	Distinct rows before maxRows or the cost limit cut the output.
truncated	true when the output was cut short, and the run's status message says why.
columns, outputFormat, outputKey, unitsCharged	The final column list, the file written and the events charged.

Pricing

This actor uses pay-per-event pricing. Platform usage is included in the event price.

Event	Price	Charged when
merged-row	$0.00005	Once per row delivered to the output file, after duplicates are removed. That is $0.05 per 1,000 rows.

Billing counts the rows in the full OUTPUT file, not the preview. Worked example: 25,000 rows after dedup is 25,000 events, 25,000 x $0.00005 = $1.25, even though the dataset preview shows only 100 of them. 1,200 rows is $0.06. A failed run is not charged for rows.

Apify's standard apify-actor-start event also applies to every run, at $0.00005 per GB of run memory, so $0.00005 at the default 1 GB. It is left out of the examples above.

FAQ

What happens when one of the files is broken? That file is skipped and listed in SUMMARY with the reason. The other files are still merged. If every source fails, the run fails with the reasons of the first few.

Why does the run say a column was not found? dedupFields names must match the merged column names exactly, including case. Check the columns list in SUMMARY of a previous run, or fix the header of the source file.

Can it read Excel files with several sheets? Yes. Only one sheet is read per file: the first, or the one named in sheetName (the same name for every XLSX file). Run again with another sheet name to merge more sheets.

What are the size limits? Up to 50 files and 20 datasets, 50 MB per file (20 MB for an XLSX file) and 60 MB for all files together, 600,000 input rows in total and 500,000 output rows. An XLSX output holds at most 1,048,575 data rows; larger results need CSV or JSON, and the run says so before writing anything.

Can the actor read Apify datasets? Yes, the ones picked in the datasetIds field. The actor runs with limited permissions: it can only read those datasets, never write to them, and cannot see any other storage in the account. Datasets of other accounts are not readable.

Which private files can be read? Only links that work without logging in. Signed links that carry their own access signature in the URL are fine, and the signature is never written to logs or to SUMMARY.

Disclaimer

This actor is not affiliated with, endorsed by or sponsored by Microsoft. Excel is a trademark of its respective owner, and the name is used only to describe the XLSX file format. Users are responsible for complying with the terms of the sources they process and for the lawful use of the data they merge.

Report issues

Report problems or request features in the Issues tab of this actor.