# Dataset Merge, Dedup and Convert (CSV, JSON, XLSX) (`enthusiastic_ginkgo_7ie/dataset-merge-dedup`) Actor

Merge CSV, JSON, JSONL and XLSX files or Apify datasets into one table. Remove duplicates by the columns you choose, ignoring case and spaces if wanted. Get one CSV, JSON or XLSX file plus a 100-row dataset preview. Unreadable files are skipped and named in a clear summary. Pay per output row.

- **URL**: https://apify.com/enthusiastic_ginkgo_7ie/dataset-merge-dedup.md
- **Developed by:** [jeenopylo](https://apify.com/enthusiastic_ginkgo_7ie) (community)
- **Categories:** Automation, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.05 / 1,000 merged rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Dataset Merge, Dedup and Convert (CSV, JSON, XLSX)

Merges CSV, JSON, JSONL and XLSX files (and Apify datasets) into one table, removes duplicate rows on the columns you choose, and saves the result as one CSV, JSON or XLSX file. It does not scrape anything, does not clean or reformat values, and does not fix files that are not tables.

#### Why use this actor

- **Clear errors instead of silent failures.** A file that cannot be downloaded or parsed is skipped, named in the run summary with the reason (HTTP status, wrong encoding, sheet not found, not an array of objects) and the rest of the run carries on. A run where nothing can be read, or where no data rows remain, fails with a message saying what to fix. A run never ends as a "success" with an empty result.
- **Real dedup controls.** Choose the columns that identify a duplicate (or leave empty to compare whole rows), ignore upper/lower case, ignore leading and trailing spaces, and keep the first or the last occurrence. Values are compared as text, so `24.7` in a CSV matches `24.7` in a JSON number or an Excel cell. Rows whose dedup columns are all empty are kept.
- **Mixed formats in one run.** Comma, semicolon, tab or pipe separated CSV (detected automatically), JSON arrays, JSONL, and XLSX (first sheet or a named one). Encodings UTF-8 (with or without BOM) and Windows-1252 are handled; anything else gets a clear error.
- **Predictable cost.** One event per output row. Removed duplicates are not billed, and the run's maximum cost is respected before any work starts.

#### Input

| Field | Type | Description |
|---|---|---|
| `sources` | array of URLs | CSV, JSON (array of objects), JSONL or XLSX files, merged in the order given. Up to 50 files, 50 MB each (XLSX 20 MB each), 60 MB in total. The type comes from the file extension, or from the content when there is none. Links must be reachable without a login. |
| `datasetIds` | array of datasets | Optional. Apify datasets picked from the account that starts the run, merged in after the files. Up to 20. The actor gets read-only access to the datasets picked here and to nothing else. |
| `dedupFields` | array of text | Column names that identify a duplicate, for example `email`. Empty means a row is a duplicate only when every column matches. |
| `caseInsensitive` | boolean | Compare dedup values ignoring letter case. The output keeps the original text. Default `false`. |
| `trim` | boolean | Compare dedup values after trimming spaces. The output keeps the original text. Default `true`. |
| `keep` | `first` or `last` | Which row survives among duplicates. Default `first`. |
| `columns` | `union` or `intersection` | Union keeps every column found in any source (missing values stay empty). Intersection keeps only the columns present in every source. Default `union`. |
| `outputFormat` | `csv`, `json` or `xlsx` | Format of the file saved as `OUTPUT.csv`, `OUTPUT.json` or `OUTPUT.xlsx`. Default `csv`. |
| `maxRows` | integer | Upper bound on output rows, 1 to 500,000. Default 500,000. |
| `sheetName` | text | Sheet to read from XLSX files. Empty means the first sheet. |

Column names are matched exactly (after trimming), so `Email` and `email` are two different columns. Nested JSON values (objects and arrays) are stored as JSON text in a single cell.

#### Output

The full merged result is the output file in the run's key-value store: `OUTPUT.csv`, `OUTPUT.json` or `OUTPUT.xlsx`, depending on `outputFormat`. It holds every output row. The run's dataset holds a preview only: the first 100 merged rows (one item per row, one field per column), for a quick look in the Console. A second view, Example columns, shows only the `id`, `name`, `email`, `city` and `amount` columns of the sample files, so it stays mostly empty for other data. For more than 100 rows, download the `OUTPUT` file; the run's output page links to it. The `SUMMARY` record in the key-value store holds:

| Field | Description |
|---|---|
| `sources` | Per source: `source` (file name without the query string), `status` (`ok` or `skipped`), `rows`, and `detail` with the reason when skipped. |
| `inputRows` | Rows read from all readable sources. |
| `duplicatesRemoved` | Rows dropped as duplicates. |
| `outputRows` | Rows in the output file, which is what is billed. The dataset preview is not billed separately. |
| `previewRows` | Rows in the dataset preview: `outputRows`, at most 100. |
| `uniqueRows` | Distinct rows before `maxRows` or the cost limit cut the output. |
| `truncated` | `true` when the output was cut short, and the run's status message says why. |
| `columns`, `outputFormat`, `outputKey`, `unitsCharged` | The final column list, the file written and the events charged. |

#### Pricing

This actor uses pay-per-event pricing. Platform usage is included in the event price.

| Event | Price | Charged when |
|---|---|---|
| `merged-row` | $0.00005 | Once per row delivered to the output file, after duplicates are removed. That is $0.05 per 1,000 rows. |

Billing counts the rows in the full `OUTPUT` file, not the preview. Worked example: 25,000 rows after dedup is 25,000 events, 25,000 x $0.00005 = $1.25, even though the dataset preview shows only 100 of them. 1,200 rows is $0.06. A failed run is not charged for rows.

Apify's standard `apify-actor-start` event also applies to every run, at $0.00005 per GB of run memory, so $0.00005 at the default 1 GB. It is left out of the examples above.

#### FAQ

**What happens when one of the files is broken?**
That file is skipped and listed in `SUMMARY` with the reason. The other files are still merged. If every source fails, the run fails with the reasons of the first few.

**Why does the run say a column was not found?**
`dedupFields` names must match the merged column names exactly, including case. Check the `columns` list in `SUMMARY` of a previous run, or fix the header of the source file.

**Can it read Excel files with several sheets?**
Yes. Only one sheet is read per file: the first, or the one named in `sheetName` (the same name for every XLSX file). Run again with another sheet name to merge more sheets.

**What are the size limits?**
Up to 50 files and 20 datasets, 50 MB per file (20 MB for an XLSX file) and 60 MB for all files together, 600,000 input rows in total and 500,000 output rows. An XLSX output holds at most 1,048,575 data rows; larger results need CSV or JSON, and the run says so before writing anything.

**Can the actor read Apify datasets?**
Yes, the ones picked in the `datasetIds` field. The actor runs with limited permissions: it can only read those datasets, never write to them, and cannot see any other storage in the account. Datasets of other accounts are not readable.

**Which private files can be read?**
Only links that work without logging in. Signed links that carry their own access signature in the URL are fine, and the signature is never written to logs or to `SUMMARY`.

#### Disclaimer

This actor is not affiliated with, endorsed by or sponsored by Microsoft. Excel is a trademark of its respective owner, and the name is used only to describe the XLSX file format. Users are responsible for complying with the terms of the sources they process and for the lawful use of the data they merge.

#### Report issues

Report problems or request features in the Issues tab of this actor.

# Actor input Schema

## `sources` (type: `array`):

Links to CSV, JSON (array of objects), JSONL or XLSX files, in the order they should be merged. Public links, or links that carry their own access signature. The file type comes from the extension, or from the content when there is none. The prefill is five synthetic sample files.

## `datasetIds` (type: `array`):

Optional. Datasets to merge in after the files. Pick datasets from your own account.

## `dedupFields` (type: `array`):

Column names that identify a duplicate, for example email. Leave empty to treat a row as a duplicate only when every column matches. A row whose dedup fields are all empty is always kept.

## `caseInsensitive` (type: `boolean`):

Compare dedup values without regard to letter case. The output keeps the original text.

## `trim` (type: `boolean`):

Compare dedup values after trimming spaces. The output keeps the original text.

## `keep` (type: `string`):

Which row survives when several rows are duplicates: the first one seen, or the last one.

## `columns` (type: `string`):

Union keeps every column found in any source (missing values stay empty). Intersection keeps only the columns present in every source.

## `outputFormat` (type: `string`):

Format of the full merged file saved as OUTPUT.csv, OUTPUT.json or OUTPUT.xlsx in the run's key-value store. The dataset holds only a preview of the first 100 rows.

## `maxRows` (type: `integer`):

Upper bound on merged rows (the first rows in order are kept). Billing is per output row.

## `sheetName` (type: `string`):

Sheet to read from XLSX files. Leave empty to read the first sheet.

## Actor input object example

```json
{
  "sources": [
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-a.csv?signature=c2fjNY6K0S5nDDUM0aSk",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-b-semicolon.csv?signature=owJd10EzKDS9vqJ9JbBx",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-c.json?signature=49kUyk1G8O46Kt3IlXdZ",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-d.jsonl?signature=1tJaeq09Aiq0fPzXZEx7p",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-e.xlsx?signature=1Ah1Wuo9fjfYaIZdPtVjJ"
  ],
  "dedupFields": [
    "email"
  ],
  "caseInsensitive": true,
  "trim": true,
  "keep": "first",
  "columns": "union",
  "outputFormat": "csv",
  "maxRows": 500000
}
```

# Actor output Schema

## `files` (type: `string`):

No description

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sources": [
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-a.csv?signature=c2fjNY6K0S5nDDUM0aSk",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-b-semicolon.csv?signature=owJd10EzKDS9vqJ9JbBx",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-c.json?signature=49kUyk1G8O46Kt3IlXdZ",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-d.jsonl?signature=1tJaeq09Aiq0fPzXZEx7p",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-e.xlsx?signature=1Ah1Wuo9fjfYaIZdPtVjJ"
    ],
    "dedupFields": [
        "email"
    ],
    "caseInsensitive": true,
    "trim": true,
    "keep": "first",
    "columns": "union",
    "outputFormat": "csv",
    "maxRows": 500000
};

// Run the Actor and wait for it to finish
const run = await client.actor("enthusiastic_ginkgo_7ie/dataset-merge-dedup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sources": [
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-a.csv?signature=c2fjNY6K0S5nDDUM0aSk",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-b-semicolon.csv?signature=owJd10EzKDS9vqJ9JbBx",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-c.json?signature=49kUyk1G8O46Kt3IlXdZ",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-d.jsonl?signature=1tJaeq09Aiq0fPzXZEx7p",
        "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-e.xlsx?signature=1Ah1Wuo9fjfYaIZdPtVjJ",
    ],
    "dedupFields": ["email"],
    "caseInsensitive": True,
    "trim": True,
    "keep": "first",
    "columns": "union",
    "outputFormat": "csv",
    "maxRows": 500000,
}

# Run the Actor and wait for it to finish
run = client.actor("enthusiastic_ginkgo_7ie/dataset-merge-dedup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sources": [
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-a.csv?signature=c2fjNY6K0S5nDDUM0aSk",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-b-semicolon.csv?signature=owJd10EzKDS9vqJ9JbBx",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-c.json?signature=49kUyk1G8O46Kt3IlXdZ",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-d.jsonl?signature=1tJaeq09Aiq0fPzXZEx7p",
    "https://api.apify.com/v2/key-value-stores/rKmTb0uCShh9ssnDK/records/u2-people-e.xlsx?signature=1Ah1Wuo9fjfYaIZdPtVjJ"
  ],
  "dedupFields": [
    "email"
  ],
  "caseInsensitive": true,
  "trim": true,
  "keep": "first",
  "columns": "union",
  "outputFormat": "csv",
  "maxRows": 500000
}' |
apify call enthusiastic_ginkgo_7ie/dataset-merge-dedup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,enthusiastic_ginkgo_7ie/dataset-merge-dedup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/faQx3HcRsjRlMoBpM/builds/JMguEj3XDizL1vzy3/openapi.json