# Dataset Merge, Dedupe & Transform (`ensorcelling_melilot/dataset-merge`) Actor

Merge several Apify datasets into one: remove duplicates by any field, filter rows, keep, drop or rename columns, flatten nested data and sort. $0.0005 per row written.

- **URL**: https://apify.com/ensorcelling_melilot/dataset-merge.md
- **Developed by:** [ensorcelling \_melilot](https://apify.com/ensorcelling_melilot) (community)
- **Categories:** Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 row writtens

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Merge, Dedupe & Transform

Combine **several Apify datasets into one clean dataset**. Remove duplicates by any field, filter rows, keep, drop or rename columns, flatten nested data and sort. No code, no scraping. It only reads datasets you already have.

Typical jobs:

- Merge the results of many scraper runs into one list.
- Remove duplicate leads, products or jobs by `url`, `email` or `id`, across runs.
- Keep only rows that match rules (`price < 20`, `country = DE`, `email exists`).
- Make a tidy table for a spreadsheet: pick columns, rename them, flatten `address.city`.

### Why this one?

- **Simple price.** $0.0005 per row written ($0.50 per 1,000). Rows you filter out or drop as duplicates are free.
- **Doesn't crash on big inputs.** Datasets are read page by page, and duplicates are removed while streaming. Only sorting or "keep last duplicate" holds rows in memory.
- **Smart duplicate matching.** Compare chosen fields (dot paths like `company.id` work), ignoring upper/lower case and extra spaces, or compare whole rows.
- **Clear errors.** A typo in a filter operator or a bad dataset ID stops the run immediately with a plain message and no charge.

### Input

| Field | What it does |
|---|---|
| Datasets to merge | One or more datasets, read in order |
| Raw rows | Optional JSON rows added after the datasets |
| Remove duplicates / Duplicate if these fields match | Fields that identify a row; empty means whole-row comparison |
| Which duplicate to keep | First (default) or last |
| Filter rules + Rows must pass | List of `{field, operator, value}`; all or any |
| Keep only / Drop / Rename fields | Choose and rename columns |
| Flatten, trim, drop empty values | Tidy rows for spreadsheets |
| Sort by + order | Numbers sort as numbers, empty values last |
| Max rows to output | Stop after N rows |
| Add `_source` column | Which dataset each row came from |

Filter operators: `equals`, `notEquals`, `contains`, `notContains`, `startsWith`, `endsWith`, `greaterThan`, `lessThan`, `exists`, `notExists`, `matches` (regular expression). Text comparisons ignore case.

#### Example

```json
{
  "datasetIds": ["<dataset id 1>", "<dataset id 2>"],
  "dedupeBy": ["url"],
  "filters": [{ "field": "price", "operator": "lessThan", "value": 20 }],
  "keepFields": ["url", "name", "price"],
  "sortBy": "price"
}
```

### Output

One row per merged item in the run's dataset, plus a `SUMMARY` record with the number of rows read per dataset, filtered out, duplicates removed and written.

### Limits and notes

- You can only read datasets from your own Apify account.
- Sorting and "keep last duplicate" support up to 1,000,000 rows per run.
- Fields are matched exactly (`Price` and `price` are different columns). Nested fields use dots.
- Only a field's value is compared for duplicates, not its type: `5` and `"5"` count as the same.

# Actor input Schema

## `datasetIds` (type: `array`):

The datasets to combine, in order. Pick them from your account (or, when you use this Actor as an integration, set it to {{resource.defaultDatasetId}}). Rows are read in this order, so with 'keep first duplicate' the earlier dataset wins.

## `items` (type: `array`):

Optional: paste rows as a JSON array of objects. They are added after the datasets. Handy for quick tests.

## `removeDuplicates` (type: `boolean`):

Drop rows that repeat. If you list fields below, rows with the same values in those fields count as duplicates. With no fields, two rows are duplicates only if every field is identical. Turned on automatically when you list fields.

## `dedupeBy` (type: `array`):

Field names that identify a row, for example url or email or company.id. Nested fields use dots (address.city). Leave empty to compare whole rows.

## `keepWhichDuplicate` (type: `string`):

Keep the first row seen (default) or the last one, for example when newer datasets should overwrite older ones. 'Last' holds rows in memory, up to 1 million rows.

## `dedupeIgnoreCase` (type: `boolean`):

Treat Alice@Example.com and alice@example.com as the same. Spaces at the start, end and doubled spaces are always ignored when comparing.

## `filters` (type: `array`):

Keep only rows that pass these rules. Each rule is an object with field, operator and value, for example {"field": "price", "operator": "lessThan", "value": 20}. Operators: equals, notEquals, contains, notContains, startsWith, endsWith, greaterThan, lessThan, exists, notExists, matches (regular expression). Text comparisons ignore case.

## `filterMode` (type: `string`):

All rules, or any one rule.

## `keepFields` (type: `array`):

Output only these columns, in this order. Nested fields use dots. Leave empty to keep everything.

## `omitFields` (type: `array`):

Remove these columns from every row.

## `renameFields` (type: `object`):

Map old names to new names, for example {"pageUrl": "url", "companyName": "company"}. Renaming happens last, so use the old names in the other settings.

## `fillMissingFields` (type: `boolean`):

When you keep specific fields, rows that lack one get null for it so every row has the same columns.

## `flatten` (type: `boolean`):

Turn {"address": {"city": "Paris"}} into a column named address.city, handy for spreadsheets.

## `flattenArrays` (type: `boolean`):

With flattening on, turn lists into numbered columns like tags.0, tags.1.

## `trimStrings` (type: `boolean`):

Remove spaces at the start and end of every text value.

## `dropEmptyFields` (type: `boolean`):

Remove fields that are null, empty text or empty lists from each row.

## `sortBy` (type: `string`):

Sort the output by this field. Numbers sort as numbers, text alphabetically, empty values last. Holds rows in memory, up to 1 million rows. Leave empty to keep the original order.

## `sortOrder` (type: `string`):

Ascending (A to Z, small to large) or descending.

## `maxItems` (type: `integer`):

Stop after this many output rows. You are charged only for rows written. Leave empty for no limit.

## `addSourceColumn` (type: `boolean`):

Adds the ID of the dataset each row came from.

## Actor input object example

```json
{
  "removeDuplicates": false,
  "keepWhichDuplicate": "first",
  "dedupeIgnoreCase": true,
  "filters": [],
  "filterMode": "all",
  "fillMissingFields": true,
  "flatten": false,
  "flattenArrays": false,
  "trimStrings": false,
  "dropEmptyFields": false,
  "sortOrder": "asc",
  "addSourceColumn": false
}
```

# Actor output Schema

## `rows` (type: `string`):

The cleaned, merged rows as a table.

## `allFields` (type: `string`):

Every output row with all fields.

## `summary` (type: `string`):

Rows read per dataset, filtered out, duplicates removed and rows written.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "filters": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("ensorcelling_melilot/dataset-merge").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "filters": [] }

# Run the Actor and wait for it to finish
run = client.actor("ensorcelling_melilot/dataset-merge").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "filters": []
}' |
apify call ensorcelling_melilot/dataset-merge --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ensorcelling_melilot/dataset-merge"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vdfi0i2lXQnqll6vS/builds/J88gChUa8kAc7g99R/openapi.json