# Dataset Merge, Dedupe & Transform (datasets, JSON, CSV) (`goldish_boss/dataset-merge-dedupe`) Actor

Merge Apify datasets and JSON/CSV/JSONL URLs into one clean dataset. Dedupe by any fields, filter rows, pick, drop or rename fields. Pay only per row written.

- **URL**: https://apify.com/goldish_boss/dataset-merge-dedupe.md
- **Developed by:** [frank zhang](https://apify.com/goldish_boss) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 row outputs

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Merge, Dedupe & Transform

Combine several Apify datasets and JSON / CSV / JSONL files into **one clean dataset**: merge, remove duplicates by the fields you choose, filter rows, pick / drop / rename fields — in one run, with no code.

### What it does

1. **Reads** any mix of Apify dataset IDs (or names) and public `http(s)` URLs to JSON, JSONL or CSV files (CSV delimiter auto-detected: `,` `;` tab `|`).
2. **Filters** rows with simple rules (`eq`, `contains`, `gt`, `regex`, `in`, `exists`, …), combined with AND or OR.
3. **Dedupes** by one or more fields (dot paths like `address.city` work), or by the whole item. Keeps the first or last occurrence. Values can be normalised (trim + lower-case) so `Ana@Example.com ` = `ana@example.com`.
4. **Transforms**: pick fields, omit fields, rename fields (nested → flat supported), optionally add `_source`.
5. **Writes** the result to the run's default dataset (or a named dataset in your account, created if missing and appended to if it exists) and a `SUMMARY` record in the key-value store.

Typical uses: combine runs of the same scraper, dedupe leads across sources, clean a CSV export before import, trim a big dataset to the columns you need.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `datasetIds` | dataset picker (array) | – | Pick datasets from your account, or paste IDs / `username~dataset-name`. The run gets read-only access to exactly these datasets |
| `sourceUrls` | request list | – | Public JSON / JSONL / CSV URLs |
| `csvDelimiter` | string | auto | Force a CSV delimiter |
| `dedupeMode` | `fields` / `fullItem` / `none` | `fields` | |
| `dedupeFields` | array | – | Required when mode = `fields` |
| `keep` | `first` / `last` | `first` | `last` holds unique rows in memory |
| `normalizeDedupeValues` | boolean | `true` | Trim + lower-case before comparing |
| `filters` | array of `{field, op, value, caseSensitive}` | – | Ops: eq, neq, contains, not_contains, starts_with, ends_with, gt, gte, lt, lte, exists, not_exists, empty, not_empty, regex, in, not_in |
| `filterMode` | `all` / `any` | `all` | AND / OR |
| `pickFields` / `omitFields` | array | – | Dot paths allowed |
| `renameFields` | object | – | e.g. `{"address.city": "city"}` |
| `addSourceField` | boolean | `false` | Adds `_source` |
| `maxItems` | integer | `0` (no limit) | Stop after N output rows (reading stops too, so no extra scan charges; with `keep: last` all rows are read first) |
| `outputDatasetName` | string | – | Write to a named dataset instead of the default |
| `maxSourceMb` | integer | `200` | Per-URL download limit |

Example:

```json
{
  "datasetIds": ["crm-export"],
  "sourceUrls": [{"url": "https://example.com/leads.csv"}, {"url": "https://example.com/more.json"}],
  "dedupeFields": ["email"],
  "filters": [{"field": "age", "op": "gte", "value": 18}],
  "pickFields": ["name", "email", "age", "address.city"],
  "renameFields": {"address.city": "city"},
  "addSourceField": true
}
```

### Output example

```json
[
  {"name": "Ivy Tan", "email": "ivy@example.com", "age": 45, "city": "Montreal", "_source": "crm-export"},
  {"name": "Cleo Park", "email": "cleo@example.com", "age": 29, "city": "Lyon", "_source": "https://example.com/more.json"}
]
```

`SUMMARY` (key-value store): `itemsRead`, `filteredOut`, `duplicatesRemoved`, `itemsOutput`, `sourceErrors`, `stoppedEarly`, `outputDataset`, `chargedEvents`, and a per-source status list.

### Why pay instead of the free dedup-datasets?

[lukaskrivka/dedup-datasets](https://apify.com/lukaskrivka/dedup-datasets) is free (you pay only platform usage) and excellent for merging and deduping Apify datasets at very large scale. If that is all you need, use it. Pay for this Actor only if you need one of these:

- **CSV, JSON and JSONL files from URLs**, not just Apify datasets. Mix both in one run.
- **No-code filtering and reshaping**: 17 filter operators, plus pick, drop and rename of fields (dot paths work). The free Actor needs JavaScript transform functions for this.
- **Forgiving match rules out of the box**: trims spaces, ignores case and treats `1` and `1.0` as equal when comparing dedupe fields. The free Actor compares exact values (normalising needs your own JavaScript).
- **Keep the last duplicate instead of the first** (`keep: last`), e.g. to keep the newest version of each URL. The free Actor keeps the first.

Neither Actor does fuzzy (similarity) matching. The free Actor also has things this one does not: loading all runs of an Actor or Task, output to key-value store records, outputting only the duplicates, and an "only new items" filter dataset.

### Pricing (pay per event)

| Event | Price | When |
|---|---|---|
| Row output | **$0.0004** per row ($0.40 per 1,000) | Each row written to the output dataset |
| Rows scanned | $0.001 per started 1,000 rows read | Covers reading large inputs that produce few rows |
| Actor start | $0.00005 | Once per run (up to 1 GB memory; +$0.00005 per extra GB) |

### Cost examples

Exact charges at the prices above (Actor start $0.00005 per run at up to 1 GB memory; +$0.00005 per extra GB):

| Job | Rows written | Started 1,000-row blocks read | Total |
|---|---|---|---|
| Scan 100,000 rows, write 10,000 unique | 10,000 × $0.0004 = $4.00 | 100 × $0.001 = $0.10 | **$4.10005** (incl. $0.00005 start) |
| Scan 100,000 rows, write 100,000 (no duplicates) | 100,000 × $0.0004 = $40.00 | 100 × $0.001 = $0.10 | **$40.10005** |
| Merge 10,000 rows into 6,000 unique | $2.40 | 10 × $0.001 = $0.01 | **$2.41005** |
| Scan 1,000,000 rows, keep 5,000 | $2.00 | 1,000 × $0.001 = $1.00 | **$3.00005** |
| Scan 250 rows, write 45 | $0.018 | 1 × $0.001 | **$0.01905** |
| Scan 250 rows, write all 250 | $0.10 | 1 × $0.001 | **$0.10105** |

Run compute is included in these prices (no separate platform-usage charge). Set **Max total charge** on the run to cap spend; the Actor stops cleanly when the cap is reached.

### Limits

- `keep: first` streams rows out and keeps only a 32-character hash per unique key in memory (roughly 100–150 MB per 1M unique keys).
- `keep: last` must hold every unique row in memory until the end; for millions of rows use `keep: first` or raise memory (max 4 GB).
- Remote files are downloaded whole (default limit 200 MB each); use more memory for very large files.
- Only `http(s)` URLs. Private URLs that need auth headers are not supported; load those into an Apify dataset first.
- CSV values stay strings; numeric filters (`gt`, `lt`, …) compare numeric strings as numbers.
- One failing source does not fail the run; it is listed in `SUMMARY.sources` with the reason (e.g. `dataset not found or not accessible`, `HTTP Error 404`). The run fails only if every source fails.

# Actor input Schema

## `datasetIds` (type: `array`):

Dataset IDs or `username~dataset-name` of datasets you can read. Items are read in this order, then the URLs below. Pick datasets from your account (read-only access is granted to this run only).

## `sourceUrls` (type: `array`):

Public http(s) URLs of JSON arrays, JSON Lines or CSV files. Format is detected from the extension, the Content-Type, or the first bytes. JSON objects that wrap a list in `items`, `data`, `results` or `records` are unwrapped.

## `csvDelimiter` (type: `string`):

Leave empty to auto-detect (`,` `;` tab or `|`).

## `dedupeMode` (type: `string`):

`fields`: rows with the same values in the fields below are duplicates. `fullItem`: only identical rows are duplicates. `none`: keep everything (merge only).

## `dedupeFields` (type: `array`):

Field names, dot paths allowed (e.g. `url`, `address.city`, `phones.0`). Uses the original field names (before rename).

## `keep` (type: `string`):

`first` keeps the earliest row and streams with low memory. `last` keeps the latest row's values (in the first row's position) and holds unique rows in memory.

## `normalizeDedupeValues` (type: `boolean`):

Trim and collapse whitespace, ignore letter case, treat `1` and `1.0` as equal.

## `filters` (type: `array`):

List of `{"field", "op", "value", "caseSensitive"}`. Ops: eq, neq, contains, not_contains, starts_with, ends_with, gt, gte, lt, lte, in, not_in, regex, exists, not_exists, empty, not_empty. Numbers in strings are compared as numbers. Filters run on the original field names, before dedupe.

## `filterMode` (type: `string`):

`all` = a row must match every filter. `any` = one match is enough.

## `pickFields` (type: `array`):

Leave empty to keep every field. Dot paths allowed.

## `omitFields` (type: `array`):

Fields to remove from each row. Dot paths allowed.

## `renameFields` (type: `object`):

Map of `oldName` → `newName`, e.g. `{"title": "name", "address.city": "city"}`. Applied last.

## `addSourceField` (type: `boolean`):

Add the dataset ID or URL each row came from.

## `maxItems` (type: `integer`):

0 = no limit. Also caps your cost.

## `outputDatasetName` (type: `string`):

Optional. Write to this named dataset (kept forever) instead of the run's default dataset.

## `maxSourceMb` (type: `integer`):

Files larger than this are skipped and reported in the SUMMARY record.

## Actor input object example

```json
{
  "sourceUrls": [
    {
      "url": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
    }
  ],
  "dedupeMode": "fields",
  "dedupeFields": [],
  "keep": "first",
  "normalizeDedupeValues": true,
  "filters": [],
  "filterMode": "all",
  "pickFields": [],
  "omitFields": [],
  "renameFields": {},
  "addSourceField": false,
  "maxItems": 0,
  "maxSourceMb": 200
}
```

# Actor output Schema

## `overview` (type: `string`):

Rows after filtering, dedupe and pick/omit/rename, in the default dataset (overview table view).

## `summary` (type: `string`):

SUMMARY record: itemsRead, filteredOut, duplicatesRemoved, itemsOutput, sourceErrors, stoppedEarly, outputDataset, chargedEvents and per-source status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sourceUrls": [
        {
            "url": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
        }
    ],
    "dedupeFields": [],
    "filters": [],
    "pickFields": [],
    "omitFields": [],
    "renameFields": {}
};

// Run the Actor and wait for it to finish
const run = await client.actor("goldish_boss/dataset-merge-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sourceUrls": [{ "url": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv" }],
    "dedupeFields": [],
    "filters": [],
    "pickFields": [],
    "omitFields": [],
    "renameFields": {},
}

# Run the Actor and wait for it to finish
run = client.actor("goldish_boss/dataset-merge-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sourceUrls": [
    {
      "url": "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
    }
  ],
  "dedupeFields": [],
  "filters": [],
  "pickFields": [],
  "omitFields": [],
  "renameFields": {}
}' |
apify call goldish_boss/dataset-merge-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,goldish_boss/dataset-merge-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VImhggLWwhWaqxz3P/builds/nW5PlEEWx2CrLFiQZ/openapi.json