# Merge, Deduplicate & Transform Datasets (`deriverge/dataset-merge-deduplicate`) Actor

Combines datasets from your actor runs into one, without duplicates. Match rows on any fields exactly, ignoring case, as URLs without tracking parameters or by similar text, then filter rows, rename fields and flatten nested objects.

- **URL**: https://apify.com/deriverge/dataset-merge-deduplicate.md
- **Developed by:** [deriverge s.r.o.](https://apify.com/deriverge) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.15 / 1,000 rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Merge, Deduplicate & Transform Datasets

Merge, Deduplicate & Transform Datasets combines datasets from your actor runs into one, in the order you list them, and removes duplicate rows. Rows match on the fields you choose: exactly, ignoring spaces and letter case, as web addresses without tracking parameters, or by similar text. Filters, field selection, renaming and flattening run in the same pass. Pick the datasets in **Datasets**, enter a field such as `url` in **Deduplicate by fields** and click **Start**. It reads datasets only, never key-value stores or request queues. It costs $0.30 per 1,000 rows on the Free plan ($0.15 on Business), and the $5 of monthly credit in Apify's Free plan covers about 16,000 rows.

### What does it return?

Each output row is a row from your datasets, unchanged apart from the field options you set, in the order of **Datasets**. With **Add the source dataset ID** on, every row also gets `sourceDatasetId`, the ID of the dataset it came from, such as `kx6fsPfmMfEw8IxVY`.

The `SUMMARY` record in the run's key-value store, linked in the **Output** tab, describes what happened:

| Field | Description |
|---|---|
| `datasets` | One entry per dataset with `datasetId`, `rowsRead`, `filteredOut`, `duplicatesRemoved`, `rowsWithoutKey` and `outputRows`. A removed duplicate is counted in the dataset it came from. |
| `inputRows`, `outputRows` | Rows read from all datasets and rows written to the output. |
| `filteredOut`, `duplicatesRemoved` | Totals of rows left out by **Keep only rows where** and of duplicates removed (or found, with **Output** set to the duplicates). |
| `rowsWithoutKey` | Rows where every **Deduplicate by fields** value was empty. They are always kept. |
| `distinctKeys` | Number of distinct rows (groups of duplicates) seen. |
| `complete`, `stoppedBecause` | `true` when every row was read; otherwise the reason, such as **Maximum output rows**, the maximum cost per run or the run timeout. |
| `restarted` | Present when the run restarted on another server, with the number of rows that were already written before. |

### How to merge and deduplicate datasets

1. In **Datasets**, pick datasets from your storage or enter dataset IDs or names such as `username~my-dataset`, in the order you want them merged. Paste IDs: Apify refuses a dataset link in this field before the run starts.
2. In **Deduplicate by fields**, enter the fields that make two rows the same, such as `url`, or `companyName` and `title` together. Nested fields use dot notation (`company.name`), and `*` compares whole rows. Leave it empty to merge without removing anything.
3. Set **Match values** to compare text exactly, ignoring extra spaces, or ignoring spaces and letter case, and turn on **Treat URL variants as the same** for addresses. For names or titles written in different ways, enter the field in **Similar text field** and set **Similarity threshold**.
4. Choose **Keep** first or last. Optionally add conditions in **Keep only rows where**, such as `price > 100`, and pick, remove, rename or flatten fields.
5. Click **Start**. The rows appear in the **Output** tab, where you can download them as CSV, Excel, JSON or HTML. Over the API, an input looks like this:

```json
{
  "datasetIds": ["kx6fsPfmMfEw8IxVY", "2Udbt7i7aspbxBKEV"],
  "dedupeFields": ["url"],
  "normalizeUrls": true,
  "keep": "last",
  "filters": ["salaryMin > 100000", "remote = true"],
  "omitFields": ["description"]
}
```

### Example output

One row from a run on 11 October 2026 that merged two job datasets of 9,905 and 9,970 rows, kept remote jobs with `salaryMin` above 100,000, removed repeated URLs, kept seven fields, renamed `companyName` to `company` and added the source dataset:

```json
{
  "title": "Product Manager, Community Support",
  "company": "Airbnb",
  "salaryMin": 200000,
  "salaryMax": 240000,
  "salaryCurrency": "USD",
  "locations": [
    "Remote"
  ],
  "url": "https://careers.airbnb.com/positions/8189782?gh_jid=8189782",
  "sourceDatasetId": "kx6fsPfmMfEw8IxVY"
}
```

The `SUMMARY` of a run on the same two datasets deduplicated by `url` alone, shortened:

```json
{
  "datasets": [
    { "datasetId": "kx6fsPfmMfEw8IxVY", "rowsRead": 9905, "duplicatesRemoved": 8861, "outputRows": 1044 },
    { "datasetId": "2Udbt7i7aspbxBKEV", "rowsRead": 9970, "duplicatesRemoved": 8818, "outputRows": 1152 }
  ],
  "inputRows": 19875,
  "duplicatesRemoved": 17679,
  "outputRows": 2196,
  "complete": true
}
```

### How much does it cost to merge and deduplicate datasets?

| | Free plan | Starter | Scale | Business |
|---|---|---|---|---|
| 1,000 rows | $0.30 | $0.24 | $0.195 | $0.15 |

You pay only for the events in the table. There is no start fee, and compute time and proxies are included.

You pay for the rows written to the output dataset. In a test on 11 October 2026, 25 datasets with 99,375 job rows deduplicated by URL gave 2,196 rows, which costs $0.6588 on the Free plan and $0.3294 on Business. Removed duplicates, filtered rows and the rows of the built-in sample are not charged.

### How rows are matched

- With **Match values** set to exactly, text is compared as it is. Ignoring extra spaces trims the ends and treats runs of spaces as one, and ignoring spaces and letter case also makes `ACME Inc` equal `acme inc`. Numbers match their text form, so `42` and `"42"` are the same.
- With **Treat URL variants as the same** on, `https://www.example.com/jobs/123/?utm_source=x#apply` and `http://example.com/jobs/123` match: the scheme, `www.`, a trailing slash, the `#fragment`, the order of query parameters and tracking parameters (`utm_*`, `gclid`, `fbclid`, `msclkid` and 45 others) are ignored. Parameters that pick a product or page, such as `?id=5`, still count, and the path stays case-sensitive. **More URL parameters to ignore** adds your own, such as `ref`.
- A field in **Similar text field** is compared by similarity, from 0.5 to 1, ignoring letter case, accents and punctuation. The score is the share of three-letter sequences two texts have in common, so `Senior Software Engineer` and `Senior Software Engineer (Remote)` score 0.87, while `iPhone 15 Pro 128GB` and `iPhone 15 Pro 256GB` score 0.80. With **Deduplicate by fields** filled in, those fields must match exactly and only the similar field may differ: same company, similar job title.
- With **Keep** set to first, the first copy in the order of **Datasets** stays, and with last the last one stays, so listing an older run before a newer one keeps the newest version of each row. Either way the output follows the original order.

### Limits

- Speed, measured on 11 October 2026 with 1 GB of memory on job rows of about 760 bytes: 99,375 rows in 25 datasets were deduplicated by URL in 6.7 seconds for the whole run (5.1 seconds of work after the start), and in 7.6 seconds with **Keep** set to last, which reads every dataset twice. Writing is the slow part, at about 4,500 rows per second: merging all 99,375 rows without removing any took 24.5 seconds.
- Memory: deduplication keeps a short key for every distinct row, and similarity matching keeps the text of every kept row. The 99,375-row run above used at most 153 MB with 256 MB of memory. A run that comes close to its memory or its timeout stops, writes `SUMMARY` and ends as succeeded with the rows written so far; raise **Memory** or **Timeout** in Run options to finish the rest.
- Similarity can merge rows that differ in one short word. In our test data, `Junior Interior Designer` and `Senior Interior Designer` scored 0.86, and two titles ending in `Web` and `iOS` scored 0.90. Set **Output** to the duplicates first to see what a threshold removes. Texts are compared on their first 500 characters.
- Rows whose **Deduplicate by fields** values are all empty, or whose similar text is empty, are kept and counted in `rowsWithoutKey`. In one test, 4,775 job rows without `companyName` stayed when matching by company and similar title.
- Flattening turns nested objects into columns such as `company.name`; lists stay lists.
- A dataset that changed in the last 15 seconds is read once its row count settles, because the Apify API shows new rows with a delay of a few seconds.
- A dataset of another Apify account can be read only when its owner shares it with you or makes it readable by anyone with the ID.

### Using it in an integration

To clean the dataset of every run of another actor or task, open that actor or task, go to **Integrations**, add this actor and enter {{resource.defaultDatasetId}} in **Datasets** (it is listed under **Available variables**). You can put a fixed dataset before it, such as last week's merged results, and set **Keep** to last to keep the newest version of each row. The variable is required: Apify grants a run with limited permissions access to the datasets in **Datasets** only, so an integration that passes the dataset just in its default payload fails with a message that explains this fix.

### FAQ

#### Is it legal to merge and deduplicate datasets?

The actor reads only the datasets you pick in your Apify account, or datasets someone shared with you, and writes the result to the dataset of its own run. It visits no website and collects nothing new. Your right to keep and combine the data depends on where it came from, so you are responsible for using it in line with the terms of those sources and the law that applies to you, including rules on personal data.

#### Can the actor read my other datasets?

No. It runs with limited permissions, so Apify gives each run read access to the datasets in **Datasets** and to the run's own storage, nothing more. A dataset ID passed in any other field is refused, and the run status says which field to use.

#### Can I see which rows were removed?

Set **Output** to the duplicates. The run then writes the rows that the same settings would remove, which is a quick way to check a similarity threshold or a URL rule before the real run. `SUMMARY` also counts removed rows per dataset.

#### What happens if the run is moved to another server?

Apify sometimes moves a running actor to another server. The run then starts again, computes the same result and skips the rows already in its dataset. In a test, a run restarted in the middle of a 99,375-row merge ended with exactly 99,375 rows in the original order.

### Related scrapers

- [Career Site Jobs Search](https://apify.com/deriverge/career-site-jobs-search)
- [Remote Jobs Scraper](https://apify.com/deriverge/remote-jobs-scraper)
- [Career Site Jobs Scraper](https://apify.com/deriverge/greenhouse-lever-workday-jobs-scraper)

### Support

This actor is built and maintained by deriverge s.r.o., a software company based in the Czech Republic. If a run fails or a field you need is missing, please open an issue in the **Issues** tab or write to us at info@deriverge.com. We respond in English and Czech. Runs can be scheduled in Apify Console or started from the **API** tab, which has examples for Python, JavaScript and cURL and works with Make, Zapier, n8n and the Apify MCP server. If the actor saves you time, a short review helps other people find it.

# Changelog

This Actor's version history is a separate document: https://apify.com/deriverge/dataset-merge-deduplicate/changelog.md

# Actor input Schema

## `datasetIds` (type: `array`):

Datasets to merge, in this order: pick them from your storage or enter dataset IDs or names such as username~my-dataset. The actor may read only the datasets listed here. In an integration of another actor or task, enter {{resource.defaultDatasetId}} to merge the dataset of the run that finished.

## `dedupeFields` (type: `array`):

Rows with the same values in all of these fields are duplicates. Use dot notation for nested fields, such as company.name. Enter \* to compare whole rows. Leave it empty to merge without removing duplicates. Rows where all of these fields are empty are always kept.

## `matchMode` (type: `string`):

How text values are compared. Numbers match their text form, so 42 and "42" are the same.

## `normalizeUrls` (type: `boolean`):

Values that are web addresses match when they differ only in http or https, www., a trailing slash, a #fragment, the order of query parameters, or tracking parameters such as utm_source, gclid, fbclid and msclkid.

## `urlParamsToRemove` (type: `array`):

Extra query parameters that do not change the page on your sites, such as ref or source. Used with Treat URL variants as the same.

## `fuzzyField` (type: `string`):

A text field compared by similarity instead of exact match, such as title or name. With Deduplicate by fields filled in, those fields must match exactly and this one only needs to be similar: same company, similar job title.

## `fuzzyThreshold` (type: `number`):

From 0.5 to 1. At 0.85, "Senior Software Engineer" and "Senior Software Engineer (Remote)" match (0.87) while "iPhone 15 Pro 128GB" and "iPhone 15 Pro 256GB" do not (0.80). Letter case, accents and punctuation are ignored.

## `keep` (type: `string`):

Which copy stays. Datasets are read in the order of Datasets, so Last keeps the copy from the latest dataset, for example the newest run. Last reads the datasets twice.

## `outputMode` (type: `string`):

Unique rows is the merged result. Only the duplicates writes the rows that would be removed, to check what the settings do.

## `filters` (type: `array`):

One condition per line: field, condition, value. Conditions: exists, not exists, = (or equals), !=, contains, not contains, >, >=, <, <=. Examples: price > 100, title contains engineer, email exists, postedAt >= 2026-10-01. Text is compared ignoring letter case; dates in the form 2026-10-01 compare as dates.

## `filterMode` (type: `string`):

With several conditions: all of them, or at least one.

## `outputFields` (type: `array`):

Write only these fields, in this order. Nested fields such as company.name keep their nesting. Empty keeps every field.

## `omitFields` (type: `array`):

Fields left out of the output, such as long descriptions or HTML.

## `renameFields` (type: `array`):

Old field name and new field name. Over the API this can also be an object such as {"title": "jobTitle"}.

## `flatten` (type: `boolean`):

Turns {"company": {"name": "Acme"}} into {"company.name": "Acme"}, one column per value in CSV and Excel. Lists stay as lists.

## `flattenSeparator` (type: `string`):

Character between the parts of a flattened field name, such as . or \_.

## `addSourceDataset` (type: `boolean`):

Adds sourceDatasetId to every row, the dataset the row came from.

## `maxItems` (type: `integer`):

Stop after writing this many rows. Leave it empty or 0 for no limit.

## Actor input object example

```json
{
  "dedupeFields": [
    "url"
  ],
  "matchMode": "exact",
  "normalizeUrls": true,
  "fuzzyThreshold": 0.85,
  "keep": "first",
  "outputMode": "unique",
  "filterMode": "all",
  "flatten": false,
  "flattenSeparator": ".",
  "addSourceDataset": false
}
```

# Actor output Schema

## `rows` (type: `string`):

The rows of all datasets in order, without duplicates, after filters and field changes.

## `summary` (type: `string`):

Rows read, filtered out, duplicates removed and rows written for each dataset, the settings used and whether the run read everything.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupeFields": [
        "url"
    ],
    "normalizeUrls": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("deriverge/dataset-merge-deduplicate").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "dedupeFields": ["url"],
    "normalizeUrls": True,
}

# Run the Actor and wait for it to finish
run = client.actor("deriverge/dataset-merge-deduplicate").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupeFields": [
    "url"
  ],
  "normalizeUrls": true
}' |
apify call deriverge/dataset-merge-deduplicate --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,deriverge/dataset-merge-deduplicate"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/b8kC1KeUOQfH2W4cT/builds/r3l2feNU3m65ArVNT/openapi.json