# Dataset Merge, Dedup & Transform – Clean Data (`innovative_cookie/dataset-merge-dedup`) Actor

Merge several Apify datasets, remove duplicates by any field, filter, rename, pick or flatten columns, and write one clean dataset. Limited permissions.

- **URL**: https://apify.com/innovative_cookie/dataset-merge-dedup.md
- **Developed by:** [Innovative Cookie](https://apify.com/innovative_cookie) (community)
- **Categories:** Integrations, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Dataset Merge, Dedup & Transform do?

**Dataset Merge, Dedup & Transform** combines the results of several Apify runs into **one clean dataset**. It **merges datasets**, **removes duplicates** by one or more fields, **filters** items with simple conditions, and **picks, removes, renames or flattens** columns, all in one run.

- 🧩 **Merge** any number of datasets, plus items pasted as JSON.
- 🧹 **Deduplicate** by one or more fields (`url`, `id`, `name + city`…), ignoring letter case and extra spaces, or remove exact duplicates.
- 🔀 Keep the **first**, the **last** or a **merged** version of each duplicate (empty fields filled from the other copies).
- 🔎 **Filter** with conditions such as `price < 20`, `title contains "shoe"`, `date >= 2026-01-01`, `email is empty`.
- ✂️ **Transform**: keep only some fields, drop others, rename them, flatten nested objects for CSV and Google Sheets.
- 🔒 **Limited permissions**: the Actor only gets **READ** access to the datasets you pick. It cannot see anything else in your account.

### Why use Dataset Merge, Dedup & Transform?

- 🗂️ **Combine scraper runs.** You ran the same scraper on 10 search terms or 5 cities: get one deduplicated list.
- 📈 **Incremental scraping.** Merge yesterday's and today's results and keep the latest version of each item (`keep: last`).
- 🧾 **Prepare exports.** Rename and flatten fields before sending data to Google Sheets, Airtable, a CRM or a database.
- 🤖 **Pipelines and AI agents.** Call it after any Actor (webhook, Make, Zapier, n8n, MCP) to clean the output before the next step.

### How to merge and deduplicate datasets

1. Click **Try for free**.
2. In **Datasets to merge**, pick the datasets of your previous runs (or paste items in **Items (JSON)**).
3. In **Deduplicate by fields**, enter the field that identifies an item, for example `url`.
4. Optionally add **Filters**, choose fields to keep, remove or rename, and turn on **Flatten nested objects**.
5. Click **Start** and download the clean dataset as **CSV, Excel or JSON**.

### Input

The default input deduplicates a few sample items, so you can see the result in seconds:

```json
{
    "datasetIds": ["<your dataset ID>"],
    "dedupFields": ["url"],
    "caseInsensitive": true,
    "keep": "first",
    "filters": [{ "field": "price", "op": "lt", "value": 53 }],
    "omitFields": ["stock.available"],
    "renameFields": { "title": "name" },
    "flattenNested": true
}
```

#### Filter operators

`eq`, `neq`, `contains`, `notContains`, `startsWith`, `endsWith`, `regex`, `gt`, `gte`, `lt`, `lte` (numbers or dates), `exists`, `notExists`, `empty`, `notEmpty`, `in`, `notIn` (value is an array). Text comparisons ignore case unless you add `"caseInsensitive": false` to the condition. All conditions must match.

#### Order of operations

1. Read inline items, then each dataset in the order given.
2. Apply **filters** (on the original fields).
3. **Deduplicate** (on the original fields).
4. **Pick / remove** fields, **flatten**, then **rename**.

### Output

Your items, cleaned. For example, with `flattenNested` and `renameFields: {"stock.count": "stock_count"}`:

```json
[
    {
        "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "title": "A Light in the Attic",
        "price": 51.77,
        "stock.available": true,
        "stock_count": 22
    }
]
```

The **`SUMMARY`** record in the key-value store lists, for each source, the items read, filtered out and removed as duplicates, and the total written.

### How much does it cost?

The Actor is **free to use** for now: there is no Actor fee, you only pay the Apify platform usage, which is very small (no browser, no proxy, no web requests). Deduplication with `keep: first` streams the data with little memory. `keep: last` and `keep: merge` hold the kept items in memory: for several million items, raise the run memory.

### Integrations

Chain it after any Actor with a **webhook** or an **Actor-to-Actor integration**, or call it from **Make**, **Zapier**, **n8n**, the **API** or an AI agent through the Apify **MCP server**. Then send the clean dataset to Google Sheets, Airtable, Slack or your database.

### FAQ

#### How do I remove duplicates from an Apify dataset?

Pick the dataset, enter the field that identifies an item (for example `url`) in **Deduplicate by fields**, and start the run. Every item after the first one with the same value is removed.

#### Can I deduplicate on several fields?

Yes. Items are duplicates only if all the listed fields are equal, for example `name` and `address.city`. Items whose key fields are all empty are always kept.

#### Why does the Actor need my datasets?

It reads only the datasets you pick in the input, with **READ** permission. It runs with Apify's **limited permissions**, so it cannot access any other data in your account and it never changes your datasets.

#### Can I write to an existing dataset?

The result is written to the run's default dataset, which you can export or pass to the next Actor.

#### Something does not work?

Open an issue in the **Issues** tab with your input (without private data) and the `SUMMARY` record.

### Related actors

More Actors by the same author ([innovative_cookie on Apify](https://apify.com/innovative_cookie)):

- [Sitemap URL Extractor](https://apify.com/innovative_cookie/sitemap-extractor): get every URL of a site from its XML sitemaps.
- [Broken Link Checker](https://apify.com/innovative_cookie/broken-link-checker): find 404s and dead links on any website.
- [Website Image Extractor](https://apify.com/innovative_cookie/website-image-extractor): list every image of a page with size, format and alt text.
- [Bulk Domain Availability Checker](https://apify.com/innovative_cookie/domain-availability): check whether domain names are free via RDAP.

# Actor input Schema

## `datasetIds` (type: `array`):

Pick one or more datasets (for example the results of several scraper runs). The Actor only gets READ access to the datasets you pick. Items are read in this order.

## `items` (type: `array`):

Optional items pasted as a JSON array. They are processed before the datasets. Handy for a quick test or for small lists.

## `dedupFields` (type: `array`):

Items with the same values in all these fields are duplicates. Use dot paths for nested fields (address.city). Leave empty to keep everything.

## `dedupWholeItem` (type: `boolean`):

When no field is given above, remove items that are exactly identical (all fields).

## `keep` (type: `string`):

first = keep the first item seen (streaming, low memory). last = keep the most recent one. merge = keep the first and fill its empty fields from the duplicates.

## `caseInsensitive` (type: `boolean`):

Treat 'Example.com' and 'example.com' as the same value.

## `trimValues` (type: `boolean`):

Trim and collapse whitespace in text values before comparing.

## `filters` (type: `array`):

Keep only items matching ALL conditions. Each condition is {"field": "price", "op": "lt", "value": 20}. Ops: eq, neq, contains, notContains, startsWith, endsWith, regex, gt, gte, lt, lte (numbers or dates), exists, notExists, empty, notEmpty, in, notIn (value = array).

## `pickFields` (type: `array`):

Output only these fields, in this order (dot paths allowed). Leave empty to keep all fields.

## `omitFields` (type: `array`):

Fields to drop from the output (dot paths allowed).

## `renameFields` (type: `object`):

Map old name to new name, applied last. Example: {"title": "name", "stock.count": "stock_count"}

## `flattenNested` (type: `boolean`):

Turn {"stock": {"count": 2}} into {"stock.count": 2}. Useful for CSV and Google Sheets.

## `flattenArrays` (type: `boolean`):

With flattening on, also turn arrays into numbered keys (images.0, images.1...).

## `skipHiddenFields` (type: `boolean`):

Drop fields starting with # (debug data) when reading datasets.

## `addSourceField` (type: `boolean`):

Add a \_source field with the dataset ID each item came from (or 'inline').

## `maxOutputItems` (type: `integer`):

Stop after writing this many items. 0 = no limit.

## Actor input object example

```json
{
  "items": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "title": "A Light in the Attic",
      "price": 51.77,
      "stock": {
        "available": true,
        "count": 22
      }
    },
    {
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "title": "Tipping the Velvet",
      "price": 53.74,
      "stock": {
        "available": true,
        "count": 20
      }
    },
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "title": "A Light in the Attic",
      "price": 51.77,
      "stock": {
        "available": true,
        "count": 22
      }
    },
    {
      "url": "https://books.toscrape.com/catalogue/soumission_998/index.html",
      "title": "Soumission",
      "price": 50.1,
      "stock": {
        "available": false,
        "count": 0
      }
    },
    {
      "url": "HTTPS://BOOKS.TOSCRAPE.COM/catalogue/soumission_998/index.html",
      "title": "Soumission (copy)",
      "price": 50.1,
      "stock": {
        "available": false,
        "count": 0
      }
    }
  ],
  "dedupFields": [
    "url"
  ],
  "dedupWholeItem": false,
  "keep": "first",
  "caseInsensitive": true,
  "trimValues": true,
  "filters": [],
  "pickFields": [],
  "omitFields": [],
  "renameFields": {},
  "flattenNested": false,
  "flattenArrays": false,
  "skipHiddenFields": false,
  "addSourceField": false,
  "maxOutputItems": 0
}
```

# Actor output Schema

## `dataset` (type: `string`):

Merged, deduplicated and transformed items

## `summary` (type: `string`):

SUMMARY record with counts per source

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "title": "A Light in the Attic",
            "price": 51.77,
            "stock": {
                "available": true,
                "count": 22
            }
        },
        {
            "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
            "title": "Tipping the Velvet",
            "price": 53.74,
            "stock": {
                "available": true,
                "count": 20
            }
        },
        {
            "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "title": "A Light in the Attic",
            "price": 51.77,
            "stock": {
                "available": true,
                "count": 22
            }
        },
        {
            "url": "https://books.toscrape.com/catalogue/soumission_998/index.html",
            "title": "Soumission",
            "price": 50.1,
            "stock": {
                "available": false,
                "count": 0
            }
        },
        {
            "url": "HTTPS://BOOKS.TOSCRAPE.COM/catalogue/soumission_998/index.html",
            "title": "Soumission (copy)",
            "price": 50.1,
            "stock": {
                "available": false,
                "count": 0
            }
        }
    ],
    "dedupFields": [
        "url"
    ],
    "caseInsensitive": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("innovative_cookie/dataset-merge-dedup").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "items": [
        {
            "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "title": "A Light in the Attic",
            "price": 51.77,
            "stock": {
                "available": True,
                "count": 22,
            },
        },
        {
            "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
            "title": "Tipping the Velvet",
            "price": 53.74,
            "stock": {
                "available": True,
                "count": 20,
            },
        },
        {
            "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
            "title": "A Light in the Attic",
            "price": 51.77,
            "stock": {
                "available": True,
                "count": 22,
            },
        },
        {
            "url": "https://books.toscrape.com/catalogue/soumission_998/index.html",
            "title": "Soumission",
            "price": 50.1,
            "stock": {
                "available": False,
                "count": 0,
            },
        },
        {
            "url": "HTTPS://BOOKS.TOSCRAPE.COM/catalogue/soumission_998/index.html",
            "title": "Soumission (copy)",
            "price": 50.1,
            "stock": {
                "available": False,
                "count": 0,
            },
        },
    ],
    "dedupFields": ["url"],
    "caseInsensitive": True,
}

# Run the Actor and wait for it to finish
run = client.actor("innovative_cookie/dataset-merge-dedup").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "title": "A Light in the Attic",
      "price": 51.77,
      "stock": {
        "available": true,
        "count": 22
      }
    },
    {
      "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
      "title": "Tipping the Velvet",
      "price": 53.74,
      "stock": {
        "available": true,
        "count": 20
      }
    },
    {
      "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
      "title": "A Light in the Attic",
      "price": 51.77,
      "stock": {
        "available": true,
        "count": 22
      }
    },
    {
      "url": "https://books.toscrape.com/catalogue/soumission_998/index.html",
      "title": "Soumission",
      "price": 50.1,
      "stock": {
        "available": false,
        "count": 0
      }
    },
    {
      "url": "HTTPS://BOOKS.TOSCRAPE.COM/catalogue/soumission_998/index.html",
      "title": "Soumission (copy)",
      "price": 50.1,
      "stock": {
        "available": false,
        "count": 0
      }
    }
  ],
  "dedupFields": [
    "url"
  ],
  "caseInsensitive": true
}' |
apify call innovative_cookie/dataset-merge-dedup --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,innovative_cookie/dataset-merge-dedup"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/unx1MbFpJ4APQG8IS/builds/9gK69Z03pdDCKwfdl/openapi.json