# Dataset Merge & Dedupe (limited permissions, dataset picker) (`fecklesspleb/dataset-merge-dedupe`) Actor

Merge the Apify datasets you pick, remove duplicates by any fields, filter, keep/drop/rename fields and keep only new items across runs. Works under limited permissions via the dataset picker. $0.25 per 1,000 output items.

- **URL**: https://apify.com/fecklesspleb/dataset-merge-dedupe.md
- **Developed by:** [fecklesspleb](https://apify.com/fecklesspleb) (community)
- **Categories:** Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 3 total users, 2 monthly users, 87.5% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Merge & Dedupe (works under limited permissions)

Merge the Apify datasets you pick, remove duplicates by any fields, filter, keep/drop/rename fields, and keep only items that earlier runs have not written. Results go to this run's dataset or to a dataset you pick.

**Why another merge Actor?** Under Apify's default *limited permissions*, an Actor can only read your datasets if you hand them over through a **dataset picker** field. This Actor's inputs are pickers (`resourceType: dataset`), so the platform grants the run access to exactly the datasets you pick. Nothing else in your account is readable.

### What it does

1. Reads the picked datasets in the order you picked them (items in dataset order).
2. Applies your **filters** (all must match).
3. Builds a **dedupe key** from your dedupe fields (dotted paths like `seller.id` work) and drops repeated keys, keeping the **first** or the **last** copy.
4. With **only new items** on, skips items whose key an earlier run already wrote.
5. Keeps, drops and renames top-level fields, optionally adds `_sourceDatasetId`.
6. Writes to the default dataset, or appends to the **output dataset** you pick.

Empty input runs a free built-in demo (two sample datasets, 8 items, 5 written), so you can see the output shape without picking anything.

### Input

| Field | Default | What it does |
|---|---|---|
| `datasetIds` | empty (= demo) | Datasets to merge. Dataset picker, READ access. You can also type a dataset ID or `username~dataset-name`. A bare dataset name is not looked up. |
| `dedupeFields` | prefilled `["url"]` | Same values in all these fields = duplicate. Empty = the whole item must be identical. Items that have none of the fields are never treated as duplicates (counted in `keptWithoutKey`). |
| `dedupe` | `true` | `false` = plain merge, every copy kept. |
| `keep` | `first` | `last` keeps the last copy. It reads every dataset twice, so read blocks are charged twice. |
| `normalizeValues` | `false` | Compare strings ignoring case and extra spaces (output is unchanged). |
| `filters` | none | List of `{"field", "op", "value"}`. Ops: `eq ne contains notContains exists notExists gt gte lt lte in notIn`. Strings ignore case unless `"caseSensitive": true`. Numbers in strings (`"20"`) compare as numbers for `gt/gte/lt/lte`. Filters run **before** dedupe. |
| `fieldsToKeep` / `fieldsToOmit` | none | Top-level fields to keep (in that order) or drop. |
| `renameFields` | none | `{"old": "new"}`, applied after keep/drop. Dedupe always uses original names. |
| `addSourceField` | `false` | Adds `_sourceDatasetId`. |
| `onlyNewItems` | `false` | Skip items whose dedupe key an earlier run wrote. Keys are stored in the named key-value store below, separately for each dedupe setting (or `stateKey`). The newest 2,000,000 keys are kept. |
| `stateStoreName` | `dataset-merge-dedupe-state` | Named key-value store for `onlyNewItems`. |
| `stateKey` | derived | Name of a pipeline, so different pipelines with the same dedupe fields don't share state. |
| `outputDatasetId` | empty (= default dataset) | Append results to this dataset instead. Picker with READ and WRITE. The default dataset then gets one free summary item. |
| `maxInputItems` | 1,000,000 | Read at most this many items (hard limit 5,000,000). |
| `maxOutputItems` | no cap | Write at most this many items. |

### Output

Your own items, merged and transformed. Problems (a dataset that could not be opened, invalid input) appear as free items with `"type": "status"`, and the run still ends successfully so schedules keep working. `RUN_SUMMARY` in the key-value store has counts per dataset: items read, filtered out, duplicates dropped, seen in earlier runs, written, and why the run stopped.

### Examples (from real local runs, 2026-11-11)

These were run with the Apify SDK's local storage (no cloud). Two datasets:
`shop-a`: `{"sku":"1","name":"Lamp","price":"20"}`, `{"sku":"2","name":"Desk","price":150}`, `{"sku":"3","name":" desk ","price":99}` (plus `seller` on the first two);
`shop-b`: `{"sku":"2","name":"Desk","price":140}`, `{"sku":"4","name":"Chair","price":45}`, `{"name":"No sku item","price":1}`.

**1. Empty input (`{}`)** runs the demo: read 8, wrote 5, dropped 3 duplicates by `url`. First item:

```json
{"url": "https://example.com/a", "title": "Alpha", "price": 10}
```

**2. Keep the latest price, only items costing 20 or more, tidy fields:**

```json
{"datasetIds": ["<shop-a id>", "<shop-b id>"], "dedupeFields": ["sku"], "keep": "last",
 "filters": [{"field": "price", "op": "gte", "value": 20}],
 "fieldsToKeep": ["sku", "name", "price"], "renameFields": {"price": "priceUsd"}}
```

Output (read 6, filtered 1, duplicates dropped 1, wrote 4):

```json
[{"sku": "1", "name": "Lamp", "priceUsd": "20"},
 {"sku": "3", "name": " desk ", "priceUsd": 99},
 {"sku": "2", "name": "Desk", "priceUsd": 140},
 {"sku": "4", "name": "Chair", "priceUsd": 45}]
```

**3. A dataset ID that does not exist** (`{"datasetIds": ["2x1LjUSyc4tESKlhw", "HP1VXZZljAqGEMZmf", "missingID123"], "dedupeFields": ["sku"], "addSourceField": true}`; the first two are the local IDs of shop-a and shop-b): the two real datasets are merged (5 items written) and one free status item explains the third:

```json
{"type": "status", "status": "error", "message": "Dataset 'missingID123' could not be opened, so it was skipped (nothing read or charged for it). Pick it with the dataset picker, or give its ID or username~dataset-name. Error: ValueError: Dataset with ID \"missingID123\" not found", "datasetId": "missingID123"}
```

### Pricing

Pay per event:

| Event | Price | When |
|---|---|---|
| `item-output` | $0.00025 ($0.25 per 1,000) | each item written |
| `input-items-1000` | $0.0015 | each started block of 1,000 items read |
| `apify-actor-start` | $0.00005 | Apify's standard run start (one per GB of memory, so 1 at the default 1,024 MB) |
| demo run, status and summary items | free | |

Your **maximum charge per run** is respected: the run stops reading and writing when the next event would go over it and says so in `RUN_SUMMARY.stoppedBecause`.

### Limits and notes

- Dedupe keys are kept in memory (about 100 bytes each, ESTIMATED from a local benchmark: 500,000 unique keys used about 70 MB). The default 1,024 MB covers a few million unique keys.
- Nested fields can be used for dedupe and filters, but keep/drop/rename work on top-level fields only.
- `1` and `1.0` are the same number. `true` is not `1`. `"20"` equals `20` only in `gt/gte/lt/lte`.
- The Actor reads your own datasets through the Apify platform. It fetches nothing from the web.

### Terms and permissions evidence

Read on 2026-11-11 from Apify's documentation. Copies are in `evidence/`.

- Limited permissions, https://docs.apify.com/platform/actors/development/permissions: "By default, limited-permissions Actors can't access user storages. However, they can access storages that users explicitly provide via the Actor input. To do so, use the input schema to add a storage picker and declare exactly which operations your Actor needs." and "Add a field with `editor: "resourcePicker"`. Set `resourceType` to one of `dataset`, `keyValueStore`, or `requestQueue`. Specify `resourcePermissions` with the minimal required scope". Also: "The user's selection is injected into the run input, and the run token is expanded to allow only the requested operations on the selected storages."
- Resource input type, https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1/resource.md: "the resource input value is a string representing either the resource ID or (unique) name. The type of the property is either `string` or `array`."
- Named stores under limited permissions, https://docs.apify.com/platform/actors/development/permissions/migration-guide: "With limited permissions, an Actor can create a named storage on its first run and will automatically retain access to it in all subsequent runs by the same user."
- Get dataset API, https://docs.apify.com/api/v2/dataset-get: "datasetId ... Dataset ID or `username~dataset-name`."

# Changelog

This Actor's version history is a separate document: https://apify.com/fecklesspleb/dataset-merge-dedupe/changelog.md

# Actor input Schema

## `datasetIds` (type: `array`):

Pick one or more of your datasets. They are read in this order. Works under limited permissions: the picker grants this run READ access to exactly the datasets you pick. You can also type a dataset ID or username~dataset-name. Leave empty to run the free built-in demo (2 sample datasets, 8 items).

## `dedupeFields` (type: `array`):

Items with the same values in all of these fields are duplicates. Dotted paths reach nested fields (e.g. "seller.id"). Empty = whole item must be identical. Items that have none of these fields are never treated as duplicates.

## `dedupe` (type: `boolean`):

Off = plain merge (keeps every copy). onlyNewItems still works with this off.

## `keep` (type: `string`):

first = the first copy in merged order (datasets in the order picked, items in dataset order). last = the last copy; this reads every dataset twice, so read blocks are charged twice.

## `normalizeValues` (type: `boolean`):

On: "  Foo  Bar" and "foo bar" count as the same value when deduping. Output values are not changed.

## `filters` (type: `array`):

Keep only items matching every filter, applied before dedupe. Each filter: {"field": "price", "op": "lt", "value": 20}. op: eq, ne, contains, notContains, exists, notExists, gt, gte, lt, lte, in, notIn. String comparisons ignore case unless "caseSensitive": true.

## `fieldsToKeep` (type: `array`):

Top-level field names, in output order. Empty = keep all fields.

## `fieldsToOmit` (type: `array`):

Top-level field names to remove from the output.

## `renameFields` (type: `object`):

{"oldName": "newName"}, top-level only, applied after keep/drop. Dedupe always uses the original names.

## `addSourceField` (type: `boolean`):

Adds "\_sourceDatasetId" to every output item.

## `onlyNewItems` (type: `boolean`):

Skips items whose dedupe key was written by an earlier run with the same dedupe settings (or the same stateKey). Keys are kept in a named key-value store in your account.

## `stateStoreName` (type: `string`):

Named key-value store for onlyNewItems. Created on the first run; under limited permissions this Actor keeps access to stores it created.

## `stateKey` (type: `string`):

Separates different pipelines that share a state store. Empty = derived from the dedupe fields and the normalize setting.

## `outputDatasetId` (type: `string`):

Pick a dataset to append the results to. Empty = this run's default dataset. Needs READ and WRITE, which the picker grants.

## `maxInputItems` (type: `integer`):

Stop reading after this many items across all datasets (hard limit 5,000,000).

## `maxOutputItems` (type: `integer`):

Stop after writing this many items. Empty = no cap.

## Actor input object example

```json
{
  "dedupeFields": [
    "url"
  ],
  "dedupe": true,
  "keep": "first",
  "normalizeValues": false,
  "addSourceField": false,
  "onlyNewItems": false,
  "stateStoreName": "dataset-merge-dedupe-state",
  "maxInputItems": 1000000
}
```

# Actor output Schema

## `items` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupeFields": [
        "url"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fecklesspleb/dataset-merge-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "dedupeFields": ["url"] }

# Run the Actor and wait for it to finish
run = client.actor("fecklesspleb/dataset-merge-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupeFields": [
    "url"
  ]
}' |
apify call fecklesspleb/dataset-merge-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fecklesspleb/dataset-merge-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4JDWv92RrbfxNCFWQ/builds/zPlpBSCeMY9jjaAEG/openapi.json
