# Dataset Merge, Dedupe & Only New Items (`genesis.labs.co/dataset-merge-dedupe`) Actor

Merge Apify datasets, remove duplicates on normalized fields (URLs, case, spaces) and output only items not seen in earlier runs. Works with limited permissions, schedules and integrations. CSV/Excel export.

- **URL**: https://apify.com/genesis.labs.co/dataset-merge-dedupe.md
- **Developed by:** [Nicholas Powers](https://apify.com/genesis.labs.co) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.08 / 1,000 output items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Merge, Dedupe & Only New Items

**Merge any number of Apify datasets, remove duplicates by URL or any field, and output only the items you haven't seen in earlier runs.** Built for scheduled scrapers, integrations and AI agents: it runs with **limited permissions**, so it starts without an extra permission prompt and only reads the datasets you give it.

- 🔀 **Merge** many datasets into one, in the order you choose
- 🧹 **Dedupe** on one or more key fields, with smart matching: spaces, letter case, accents, and URL variants (`http`/`https`, `www.`, trailing slashes, `#fragments`, parameter order, `utm_*`, `fbclid`, `gclid`...)
- 🆕 **Only new items**: remembers every key it has output under a *Memory name*, so a daily scraper run returns just today's new listings, jobs, products or posts. Scales to millions of keys (8 bytes each, loaded in well under a second for 100k keys)
- ✂️ **Pick, remove and rename fields**, including nested ones (`price.value` → `price`)
- 📤 **CSV, Excel and JSON** download links for the result
- 🔗 **Integration-ready**: add it as "Run another Actor when a run succeeds" with `{"datasetId": "{{resource.defaultDatasetId}}"}`
- 💸 **Duplicates and already-seen items are never charged**

### Pricing

Pay per event. Compute is included.

| Event | Free & Bronze | Silver | Gold & above |
|---|---|---|---|
| Output item (unique or new) | $0.10 / 1,000 | $0.09 / 1,000 | $0.08 / 1,000 |
| Items read and compared | $0.002 / 1,000 | $0.002 / 1,000 | $0.002 / 1,000 |
| Run start | $0.00005 per GB | same | same |

**Examples (Free/Bronze prices):**

| Job | Read | Output | Cost |
|---|---|---|---|
| Merge 3 scraper runs and remove duplicates | 150,000 | 120,000 | about **$12.30** |
| Daily "only new" check on a 30,000-item scrape with 500 new items | 30,000 | 500 | about **$0.11 a day** (about $3.30 a month) |
| Hourly check of 2,000 items, usually nothing new | 2,000 | 0 | about **$0.004 per run** |
| Dedupe 1,000,000 leads down to 400,000 | 1,000,000 | 400,000 | about **$42** |

Set **Max cost per run** in the run options: the Actor stops before it goes over, and items it couldn't output are not remembered, so the next run picks them up.

Running with no datasets selected starts a free 4-item demo (useful for Store health checks and trying the Actor); pick a dataset or pass `datasetId` for real data.

### How to use it

#### Dedupe or merge datasets

1. Pick one or more datasets in **Datasets** (from your runs, or paste dataset IDs).
2. Choose **Mode**: *Merge + remove duplicates* or *Merge only*.
3. Set **Key fields**, for example `url`, or `title` + `price`. Leave empty to compare whole items.
4. Start. Download the result as CSV or Excel from the **Output** tab.

When two items share a key, the first one wins, so list your most trusted dataset first.

#### Only new items after every scraper run

1. Open your scraper (or its task) → **Integrations** → **Run another Actor** → pick this Actor.
2. Use this input:

```json
{
  "datasetId": "{{resource.defaultDatasetId}}",
  "mode": "onlyNew",
  "memoryName": "amazon-laptops",
  "dedupeFields": ["url"]
}
```

3. Each scraper run now produces a second dataset with only the items that never appeared before. Chain it into Google Sheets, Slack, email or a webhook.

The memory lives in a key-value store in your account named `dedupe-memory-<memory name>`. Use a different memory name per scraper or per search. **Reset memory** starts over.

### Input example

```json
{
  "datasetIds": ["flob8zAqUIziccWR2", "dzm4JlQiUbnu3dA3C"],
  "mode": "dedupe",
  "dedupeFields": ["url"],
  "normalizeUrls": true,
  "caseInsensitive": true,
  "pickFields": ["url", "title", "price.value"],
  "renameFields": { "price.value": "price" }
}
```

### Output

The default dataset holds your items with their original fields (after pick, remove and rename). The `OUTPUT` record summarizes the run:

```json
{
  "mode": "onlyNew",
  "itemsRead": 30000,
  "itemsOutput": 15000,
  "duplicatesRemoved": 0,
  "seenInEarlierRuns": 15000,
  "memoryKeys": 135000,
  "exports": {
    "csv": "https://api.apify.com/v2/datasets/<id>/items?format=csv",
    "xlsx": "https://api.apify.com/v2/datasets/<id>/items?format=xlsx"
  }
}
```

### Matching rules

| Option | Default | Effect |
|---|---|---|
| Trim spaces | on | `" Apple "` = `"Apple"` |
| Collapse inner whitespace | on | `"New  York"` = `"New York"` |
| Ignore letter case | on | `"IPHONE"` = `"iPhone"` |
| Normalize URLs | on | `http://www.shop.com/p/1/?utm_source=x` = `https://shop.com/p/1` |
| Ignore all URL parameters | off | `shop.com/p?id=1` = `shop.com/p?id=2` |
| Ignore accents | off | `"Café"` = `"Cafe"` |

Numbers, booleans and objects are compared by value. Items whose key fields are all empty are kept by default (choose *Drop them* to remove them).

### Performance

Measured on Apify with 1 GB of memory: 150,000 items from 3 datasets deduped in 35 seconds; an only-new run over 30,000 items against 120,000 remembered keys took 4 seconds (memory loaded in 0.18 s). Works on 256 MB for most jobs.

### Troubleshooting

- **"No access to dataset"**: this Actor uses limited permissions. Select datasets in **Datasets** or pass them in `datasetId`/`datasetIds`. IDs placed elsewhere can't be read.
- **"is not a dataset ID"**: use the 17-character ID (e.g. `WkzbQMuFYuamGv3YF`) or the picker, not a dataset name.
- **"template that was not filled in"**: `{{resource.defaultDatasetId}}` only works inside an integration.
- **Everything shows up as new again**: the memory name or key fields changed. The log warns when key fields differ from the ones the memory was built with.
- **Two runs at once with the same memory name**: avoid it; the second run may not see the first run's items.

### FAQ

**How accurate is "only new"?** Keys are stored as 64-bit fingerprints. The chance that any two different keys collide is about 1 in 37 million at 1 million remembered keys, and about 1 in 370,000 at 10 million.

**Do I pay for duplicates?** No. You pay for items written to the output and a small fee per 1,000 items read.

**Where is my data stored?** Only in your own Apify account: the output dataset and the memory key-value store.

Questions or ideas? Open an issue on the **Issues** tab.

# Actor input Schema

## `datasetIds` (type: `array`):

Datasets to merge, in priority order (when two items share a key, the one from the earlier dataset is kept). Pick from your runs or paste dataset IDs.

## `datasetId` (type: `string`):

For integrations: set to {{resource.defaultDatasetId}} in "Run another Actor when a run succeeds" to process each finished scraper run. Merged after "Datasets".

## `mode` (type: `string`):

Only new items: outputs items whose key was never output by an earlier run with the same Memory name. Great for scheduled scrapers and alerts.

## `dedupeFields` (type: `array`):

Fields that identify an item, e.g. url, or title + price. Dot paths work (e.g. location.city). Leave empty to compare whole items.

## `memoryName` (type: `string`):

Letters, digits and hyphens. Each name keeps its own list of seen keys in a key-value store in your account, so runs with the same name never output the same item twice.

## `resetMemory` (type: `boolean`):

Forget all keys stored under this Memory name and start over.

## `emptyKeyItems` (type: `string`):

What to do when all key fields of an item are empty.

## `trim` (type: `boolean`):

Ignore leading and trailing spaces when comparing keys.

## `collapseWhitespace` (type: `boolean`):

Treat runs of spaces, tabs and newlines as one space.

## `caseInsensitive` (type: `boolean`):

"iPhone 15" and "IPHONE 15" match.

## `normalizeUrls` (type: `boolean`):

http/https, www., trailing slashes, #fragments, parameter order and tracking parameters (utm\_\*, fbclid, gclid...) don't create duplicates.

## `stripUrlQuery` (type: `boolean`):

Compare URLs without any ?query parameters.

## `removeDiacritics` (type: `boolean`):

"Café" and "Cafe" match.

## `pickFields` (type: `array`):

Output only these fields (dot paths allowed). Leave empty to keep all.

## `omitFields` (type: `array`):

Top-level fields to drop, e.g. #debug, html.

## `renameFields` (type: `object`):

Map of old name to new name, e.g. {"url": "productUrl", "price.value": "price"}.

## `addSourceField` (type: `boolean`):

Adds \_sourceDatasetId to every output item.

## `maxItems` (type: `integer`):

Stop after this many output items.

## Actor input object example

```json
{
  "mode": "dedupe",
  "dedupeFields": [
    "url"
  ],
  "resetMemory": false,
  "emptyKeyItems": "keep",
  "trim": true,
  "collapseWhitespace": true,
  "caseInsensitive": true,
  "normalizeUrls": true,
  "stripUrlQuery": false,
  "removeDiacritics": false,
  "renameFields": {},
  "addSourceField": false
}
```

# Actor output Schema

## `items` (type: `string`):

The merged, deduplicated items.

## `csv` (type: `string`):

All output items as CSV.

## `xlsx` (type: `string`):

All output items as an Excel file.

## `summary` (type: `string`):

Counts of items read, output, duplicates removed and seen in earlier runs.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dedupeFields": [
        "url"
    ],
    "renameFields": {}
};

// Run the Actor and wait for it to finish
const run = await client.actor("genesis.labs.co/dataset-merge-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "dedupeFields": ["url"],
    "renameFields": {},
}

# Run the Actor and wait for it to finish
run = client.actor("genesis.labs.co/dataset-merge-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dedupeFields": [
    "url"
  ],
  "renameFields": {}
}' |
apify call genesis.labs.co/dataset-merge-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,genesis.labs.co/dataset-merge-dedupe"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/uUWsyWl1CEjf5VxRe/builds/SiNge2e8cBS4fHpp1/openapi.json