# Apify Dataset New Items Deduplicator (`davidbenittah/apify-dataset-new-items-deduplicator`) Actor

Return only never-before-seen rows from repeated Apify datasets and stop duplicate alerts, CRM records, and downstream automation actions.

- **URL**: https://apify.com/davidbenittah/apify-dataset-new-items-deduplicator.md
- **Developed by:** [David](https://apify.com/davidbenittah) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Apify Dataset New Items Deduplicator

Return only rows that have never been seen before. This Actor prevents repeated Apify runs and webhook retries from creating duplicate alerts, duplicate CRM records, repeated Slack messages, or repeated actions in Make and Zapier.

It is an independent, unofficial utility and is not affiliated with Apify.

### The problem it removes

A scheduled scraper often creates a fresh dataset on every run. Downstream workflows may then receive the same business, product, job, review, or page again. A webhook retry can repeat the damage. Comparing the newest dataset by hand is slow, while a stateless prompt cannot remember what previous runs already delivered.

This Actor keeps only compact fingerprints between runs. It does not copy the complete source dataset into its state. Replay the same source and the output is empty; add one genuinely new row and the output contains exactly that row.

### Ready-to-run example

```json
{
  "datasetId": "YOUR_DATASET_ID",
  "uniqueFields": ["url"],
  "stateName": "daily-crm-import",
  "resetState": false,
  "maxItems": 1000
}
```

Replace `YOUR_DATASET_ID` with a dataset owned by your Apify account or a public dataset. Run once to establish the baseline. Keep `stateName` and `uniqueFields` unchanged on later runs.

Private datasets require your Apify API token in the encrypted `apiToken` field. Actor run tokens are intentionally limited and cannot read a different private dataset. The prefilled `example` source needs no token and processes two built-in rows, so you can verify the output before connecting your own storage.

### Inputs

- **datasetId** — source Apify dataset ID.
- **apiToken** — encrypted Apify token used only to read a private source dataset; leave it empty for the built-in example or a public dataset.
- **uniqueFields** — stable fields that identify a row. Dotted fields such as `company.id` are accepted. Use more than one when no single field is unique, for example `company.domain` and `job.id`.
- **stateName** — separates independent automations. Two workflows can read the same source without sharing history.
- **resetState** — forgets the saved fingerprints for this exact dataset, field set, and state name.
- **maxItems** — hard safety limit on source rows read in one run.

### Output

Only new source rows are written to the default dataset. Original fields remain unchanged. Three metadata fields are added: `_dedupKey`, `_firstSeenAt`, and `_sourceDatasetId`. A `SUMMARY` record reports `readCount`, `newCount`, `duplicateCount`, and total saved fingerprints. This makes zero-new-item runs explicit instead of looking like silent failures.

### Common workflows

#### Make, Zapier, or n8n

Run the scraper, pass its dataset ID to this Actor, then send only this Actor's dataset to the next step. This removes repeated CRM inserts or messages without asking the automation platform to maintain its own database.

#### Recurring alerts

Use a stable URL, listing ID, review ID, or product ID. Schedule the Actor after each collection. A retry of the same dataset returns no repeated alerts.

#### Composite identity

If URLs contain tracking parameters or change frequently, select deterministic business fields instead. For example, combine `company.domain` with `job.id`. The Actor compares normalized field values, not the whole row, so changing a description does not create a false new item.

### Limits and behavior

The Actor reads up to 100,000 rows per run and stores at most 500,000 fingerprints per state. It does not send emails, call arbitrary URLs, or export contact details. Do not overlap two runs using the same state: schedule the next run after the previous one finishes. If identity fields are missing, those missing values are deterministic and can collapse several rows into one; choose fields present on every source item.

State is isolated by source dataset, ordered field list, and `stateName`. Changing any of them deliberately starts a separate history. Use `resetState` only when you want the next run to return the full baseline again.

# Actor input Schema

## `datasetId` (type: `string`):

ID of the Apify dataset to read. The prefilled example value runs safely on two built-in sample rows.

## `apiToken` (type: `string`):

Required for private source datasets because Actor run tokens cannot read other datasets. Apify encrypts this value.

## `uniqueFields` (type: `array`):

One or more stable dotted field paths, such as url or company.id. Their normalized values form the deduplication key.

## `stateName` (type: `string`):

Separates independent workflows that read the same dataset. Keep this unchanged between repeated runs of one workflow.

## `resetState` (type: `boolean`):

When enabled, clears this workflow's saved fingerprints before reading the source dataset.

## `maxItems` (type: `integer`):

Safety limit for one run. The Actor reads at most this many newest source rows.

## Actor input object example

```json
{
  "datasetId": "example",
  "uniqueFields": [
    "url"
  ],
  "stateName": "default",
  "resetState": false,
  "maxItems": 1000
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "datasetId": "example",
    "uniqueFields": [
        "url"
    ],
    "stateName": "default",
    "resetState": false,
    "maxItems": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("davidbenittah/apify-dataset-new-items-deduplicator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "datasetId": "example",
    "uniqueFields": ["url"],
    "stateName": "default",
    "resetState": False,
    "maxItems": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("davidbenittah/apify-dataset-new-items-deduplicator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "datasetId": "example",
  "uniqueFields": [
    "url"
  ],
  "stateName": "default",
  "resetState": false,
  "maxItems": 1000
}' |
apify call davidbenittah/apify-dataset-new-items-deduplicator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=davidbenittah/apify-dataset-new-items-deduplicator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/iJtlAZ3aCAP5Fjnnm/builds/MVhu6UqWgfd9foljl/openapi.json