Apify Dataset New Items Deduplicator

Return only rows that have never been seen before. This Actor prevents repeated Apify runs and webhook retries from creating duplicate alerts, duplicate CRM records, repeated Slack messages, or repeated actions in Make and Zapier.

It is an independent, unofficial utility and is not affiliated with Apify.

The problem it removes

A scheduled scraper often creates a fresh dataset on every run. Downstream workflows may then receive the same business, product, job, review, or page again. A webhook retry can repeat the damage. Comparing the newest dataset by hand is slow, while a stateless prompt cannot remember what previous runs already delivered.

This Actor keeps only compact fingerprints between runs. It does not copy the complete source dataset into its state. Replay the same source and the output is empty; add one genuinely new row and the output contains exactly that row.

Ready-to-run example

{
  "datasetId": "YOUR_DATASET_ID",
  "uniqueFields": ["url"],
  "stateName": "daily-crm-import",
  "resetState": false,
  "maxItems": 1000
}

Replace YOUR_DATASET_ID with a dataset owned by your Apify account or a public dataset. Run once to establish the baseline. Keep stateName and uniqueFields unchanged on later runs.

Private datasets require your Apify API token in the encrypted apiToken field. Actor run tokens are intentionally limited and cannot read a different private dataset. The prefilled example source needs no token and processes two built-in rows, so you can verify the output before connecting your own storage.

Inputs

• datasetId — source Apify dataset ID.
• apiToken — encrypted Apify token used only to read a private source dataset; leave it empty for the built-in example or a public dataset.
• uniqueFields — stable fields that identify a row. Dotted fields such as company.id are accepted. Use more than one when no single field is unique, for example company.domain and job.id.
• stateName — separates independent automations. Two workflows can read the same source without sharing history.
• resetState — forgets the saved fingerprints for this exact dataset, field set, and state name.
• maxItems — hard safety limit on source rows read in one run.

Output

Only new source rows are written to the default dataset. Original fields remain unchanged. Three metadata fields are added: _dedupKey, _firstSeenAt, and _sourceDatasetId. A SUMMARY record reports readCount, newCount, duplicateCount, and total saved fingerprints. This makes zero-new-item runs explicit instead of looking like silent failures.

Common workflows

Make, Zapier, or n8n

Run the scraper, pass its dataset ID to this Actor, then send only this Actor's dataset to the next step. This removes repeated CRM inserts or messages without asking the automation platform to maintain its own database.

Recurring alerts

Use a stable URL, listing ID, review ID, or product ID. Schedule the Actor after each collection. A retry of the same dataset returns no repeated alerts.

Composite identity

If URLs contain tracking parameters or change frequently, select deterministic business fields instead. For example, combine company.domain with job.id. The Actor compares normalized field values, not the whole row, so changing a description does not create a false new item.

Limits and behavior

The Actor reads up to 100,000 rows per run and stores at most 500,000 fingerprints per state. It does not send emails, call arbitrary URLs, or export contact details. Do not overlap two runs using the same state: schedule the next run after the previous one finishes. If identity fields are missing, those missing values are deterministic and can collapse several rows into one; choose fields present on every source item.

State is isolated by source dataset, ordered field list, and stateName. Changing any of them deliberately starts a separate history. Use resetState only when you want the next run to return the full baseline again.