Remove Duplicates & Merge Datasets: Dedup, Only New

Remove Duplicates & Merge Datasets: Dedup, Only New

Dedup and merge Apify datasets by any field. Pick one or more Apify datasets. The Actor reads them in order, and returns:

• Unique items: the first item of each key, with all its original fields. Or choose only the duplicates.
• Dedup by any fields: for example email, url, or name + address.city. Leave it empty to compare whole items.
• Only new items: remember every key across runs and skip items an earlier run already saw. Put it after a daily scraper to get only today's new results.
• A summary in the OUTPUT record: items read, unique, duplicates, items written, and items read per dataset.

$0.02 per 1,000 items. It works with limited permissions: the Actor can read only the datasets you pick.

How to use

1. Open the Actor in the Apify Console and go to the Input tab.
2. Pick your datasets in Datasets to merge (Storage > Datasets), or paste items into JSON items.
3. Type the fields that make an item unique in Dedup by fields, for example email.
4. Click Start.
5. Open the Output tab and download the unique items as CSV, Excel or JSON, or read them through the API.

The prefilled input dedups 5 sample items by email and returns 3, so a one-click run shows how it works.

What can you do with it?

• Merge runs: join the results of 10 scraper runs into one clean list.
• Clean leads: keep one row per email or phone, and ignore case and extra spaces with Ignore case and extra spaces.
• Get only new results each day: run a scraper on a schedule, add this Actor as an integration with {{resource.defaultDatasetId}} and Only items not seen in earlier runs, and get only the new jobs, listings or posts.
• Find duplicates: set Output to Only the duplicates to see what repeats.
• AI agents: call the Actor as a tool (Apify API or Apify MCP server) to dedup a dataset before it reads it.

Input example

{
  "datasetIds": ["<first dataset ID>", "<second dataset ID>"],
  "dedupFields": ["email"],
  "normalize": true,
  "output": "unique",
  "onlyNew": true,
  "memoryKey": "daily-leads",
  "maxItems": 100000
}

Output example

The dataset holds your items exactly as they were, minus the duplicates. The OUTPUT record holds the summary:

{
  "itemsRead": 5,
  "uniqueItems": 3,
  "duplicateItems": 2,
  "itemsOutput": 3,
  "output": "unique",
  "dedupFields": ["email"],
  "onlyNew": false,
  "sources": [{ "datasetId": "input items", "itemsRead": 5 }],
  "notes": []
}

Run it after another Actor (integration)

1. Open your scraper, go to the Integrations tab, and add this Actor.
2. Set Datasets to merge to {{resource.defaultDatasetId}}.
3. Turn on Only items not seen in earlier runs and give a Memory name, for example the scraper's name.

Each run now writes only the items that are new since the last run.

How duplicates are found

• Two items are duplicates when all fields in Dedup by fields have the same values. Dots read nested fields: address.city.
• Without fields, whole items are compared. The order of keys does not matter.
• With Ignore case and extra spaces, "HI@x.com " equals "hi@x.com". The output keeps the original text.
• An item where every dedup field is empty is always kept, so items with no email are not merged into one.
• The first item of each key wins, in the order of Datasets to merge, then JSON items.
• Keys are stored as short hashes, so a run uses little memory. Only new items mode keeps up to 5,000,000 keys in the key-value store dedup-memory-<name> in your account. Delete that store to start fresh.

Pricing

• $0.02 per 1,000 items processed ($0.00002 per item read and checked).
• A tiny Actor start event (Apify's standard start charge) applies once per run.
• Set Max items to read or a maximum total charge on the run, and the Actor stops before it goes over.

The prefilled input reads 5 items, so it costs about $0.00015: $0.0001 for the 5 items plus $0.00005 for the start event (one event at the default 1 GB of memory).

Apify free plan: each run reads the first 10,000 items. Upgrade to a paid Apify plan to process any number in one run.

FAQ

Why do other dedup Actors fail with a permission error? Under limited permissions, Apify gives an Actor access only to the datasets picked in a dataset picker field. A plain text field with a dataset ID is not enough. This Actor uses the picker, so it can read your datasets without full account access.

Can I dedup a dataset from another account? Only if your account can read it.

Does it change my datasets? No. It only reads them and writes the result to the run's own dataset.

How does it read big datasets? In pages of 5,000 items, so memory stays low. Only the short hash of each key is kept.

Limits

• Apify free plan: up to 10,000 items per run. Paid plans have no fixed cap.
• Max items to read and the run's maximum total charge stop the run early. The OUTPUT record then has a note.
• Only new items remembers up to 5,000,000 keys per memory name. Older keys are forgotten first.
• The Actor can read only the datasets picked in Datasets to merge. A dataset it cannot read stops the run with an error.
• JSON items must be valid JSON: an array of objects, or one object.

Related Actors

• Google Sheets Export & Sync: send the unique items to a Google Sheet.
• ATS Jobs API: Greenhouse, Lever, Ashby, Workable: pair it with only new items for daily new job alerts.

Issues and support

Found a bug or need a feature? Open an issue in the Actor's Issues tab. We check it every day and reply there.