Dedupe and Merge Datasets: Only New Items

What this Actor does

Dataset Merge, Dedupe and Only New Items takes one or more Apify datasets (or JSON and CSV files), merges them into one list and removes duplicate records. You choose the fields that identify a record, such as a URL, an email, a phone number or a company name. Values are cleaned before they are compared, so https://www.shop.com/item/?utm_source=ad and shop.com/item count as the same page, and, with US as the default phone country, (415) 555-2671 and +1 415 555 2671 count as the same phone.

It can also remember what it has already returned. Put it after a scheduled scraper, turn on Only output items not seen in earlier runs, and each run outputs only the records that are new since the last run, plus, if you want, the ones whose details changed.

Typical uses:

• Combine the results of several scraper runs or tasks into one clean list
• Remove repeated leads, listings or products before sending them to a CRM, Google Sheets or an email tool
• Get only the new job posts, listings or products from a daily or hourly scrape
• Drop records you already have by comparing against an older dataset
• Clean up a JSON or CSV export from another tool

How duplicates are matched

Records are duplicates when all Dedupe key fields (dedupeFields) match after cleaning. Leave the list empty to remove only records that are identical in every field.

Key type	What is ignored when comparing
Text (default)	Spaces at the ends, repeated spaces, and upper or lower case (unless Case-sensitive keys is on)
Treat as URLs (urlFields)	http/https, www., a trailing slash, #fragment, default ports, index.html, and tracking parameters such as utm_*, gclid, fbclid, msclkid, mc_cid, _ga. Other query parameters are kept and sorted. Paths stay case-sensitive.
Treat as emails (emailFields)	Case, a mailto: prefix, and a display name such as Jane <jane@x.com>
Treat as phone numbers (phoneFields)	Formatting. Numbers are converted to international format with libphonenumber, using Default phone country (defaultPhoneCountry) for numbers without a + code. Numbers it can't parse are compared by their digits.
Fuzzy match field (fuzzyField)	Case, accents, punctuation and word order, and small spelling differences up to Fuzzy similarity threshold (fuzzyThreshold, default 90%)

Fuzzy similarity is 100 minus the edit distance as a percentage of the longer text. "Acme Corporation" and "Acme Corporaton" are 94% similar. To keep runs fast, a record is only compared with earlier records that share at least one word with it and have the same values in the other key fields. Fuzzy matching suits names and titles. It is a poor fit for codes that differ by one digit, such as "Product 101" and "Product 102", which are 91% similar.

Which record to keep (keep) decides what happens to each group of duplicates:

• first: the first record. Fastest, and the output keeps the input order.
• last: the latest record.
• most-complete: the record with the most filled-in fields. Ties go to the earlier one.
• merge: the first record, with its empty fields filled from the later duplicates.

Records where a key field is empty are handled by Records with an empty key field (missingKeyAction): keep outputs each of them, drop leaves them out, and compare-as-empty lets records with the same empty field match each other.

Only new items, for scheduled scrapers

With Only output items not seen in earlier runs (onlyNewItems) on, the Actor saves a short hash of each output record's key in a key-value store named plainsource-dataset-merge-dedupe-state in your account. On the next run, records with a known key are left out.

• State key (stateKey) keeps separate lists apart. Use a different one for each scraper or task.
• Also output changed items (outputChanged) returns records seen before whose other fields changed since they were last output, for example a new price. Add fields that change on every scrape, such as scrapedAt, to Fields to ignore when checking for changes (changeIgnoreFields).
• Add a _status field (addStatusField) marks each output record as new or changed.
• Remember items from this run (rememberItems) off lets you test settings without updating the list. Forget remembered items first (forgetState) starts the list over.
• If you change the key fields or their cleaning settings, the saved list no longer fits, so the next run treats every item as new and starts a fresh list.

Only records that were actually written are remembered. If a run stops at your maximum cost, the records it didn't read or write come back on the next run.

Run it after another Actor

To dedupe a scraper's results every time it finishes:

1. Open the scraper (or its task) in Apify Console and go to the Integrations tab.
2. Choose Connect Actor or task, pick this Actor, and the event Run succeeded.
3. Set the input, keeping datasetIds as shown:

{
    "datasetIds": ["{{resource.defaultDatasetId}}"],
    "dedupeFields": ["url"],
    "urlFields": ["url"],
    "onlyNewItems": true,
    "stateKey": "my-scraper"
}

Each run then reads the scraper's dataset and writes only the new items to its own default dataset. Add an integration on this Actor in turn (for example Google Sheets or a webhook) to send them on. To collect every run's new items in one place instead, add "outputDataset": "my-scraper-new-items": they are then written to that named dataset, which grows with each run, and the run's default dataset stays empty.

Integrations and webhooks also pass a payload object. The Actor reads payload.resource.defaultDatasetId from it and adds that dataset to the input, so a plain webhook-style input works too. Prefer datasetIds: Apify runs this Actor with limited permissions, and depending on your account's storage access settings, it may only be allowed to read datasets passed in the Datasets field. If a dataset from payload is refused, the run fails with a message saying so; put the ID in datasetIds instead.

From code, call it after your scraper finishes:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const scrape = await client.actor('<your-scraper>').call({ /* scraper input */ });
const dedupe = await client.actor('fernandosord/dataset-merge-dedupe').call({
    datasetIds: [scrape.defaultDatasetId],
    dedupeFields: ['url'],
    urlFields: ['url'],
    onlyNewItems: true,
    stateKey: 'my-scraper',
});
const { items } = await client.dataset(dedupe.defaultDatasetId).listItems();

Input

Give at least one of Datasets, Run IDs, JSON or CSV URLs or Inline JSON data. Everything else is optional.

Field	What it does
Datasets (datasetIds)	Apify datasets to merge, read in the order given. Pick them or paste IDs.
Run IDs (runIds)	Runs whose default dataset should be read.
JSON or CSV URLs (dataUrls)	Public files: a JSON array, JSON Lines, or CSV with a header row (comma, semicolon, tab or pipe). Up to 50 MB each and about 150 MB in total at the default 1 GB memory.
Inline JSON data (rawData)	Up to 10,000 items as a JSON array.
Compare against datasets (compareDatasetIds)	Records in these datasets are never output, and records that match them by key are removed.
Dedupe key fields (dedupeFields)	Fields that identify a record. Dot paths such as address.city work.
Treat as URLs / emails / phone numbers (urlFields, emailFields, phoneFields)	Key fields that get the extra cleaning described above.
Default phone country (defaultPhoneCountry)	Two-letter code such as US or GB.
Case-sensitive keys (caseSensitive)	Off by default.
Fuzzy match field and Fuzzy similarity threshold (fuzzyField, fuzzyThreshold)	One text field compared by similarity.
Which record to keep (keep)	first, last, most-complete or merge.
Records with an empty key field (missingKeyAction)	keep, drop or compare-as-empty.
Only output items not seen in earlier runs (onlyNewItems) and the fields below it	See the section above. Maximum remembered items (rememberMaxKeys) caps the list; the keys not seen for the longest time are forgotten first.
What to output (outputItems)	unique (default), duplicates (the removed records) or nothing (counts only).
Output dataset name (outputDataset)	Write to a named dataset instead of the run's default dataset. The Actor creates it on the first run and adds to it on later runs.
Output fields, Fields to remove, Rename fields (outputFields, omitFields, renameFields)	Shape the written records. renameFields is an object such as {"title": "name"}.
Write cleaned key values (writeNormalizedValues)	Write the cleaned URL, email or phone instead of the original.
Add a _status field / Add a _source field (addStatusField, addSourceField)	Adds _status (new, changed, unique or duplicate) or _source (the dataset ID or URL a record came from).
Filters and Records must pass (filters, filterMatch)	Keep only records that pass, for example [{"field": "price", "op": "less-than", "value": 50}]. Ops: equals, not-equals, contains, not-contains, starts-with, ends-with, greater-than, greater-or-equal, less-than, less-or-equal, exists, not-exists, matches-regex.
Skip the first items / Maximum items to read (offset, limit)	Counted across all inputs in order.

Output

The kept records go to the run's default dataset, unchanged apart from the output options you chose. This is the output of the example input (the form's prefilled values: seven company records, key website treated as a URL, keep merge), from a test run on Apify:

[
    { "company": "Northwind Traders", "website": "https://www.northwind.example/?utm_source=directory", "email": "Sales@Northwind.example", "phone": "(415) 555-2671", "city": "San Francisco", "employees": 120 },
    { "company": "Contoso Ltd", "website": "https://contoso.example/about", "phone": "212-555-0187", "city": "New York", "email": "hello@contoso.example" },
    { "company": "Fabrikam", "website": "https://fabrikam.example", "email": "info@fabrikam.example", "city": "Austin" },
    { "company": "Tailspin Toys", "website": "https://tailspin.example/shop?gclid=abc123", "city": "Denver", "phone": "720 555 0144" }
]

The two Northwind records matched although one URL had www., a tracking parameter and https, and the other had none of them. The merged record took employees from the second one. The two Contoso records matched on /about and /about/#team.

The run also saves a summary in the OUTPUT record of its key-value store:

{
    "status": "succeeded",
    "inputItems": 7,
    "compareItems": 0,
    "filteredOut": 0,
    "missingKey": 0,
    "missingKeyDropped": 0,
    "duplicatesRemoved": 3,
    "fuzzyDuplicates": 0,
    "matchedCompareData": 0,
    "uniqueItems": 4,
    "newItems": null,
    "changedItems": null,
    "unchangedItems": null,
    "outputItems": 4,
    "stoppedAtCostLimit": false,
    "fuzzyIndexFull": false,
    "keep": "merge",
    "outputMode": "unique",
    "dedupeFields": ["website (url)"],
    "onlyNewItems": null,
    "outputDataset": { "id": "tSPt0JxjtRy1cft20", "name": null },
    "inputs": [{ "type": "rawData", "id": "rawData", "items": 7 }],
    "startedAt": "2026-10-26T02:53:05.367Z",
    "finishedAt": "2026-10-26T02:53:05.639Z",
    "durationSecs": 0.3
}

In only-new mode, newItems, changedItems (changed since last output) and unchangedItems (seen before) are numbers, and onlyNewItems shows the list. From a test run with a remembered list of 2,100 items, where one record was new and two had changed:

"onlyNewItems": {
    "stateKey": "platform-test",
    "rememberedBefore": 2100,
    "lastUpdatedAt": "2026-10-26T02:51:10.805Z",
    "rememberedAfter": 2101,
    "forgotOldest": 0,
    "saved": true
}

status is succeeded, stopped-at-cost-limit (the run reached your maximum cost; see Pricing) or failed (with an error message, for example a dataset ID that doesn't exist or a URL that was refused).

Pricing

You pay per event:

Event	Price
Run start	$0.002 per run
Item processed	$0.00002 per input item read ($0.02 per 1,000)
Item output	$0.00001 per item written ($0.01 per 1,000)

Every input item counts as processed, including duplicates, filtered-out items and items in Compare against datasets. Only written items count as output, so "only new" runs that find nothing new pay only for reading.

Examples:

• Dedupe 10,000 items down to 7,000: $0.002 + $0.20 + $0.07 = $0.272.
• A daily only-new run over 1,000 items that finds 50 new ones: about $0.0225 a day, or $0.68 a month.
• Dedupe 100,000 items down to 70,000: about $2.70.

If you set a maximum cost per run, the Actor reads only as many items as it can also afford to write, then stops and reports stopped-at-cost-limit. Nothing is charged beyond the limit.

Limits

• Up to 100 datasets and runs, 20 URLs and 10,000 inline items per run. Downloaded files are held in memory, so each URL can be up to 50 MB and all of them together up to 15% of the run memory (about 150 MB at the default 1 GB). For more, give the run more memory or load the files into datasets first.
• Datasets are read 10,000 items at a time, so their size is limited by memory for the keys, not by the data. In a local benchmark of 1 million items (700,000 unique) with only-new mode on, memory peaked at 526 MB with Which record to keep set to first, 718 MB with last and 1.03 GB with merge, which holds partly merged records until each group's last duplicate is read. The default run memory is 1 GB. For merge or last on more than about 500,000 items, or only-new mode with more than 1 million remembered items, raise the run memory to 2 GB or more.
• last, most-complete and merge read the input twice. Inputs up to 50,000 items are kept in memory after the first read instead.
• Output order: first keeps the input order. The other modes write each record where its kept version appears (merge: where the group's last record appears).
• Fuzzy matching works within one run, holds up to 300,000 distinct texts, and compares each record with at most 2,000 earlier ones. Between runs, only-new mode compares exact cleaned keys.
• Remembered keys are 96-bit hashes, 20 bytes each in storage. Up to 10 million per state key.
• Dataset fields whose names start with # (hidden fields) are not read.
• Private and local network addresses (such as localhost, 10.x.x.x and 169.254.169.254, also when written as IPv6) are refused, including through redirects. Downloads time out after 120 seconds.
• Run IDs: Apify's limited permissions may block looking up a run by its ID. If that happens, the error says so; paste the run's dataset ID into Datasets instead.
• Output dataset name: use a new name. The Actor creates the dataset and can add to it on later runs, but limited permissions don't let it write to a dataset it didn't create.
• Custom JavaScript functions are not run.

Moving from lukaskrivka/dedup-datasets

This Actor accepts the main input field names of lukaskrivka/dedup-datasets and lukaskrivka/dedup-datasets-limited-permissions, so an existing task or integration needs few changes. It is a separate project and is not affiliated with them.

Their field	Here
datasetIds, offset, limit	Same meaning
fields	dedupeFields
output (unique-items, duplicate-items, nothing)	outputItems (unique, duplicates, nothing)
outputDatasetId	outputDataset (a name)
fieldsToLoad	outputFields
datasetIdsOfFilterItems	compareDatasetIds
nullAsUnique	missingKeyAction: true is keep, false is compare-as-empty
appendDatasetIds	Adds __datasetId__ to each record, like addSourceField
mode, parallelLoads, parallelPushes, uploadBatchSize, batchSizeLoad, verboseLog, customInputData	Ignored; datasets are always read in pages
preDedupTransformFunction, postDedupTransformFunction	Not supported; a warning is logged. Use Filters, Output fields and Rename fields.
outputTo: key-value-store	Not supported; results go to a dataset

FAQ

Why did two records that look the same not match? Check that the field is in Dedupe key fields and has the right type (URL, email or phone). Text keys ignore case and extra spaces but not punctuation; use a fuzzy match field for names that differ in punctuation or spelling.

Can it dedupe on several fields at once? Yes, when all of them must match. "Same email or same phone" needs two runs: first with email as the key, then with phone as the key on the first run's output.

How do I see what was removed? Set What to output to Duplicates, or turn on Add a _source field to see where each kept record came from.

How do I start the only-new list over? Turn on Forget remembered items first for one run, or use a new State key.

Does it change my source datasets? No. They are only read.

Can I call it from an AI agent? Yes, through the Apify API or the Apify MCP server, with the same input as above.

Found a bug or need an option? Open an issue on this Actor's Issues tab.