Dataset Dedup, Merge & Compare: Only New Items, Reconcile

Remove or merge duplicates across Apify datasets, get only items that are new since earlier runs, or compare two datasets (or CSV files) and see exactly what changed. It runs with limited permissions and works on your private datasets: pick them in the input, or pass {{resource.defaultDatasetId}} from an integration.

Use it after any scraper: dedupe Google Maps leads by phone or website domain, merge overlapping runs into one complete row, send only new job or real-estate listings to Slack every day, or reconcile a price list or bank export against last month's.

What it does

• Merge datasets: combine any number of datasets, in the order you list them.

• Deduplicate: by one or more fields (url, title + company, nested location.city), or by the whole item. Choose which copy survives:

  • First (fastest, single streaming pass)
  • Last (newest data wins)
  • Merge duplicates: all copies become one row with the most complete value per field. Empty fields are filled from other copies, lists such as emails are combined, and the row records its sources (_mergedCount, _mergedFrom, _filledFrom, _conflictingFields).

• Normalized matching (optional):

  • ignore case, extra whitespace and accents
  • URLs: ignores http/https, www., trailing slash, tracking parameters and parameter order
  • phone numbers: digits only, so +60 12-345 6789 = 012-345 6789
  • domain only: https://www.acme.com/contact = sales@acme.com = acme.com

• Only new items, across runs: give a memory name (e.g. daily-jobs). The Actor remembers every item it has output under that name, in a key-value store in your own account, and later runs output only items it has never seen. You can also exclude items found in specific datasets.

• Compare & reconcile two datasets or CSV/JSON files: match rows by key fields (invoiceNo=reference when names differ) and get one row per record with status:

  • matched
  • different, with field-level differences and the numeric or date gap
  • only_in_a
  • only_in_b

  Number tolerance (absolute or %) and date tolerance (days) let rounding and posting delays count as equal.

• Filter and transform: equals, contains, greater/less than, regex, exists, or in a list; keep, drop or rename fields; tag rows with their source dataset.

• Reports:

  • REPORT is an HTML report: duplicate clusters and field fill rates in dedup mode, a reconciliation summary in compare mode.
  • RUN_SUMMARY (JSON) has all counts, data-quality stats, memory size, errors by code and the events charged.

Built for large datasets and limited permissions

• Streams datasets page by page, prefetching the next page. Keys are stored as compact hashes.

• In a local test with 1,000,000 rows (25% duplicates):

  • Keep first: 6 s, about 230 MB of memory
  • Keep last: 11 s, about 280 MB
  • Merge duplicates: 15 s, about 600 MB
  • Excluding against a 750,000-key memory: about 235 MB

  On the platform, reading and writing through the Apify API takes longer than this local test. The default memory is 1 GB.

• Datasets you pick in the input are readable under limited permissions, so no full-permission approval is needed. This matters for schedules, integrations and AI agents.

• If a dedup run of another tool failed with "Insufficient permissions for the dataset", it was most likely given a private dataset ID as plain text under limited permissions. Here the dataset fields are dataset pickers, which grant read access to exactly the datasets you choose.

• Dataset IDs, run IDs and Console URLs pasted into Sources as text are recognised. Under limited permissions, text gives no access to private data, so for those you get a clear error record that says which dataset ID to pick instead. Public datasets work directly.

Try it free

• Dedup demo: run with no datasets selected. It processes a public sample dataset (40 fictional products, 10 duplicate URLs) and outputs 30 unique rows, free.
• Compare demo: set What to do to Compare & reconcile and leave both sides empty. It compares last week's and this week's sample products (changed prices, stock and titles, plus removed and new items), free.

Input examples

Only new listings after every scraper run (add as an integration on your scraper's task):

{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "dedupFields": ["url"],
  "normalizeUrls": true,
  "memoryName": "daily-listings"
}

Merge several runs of the same scraper (e.g. overlapping Google Maps searches) into one complete row per place:

{
  "datasetIds": ["<run1DatasetId>", "<run2DatasetId>", "<run3DatasetId>"],
  "dedupFields": ["placeId"],
  "keep": "merge"
}

Reconcile invoices against payments (amounts within 0.01, dates within 3 days):

{
  "mode": "compare",
  "fileUrlA": "https://example.com/invoices.csv",
  "fileUrlB": "https://example.com/bank-export.csv",
  "compareKeyFields": ["invoiceNo=reference"],
  "compareFields": ["amount=paidAmount", "dueDate=paidOn"],
  "numericTolerance": 0.01,
  "dateToleranceDays": 3,
  "includeMatched": false
}

Output

Rows go to the run's default dataset, or to a dataset you pick in Write to existing dataset. Export them as JSON, CSV, Excel or HTML, or fetch them via the API. Dataset views:

• Results: everything
• Compare: summary per row
• Compare: details
• Merge: source tracking
• Errors

A compare row looks like this:

{ "status": "different", "key": { "url": "https://shop.example.com/p/2" }, "differenceCount": 1, "differentFields": "price",
  "differences": [{ "field": "price", "a": 42.99, "b": 38.69, "kind": "number", "difference": -4.3, "percent": -10.0023 }],
  "a": { "...": "row from A" }, "b": { "...": "row from B" } }

Pricing

Pay per event, with no separate platform-usage charges:

• $0.20 per 1,000 output rows (output-row, $0.0002 per row written: a unique or merged item, or a comparison row)
• $0.001 per run (run, charged once, after the input is validated)

These are free: removed duplicates, filtered rows, already-seen rows, error records and demo runs. In compare mode, turning off Output identical rows means you pay only for differences and unmatched rows.

Examples:

• Merging 50,000 rows into 30,000 unique rows costs $6.00 + $0.001.
• A daily "only new" run that finds 40 new items costs $0.009.

If you set a maximum cost per run, the Actor writes only as many rows as fit and then stops cleanly. RUN_SUMMARY shows when that happened.

How to remove duplicates from an Apify dataset

1. Pick one or more datasets under Datasets to merge.
2. Under Dedup by fields, enter the fields that identify a duplicate, e.g. url, or phone with Normalize phone numbers on.
3. Run it. Unique rows go to the dataset. REPORT shows the duplicate clusters and how complete each field is.

How to get only new items since the last run

Set a memory name and run the Actor after each scraper run, with {{resource.defaultDatasetId}} in an integration. The first run outputs everything and remembers it; later runs output only new items. To start over, tick Reset memory or use a new name. The memory is the key-value store dedup-memory-<name> under Storage in your account.

How to compare two datasets or CSV files

Choose Compare & reconcile, pick dataset A and B (or paste public CSV/JSON URLs), set the key fields, and optionally the fields to compare and the tolerances. Filter the output by status, or open REPORT for totals, differences per field and the first 100 examples of each kind. Rows are matched one-to-one. When a key appears more than once, the closest pairs are matched first and extra rows are listed as only in A or B (many-to-one matching is not supported yet).

FAQ

Does it need full permissions? No. It reads only the datasets you pick, writes to its own storage (or to a dataset you pick with write access), and keeps memory in a key-value store it creates in your account.

Is matching fuzzy? It's normalized, not similarity-based. Values match when they're equal after the normalization options you choose. That avoids merging rows that only look alike.

How is "most complete" decided when merging? Non-empty values beat empty ones, lists are combined without repeats, and nested objects are merged. When two copies have different non-empty values, the first one (in dataset order) is kept and the field is listed in _conflictingFields.

What about rows where the dedup field is empty? They can't be matched reliably, so by default they're kept as separate rows and counted as emptyKeyRows in RUN_SUMMARY (and they're not stored in memory). To treat them as duplicates of each other, turn on Treat empty keys as duplicates.

How big can datasets be? Dedup modes stream, so millions of rows work on the default 1 GB. Merge mode keeps only open duplicate groups in memory. Compare mode keeps the smaller side in memory: a few hundred thousand rows per side fit in 1 GB, and for more you can raise memory.

Troubleshooting

A source that can't be read doesn't stop the run. Each problem is written as an error record to the run's default dataset, with isError: true, an errorCode, a plain-English errorMessage and the offending input. The Errors view shows them. Error records are free, and counts per code are in RUN_SUMMARY.

Code	Meaning	What to do
INVALID_INPUT	Malformed input: a bad filter, a bad ID, a non-URL in a file field. Fatal problems stop the run with this code in the status message.	Fix the field named in the message.
DATASET_NOT_ACCESSIBLE	A dataset doesn't exist, or wasn't picked in a dataset field (text gives no access under limited permissions).	Pick the dataset with the dataset picker, or check the ID.
RUN_ID_NOT_SUPPORTED	A run ID or run URL was given. Limited-permission Actors can't look up your runs.	Open the run → Storage tab → copy the Dataset ID and pick it. In integrations use {{resource.defaultDatasetId}}.
FILE_READ_FAILED	A CSV/JSON URL couldn't be downloaded or parsed.	Make sure the URL is public (no login) and the file is CSV, JSON or JSON Lines.
DATASET_READ_FAILED	Reading stopped partway through (API error after retries).	Re-run. If it keeps happening, report it.
OUTPUT_NOT_WRITABLE	The output dataset can't be opened for writing.	Pick it in Write to existing dataset, or leave it empty.
MEMORY_NOT_AVAILABLE / MEMORY_NOT_SAVED	The memory store couldn't be opened or saved.	Try another memory name. If it persists, report it.
FIELD_NOT_FOUND (warning)	A dedup, filter or key field never appears. It's usually a typo or a field the scraper renamed.	Check the field path, e.g. location.city.
MEMORY_CONFIG_CHANGED (warning)	Dedup fields or matching options changed since the memory was created.	Use a new memory name to start clean.
DUPLICATE_KEYS (warning)	In compare mode, some keys appear more than once.	Add key fields to make keys unique.
NO_KEY_FIELDS (warning)	Compare without key fields only matches identical rows.	Set key fields.
DEMO_MODE (warning)	Nothing was selected, so the free demo ran.	Select your data.
DATASET_EMPTY (warning)	A selected dataset or file has no rows.	None.
MAX_CHARGE_REACHED (warning)	Your maximum cost per run was reached and the run stopped early.	Raise the limit.
INTERNAL_ERROR	An unexpected bug.	Please report it (see below).

Found a bug or a wrong result? Open an Issue on this Actor's page in Apify Store (the Issues tab) and describe what you expected. Please paste the run URL into the issue: Apify then shares that run (log, input, dataset and RUN_SUMMARY) with the developer automatically, and only that run. Alternatively, enable Settings → Login & Privacy → Share run data with developers for this Actor. The Actor sends no telemetry anywhere; everything stays in your own run storage.