Dataset Diff: See What Changed & Why

Compare two Apify datasets (or today's data against the last saved snapshot) by key, and get back only what changed: new records, removed records and changed records, each explained in plain English.

Re-running a scraper every day gives you two big datasets and a diff that is mostly noise: whitespace, timestamps, 19.99 versus "19.99", reordered lists. This Actor ignores that noise, tells you what it ignored, and reports the real changes as ready-to-filter records such as price decreased 20.0% (19.99 -> 15.99). It never scrapes anything, and it keeps nothing after the run except a snapshot in your own key-value store, if you ask for one.

What you get

• Only the real changes. New, removed and changed records, with before/after values, numeric deltas (delta, deltaPct, direction), text similarity and list additions/removals.
• Noise control with receipts. Whitespace, case (optional), numeric strings ("$1,299.00" equals 1299), ISO dates, empty-versus-missing, reordered lists, timestamps and run metadata (scrapedAt, crawledAt, #debug, ...) are normalized or ignored. The run summary counts what was suppressed and lists the fields it ignored.
• Alert rules. Say what matters ("price dropped 10% or more", "stock became 0", "title changed") and matching records get alerts[] and a severity you can filter on in Zapier, Make, n8n or a Slack integration.
• A key you do not have to guess. Leave the key empty and the Actor picks one (and says why), or stops with an explanation. URL keys ignore http/https, www, tracking parameters, trailing slashes and parameter order. Composite keys, duplicate and missing-key policies are supported.
• Snapshot mode. Give a snapshot name and the Actor compares today's data with the last run and saves today's data as the new snapshot. The first run only creates the snapshot and is free.
• A free report. REPORT.html and REPORT.md in the key-value store: counts, top alerts, biggest numeric movers, what was ignored.

How to use it

1. Pick your two datasets (current and baseline), or give a snapshot name and one current dataset.
2. Set the key (for example url or sku), or leave it empty for auto-detection.
3. Run it. Read the changes from the Dataset tab, or connect the run to a Slack, Zapier, Make or n8n integration and filter on severity or alerts.

Prefilled example input (try it as is):

{
  "keyFields": ["sku"],
  "currentRecords": [
    { "sku": "A-100", "title": "Acme Mug", "price": 15.99, "stock": 0, "tags": ["new", "sale"], "scrapedAt": "2026-10-24T08:00:00Z" },
    { "sku": "B-200", "title": "Blue Pen", "price": 2.5, "stock": 40, "tags": [], "scrapedAt": "2026-10-24T08:00:00Z" },
    { "sku": "D-400", "title": "Desk Lamp", "price": 30, "stock": 7, "tags": ["eco"], "scrapedAt": "2026-10-24T08:00:00Z" }
  ],
  "baselineRecords": [
    { "sku": "A-100", "title": "Acme Mug", "price": 19.99, "stock": 12, "tags": ["new"], "scrapedAt": "2026-10-23T08:00:00Z" },
    { "sku": "B-200", "title": "Blue Pen", "price": 2.5, "stock": 40, "tags": [], "scrapedAt": "2026-10-23T08:00:00Z" },
    { "sku": "C-300", "title": "Cable", "price": 5, "stock": 9, "tags": [], "scrapedAt": "2026-10-23T08:00:00Z" }
  ],
  "rules": [
    { "id": "price-drop-10", "field": "price", "when": "decreased", "minPercent": 10, "severity": "high", "label": "Price dropped 10%+" },
    { "id": "oos", "field": "stock", "when": "became", "value": 0, "severity": "high" }
  ]
}

That run returns three items (a new record, a removed record and one changed record). The changed one, from a real local run:

{
  "changeType": "changed",
  "key": "[\"a-100\"]",
  "keyParts": { "sku": "A-100" },
  "matchedBy": "key",
  "matchConfidence": 1,
  "severity": "high",
  "summary": "price decreased 20.0% (19.99 -> 15.99); stock 12 -> 0; tags: +1 (sale)",
  "alerts": [
    { "rule": "price-drop-10", "field": "price", "severity": "high", "message": "Price dropped 10%+: price decreased 20.0% (19.99 -> 15.99)" },
    { "rule": "oos", "field": "stock", "severity": "high", "message": "stock became 0 (was 12)" }
  ],
  "changedFieldNames": ["price", "stock", "tags"],
  "fieldChanges": [
    { "field": "price", "op": "changed", "type": "number", "from": 19.99, "to": 15.99, "delta": -4, "deltaPct": -20.01, "direction": "decreased", "explanation": "price decreased 20.0% (19.99 -> 15.99)" },
    { "field": "stock", "op": "changed", "type": "number", "from": 12, "to": 0, "delta": -12, "deltaPct": -100, "direction": "decreased", "explanation": "stock 12 -> 0" },
    { "field": "tags", "op": "changed", "type": "set", "added": ["sale"], "removed": [], "explanation": "tags: +1 (sale)" }
  ],
  "before": { "price": 19.99, "stock": 12, "tags": ["new"] },
  "after": { "price": 15.99, "stock": 0, "tags": ["new", "sale"] },
  "baselineIndex": 0,
  "currentIndex": 0
}

Note that scrapedAt differs on every record and produced no change: it is on the built-in ignore list.

Monitor a scraper every day (snapshot mode)

1. Run your scraper on a schedule.
2. Schedule this Actor right after it with currentDatasetId set to the scraper's dataset (in an integration or through the API) and a snapshotName such as daily-prices.
3. The first run saves the snapshot and returns nothing. Every later run returns only what changed since the previous run, then rolls the snapshot forward.

Snapshots are stored in a key-value store named dataset-diff-snapshots in your own account. You can delete the store at any time. A run that stops early (for example at your maximum cost per run) does not update the snapshot, so no change is lost. updateSnapshot: false gives a dry run.

To stop a failed scrape from showing every record as removed, set abortIfShrinksPercent (for example 50). Without it you still get a warning when the new dataset has fewer than half the records of the old one.

Chain it after other Actors

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

// 1. Run your scraper (replace with the Actor and input you use)
const scraperRun = await client.actor('<your-scraper>').call({ /* scraper input */ });

// 2. Diff its dataset against the last snapshot
const diffRun = await client.actor('hiredhandai/dataset-diff-explainer').call({
  currentDatasetId: scraperRun.defaultDatasetId,
  snapshotName: 'competitor-prices',
  keyFields: ['url'],
  rules: [{ field: 'price', when: 'decreased', minPercent: 10, severity: 'high' }],
});

// 3. Read only the high-severity changes
const { items } = await client.dataset(diffRun.defaultDatasetId).listItems();
console.log(items.filter((c) => c.severity === 'high'));

It works with any Actor that writes records to a dataset. Put hiredhandai/lead-data-cleaner in front of it for lead lists, so that cleaning differences do not show up as changes.

Input

Field	What it does
currentDatasetId / currentRecords	The newer data: an Apify dataset, or up to 10,000 inline records.
baselineDatasetId / baselineRecords / snapshotName	The older data. Use exactly one of the three.
updateSnapshot	Save today's data as the new snapshot after a complete run (default on).
keyFields	Field(s) that identify a record. Empty = auto-detect, with the reason in the summary.
keyNormalization, keyCaseSensitive, fallbackKeyFields	How keys are compared, and a backup key for records missing a part.
duplicateKeyPolicy, missingKeyPolicy	What to do with repeated keys and with records that have no key.
fuzzyMatch	Optional re-pairing of a removed and a new record with almost the same text.
compareFields, ignoreFields, ignoreVolatileFields	Which fields count. Wildcards * and ** are supported.
arrayKeys, arrayMode	Match list items by a key (variants[sku=B2].stock), or compare lists as sets or in order.
normalize, numericTolerance	Noise control: trimming, case, numeric strings, dates, tiny numeric movements.
rules, watchFields	Alert rules and the fields listed first in each summary.
emit, alertsOnly, outputMode, includeRecords, includeSuppressed	What is returned (and charged), and in which shape.
abortIfShrinksPercent, strategy, maxRecordsPerSide, maxChangesReported	Safety limits and memory strategy.

Rule when values: changed, increased, decreased, became (with value), became_empty, became_nonempty, added, removed, matches (with pattern), crossed (with threshold and direction). minPercent and minAbsolute apply to numeric changes. Use the field $record with added or removed to flag new and disappeared records.

Output

• Dataset: one item per added, removed or changed record, as shown above. With outputMode: "fieldChanges" you get one item per changed field, which is easier to put in a spreadsheet.
• OUTPUT (key-value store): status, counts, the key and why it was chosen, top changed fields, numeric movement, what was suppressed and ignored, the strategy used, warnings, and the charges of the run.
• REPORT.html and REPORT.md: the same in a human-readable form.

How it decides

• Keys. Key values are compared after normalization: text keys ignore case and spacing; URL keys ignore scheme, www, tracking parameters, fragments, trailing slashes and parameter order. The number 123 and the text "123" are the same key; "007" stays "007". Composite keys are collision-safe.
• Auto-detect. Candidates are fields present in at least 99% of records on both sides. Common identifier names (url, id, sku, asin, slug, ...) are preferred, and a candidate needs a score of 0.98 or more (presence times uniqueness). Fields on the built-in ignore list (scrapedAt, timestamp, ...) are never chosen. If nothing qualifies, the run stops and lists the best candidates instead of guessing.
• Values. Text is trimmed and whitespace-collapsed, NFKC-normalized and repaired when it was garbled by encoding errors (Café). Numeric strings with currency symbols, thousands separators and percent signs are read as numbers; "1,234" is 1234 and "12,5" is 12.5; ids with leading zeros are left alone. ISO dates are compared as instants; a date-only value never equals a date-time value. null, empty text, [], {} and missing are equal by default.
• Tolerance. numericTolerance ignores movements below an absolute and/or relative size, per field if you like. The summary counts how many were ignored.
• Lists. By default lists are compared as sets, so reordering is not a change. With arrayKeys, items are matched by key and reported as variants[sku=B2].stock.
• Renamed records. With fuzzyMatch, a removed and a new record are paired only if their text similarity clears your threshold, they are each other's best match, they beat every runner-up by a margin, and all numbers inside the text are identical (Store 12 is never paired with Store 21).
• Large datasets. Small datasets are compared in memory. Larger ones use a multi-pass method that keeps one short fingerprint per record, so memory stays flat. Both methods return identical results (checked on 200 random datasets in the test suite). On a development machine, 100,000 against 100,000 records took about 9 to 12 seconds with the low-memory method. Timings on the platform depend on the memory you give the run.
• Limits. 500,000 records per dataset by default (maxRecordsPerSide), 100,000 reported changes per run (maxChangesReported), records larger than 100,000 characters are skipped and counted (their key is excluded from both sides, so they never appear as fake additions or removals). No Excel, CSV or Google Sheets input: use a dataset or inline JSON.

Pricing

Pay per event. Prices:

Event	Price
Actor start	$0.00005 (Apify default)
Compared record (each record in the current dataset)	$0.0001 ($0.10 per 1,000)
Reported change (each added, removed or changed record returned)	$0.002 ($2.00 per 1,000)

Worked examples per 1,000 records compared: 1% churn costs about $0.12, 5% churn about $0.20, 20% churn about $0.50, and if every record changed $2.10. The first run of a snapshot compares nothing and is free. Changes you filter out with emit or alertsOnly are not returned and not charged. The report is free.

Your maximum cost per run is respected: the Actor stops before spending more, returns exactly what it could afford, says so in the run status, and does not update a snapshot after a partial run. If the limit cannot even cover comparing the current dataset, the run stops at the start and charges nothing.

FAQ

Does it scrape anything? No. It only reads the datasets you point it at.

Does it store my data? No. The only thing that persists is a snapshot, in your own key-value store named dataset-diff-snapshots, and only if you use a snapshot name.

Why was a record I expected not reported? Most likely it differed only by noise: whitespace, case (if caseInsensitive), a numeric movement within numericTolerance, list order, or an ignored field. Check the suppressed block and ignoredFields in OUTPUT, and use includeSuppressed for notes on changed records.

The first run returned nothing. In snapshot mode the first run only saves the snapshot. Run it again later.

How do I reset a snapshot? Delete the records starting with MANIFEST-<name> and SNAP-<name>- from the dataset-diff-snapshots store, or use a new name.

What if the key changed for some records? Turn on fuzzyMatch to pair a removed and a new record by similar text, or choose a more stable key.

Support

Found a bug or a case that is not handled? Open an issue from the Actor page; issues get a reply within 48 hours. See CHANGELOG.md for what changed in each version. If the Actor saves you time, an honest rating on the Actor page helps others find it.

Built by hiredhandai. Related Actors by hiredhandai: Lead Data Cleaner, Deduper & Scorer.