Dataset Diff: Detect Price & Stock Changes Between Two Scrapes

Compare two Apify datasets (or two JSON arrays) and get only what changed: added records, modified records with before/after values and numeric deltas (price drops, stock changes), and records that disappeared. Built for price monitoring, catalog tracking and any scheduled scraper where you only care about what moved. No scraping, no proxy, no target website.

What does Dataset Diff do?

Every scheduled scraper returns a full snapshot. Dataset Diff takes two snapshots, matches records by a key you choose (id, sku, url, or several fields together) and writes:

• one summary row with counts (added, modified, missing, removed, unchanged)
• one change row per affected record, with the field-level changes (before, after, numericDelta) or the full record for added/removed ones

It reads datasets through the official Apify API with your run's token, so it works after any Actor in the Apify Store or your own.

Who is it for?

• E-commerce and pricing teams: price drops, price increases, back-in-stock and out-of-stock alerts from a daily product scrape
• Marketplace and catalog managers: new and delisted products, edited titles or descriptions
• Lead and data teams: which rows in a list changed since last week
• Automation builders (Make, Zapier, n8n): send only the delta downstream instead of thousands of unchanged rows
• Anyone auditing two exports of the same data

Why use this Actor?

• Missing vs. removed: by default, a record absent from the new snapshot is labelled missing, not removed, so an incomplete scrape does not look like mass deletion. Set snapshotsComplete: true when both crawls are complete.
• Numeric-aware: "$1,299.00" → "$1,249.00" gives numericDelta: -50.0. Tolerance lets you ignore tiny changes.
• Noise control: timestamps like scrapedAt, fetchedAt, crawledAt are ignored by default; you can limit comparison to fields such as price and inStock.
• Composite keys: ["sku", "variant"] works.
• Honest limits: if a snapshot is larger than maxRowsPerSnapshot, the run stops with an error instead of silently comparing a truncated list.
• Fast and small: Python standard library only; the demo run finishes in about 1.5 s.

What data do you get?

Field	On	Meaning
recordType	all rows	summary or change
oldCount, newCount, evaluatedKeys	summary	Sizes of the snapshots and number of distinct keys
added, modified, missing, removed, unchanged	summary	Counts per change type
snapshotsComplete, keyFields, usedDemoData, errors	summary	Settings used and non-fatal issues (duplicate keys, missing key fields)
changeType	change	added / modified / missing / removed / unchanged
key	change	The record key (composite keys are joined with ␟)
changes[]	modified	{field, before, after, numericDelta, beforePresent, afterPresent}
record	added / missing / removed	The full record
comparedAt	all rows	UTC timestamp of the comparison

How much does it cost?

Pay per event: $0.005 per output row plus $0.00005 per run start. Platform usage is included. You pay for rows written (the summary row plus one row per change). Unchanged records are compared but not written, so they cost nothing (unless you turn on includeUnchanged).

Scenario	Rows written	Cost
Demo run (no input)	5	about $0.025
10,000 products, 20 changed	21	about $0.11
10,000 products, 200 changed	201	about $1.01
Daily run, 20 changes a day, 30 days	630	about $3.15

If you set a maximum cost per run, the Actor keeps the summary row first and stops writing change rows at the limit; the summary then shows truncatedByMaxCharge: true and rowsNotReturned.

How to use it

1. Run your scraper twice (for example, yesterday and today). Copy the two dataset IDs from the run's Storage tab.
2. Open Dataset Diff, paste them into Old dataset ID and New dataset ID.
3. Set Key fields to the field that identifies a record (id, sku, url...).
4. Optional: set Numeric fields (price, stock) and Compare only these fields.
5. Click Start. Filter the output by changeType.

Run it with no input first to see the output format on a built-in 3-product demo.

For daily monitoring, schedule your scraper and then this Actor (or call it from a webhook / Make / Zapier / n8n step with the two latest dataset IDs).

Input

Field	Type	Default	Description
oldDatasetId	string	—	Dataset ID of the previous snapshot. Your token must be able to read it.
newDatasetId	string	—	Dataset ID of the current snapshot
oldItems / newItems	array	—	Inline JSON arrays, used when the dataset IDs are empty
keyFields	array	["id"]	Field(s) identifying a record
compareFields	array	[]	Only compare these fields (empty = all except keys and ignored)
ignoreFields	array	["scrapedAt","fetchedAt","crawledAt","#debug"]	Volatile fields to skip
numericFields	array	["price"]	Compared as numbers; currency symbols and thousands separators are tolerated
numericTolerance	integer	0	Ignore numeric changes at or below this absolute difference
snapshotsComplete	boolean	false	Report absent records as removed instead of missing
includeUnchanged	boolean	false	Also write unchanged records
maxRowsPerSnapshot	integer	50000	Safety cap; the run errors instead of truncating

{
  "oldDatasetId": "YESTERDAY_DATASET_ID",
  "newDatasetId": "TODAY_DATASET_ID",
  "keyFields": ["sku"],
  "numericFields": ["price", "stock"],
  "compareFields": ["price", "stock", "inStock"]
}

Output example

Real output from the built-in demo (run with no input on 2026-09-24):

[
  { "recordType": "summary", "oldCount": 3, "newCount": 3, "evaluatedKeys": 4,
    "added": 1, "modified": 2, "missing": 1, "removed": 0, "unchanged": 0,
    "snapshotsComplete": false, "keyFields": ["id"], "errors": [], "usedDemoData": true },
  { "recordType": "change", "changeType": "added", "key": "sku-4", "changes": [],
    "record": { "id": "sku-4", "title": "Mechanical Keyboard", "price": 79.0, "inStock": true } },
  { "recordType": "change", "changeType": "missing", "key": "sku-3", "changes": [],
    "record": { "id": "sku-3", "title": "Laptop Stand", "price": 42.0, "inStock": false } },
  { "recordType": "change", "changeType": "modified", "key": "sku-2",
    "changes": [ { "field": "inStock", "before": true, "after": false, "numericDelta": null,
                   "beforePresent": true, "afterPresent": true } ] },
  { "recordType": "change", "changeType": "modified", "key": "sku-1",
    "changes": [ { "field": "price", "before": 19.99, "after": 17.99, "numericDelta": -2.0,
                   "beforePresent": true, "afterPresent": true } ] }
]

Real output with currency strings and snapshotsComplete: true (inline items, key sku):

{ "recordType": "change", "changeType": "modified", "key": "A1",
  "changes": [ { "field": "price", "before": "$1,299.00", "after": "$1,249.00", "numericDelta": -50.0,
                 "beforePresent": true, "afterPresent": true } ] }
{ "recordType": "change", "changeType": "removed", "key": "B2",
  "record": { "sku": "B2", "price": "$20.00", "stock": 0 } }

(comparedAt and attribution omitted above for brevity.)

Tips

• Pick a key that is stable between runs (an internal id, sku or product URL), not a position or timestamp.
• Price drop = changeType: "modified", field: "price", negative numericDelta.
• Use compareFields to ignore fields that change on every scrape (ratings counts, "viewed by N people").
• Leave snapshotsComplete off unless you trust both scrapes; missing is a softer signal than removed.
• Duplicate keys keep the last occurrence and are reported in the summary errors.

Integrations and API

Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("panda_studio/dataset-diff-change-detector").call(run_input={
    "oldDatasetId": "OLD_ID", "newDatasetId": "NEW_ID", "keyFields": ["sku"], "numericFields": ["price"]})
changes = client.dataset(run["defaultDatasetId"]).list_items().items

Node.js

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('panda_studio/dataset-diff-change-detector').call({
  oldDatasetId: 'OLD_ID', newDatasetId: 'NEW_ID', keyFields: ['sku'], numericFields: ['price'] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

cURL

curl -X POST "https://api.apify.com/v2/acts/panda_studio~dataset-diff-change-detector/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"oldDatasetId":"OLD_ID","newDatasetId":"NEW_ID","keyFields":["sku"]}'

MCP (Claude, Cursor, VS Code): https://mcp.apify.com?tools=panda_studio/dataset-diff-change-detector

Data and privacy

The Actor processes only data you supply (inline JSON or datasets your token can read) through the documented Apify API. It does not visit any website and does not log your input values.

FAQ

Does it scrape anything? No. It only compares data you give it.

Can it read datasets from another Actor's runs? Yes, any dataset ID your Apify token can read, including runs of Store Actors in your account.

Why "missing" and not "removed"? A record can vanish because the product was delisted or because the scrape stopped early. missing avoids false deletion alerts. Set snapshotsComplete: true to get removed.

Does it keep state between runs? No. You pass both snapshots each time, so the result is always reproducible.

How large can the snapshots be? Up to maxRowsPerSnapshot (default 50,000, hard limit 500,000 rows).

Do unchanged records cost money? No, unless you enable includeUnchanged.

Can I compare nested fields? Top-level fields are compared as whole values; a changed nested object shows up as one field change with the full before/after value.

Changelog

• 2026-09-24 — Respects your maximum cost per run (summary row kept first, flags truncatedByMaxCharge). README rewritten with real outputs and cost examples.
• 2026-09-20 — First public version.