Dataset Diff, Dedupe & Merge - Change Detection

Run your scraper every day, then run this Actor to get only what changed: new rows, changed rows (with the list of changed fields and their previous values) and removed rows. Also deduplicates and merges Apify datasets, CSV, TSV, JSON and JSON Lines files by any key fields.

Typical uses: price and stock monitoring, new job/real-estate/product listings alerts, catalogue sync, cleaning duplicated scraping results before export.

How it works

1. Pick your data: Apify datasets (resource picker; only the datasets you select are shared, read-only), file URLs, or inline JSON.
2. Choose key fields (e.g. url, sku, product.id).
3. Pick a mode
   • diff + snapshot name: compares with the previous run's data and saves today's data for tomorrow. The first run only saves the baseline (free).
   • diff + baseline: compares against datasets/files you pick (e.g. yesterday's run dataset).
   • dedupe: removes duplicates (keep first or last).
   • merge: combines all sources, latest row per key wins.

Use ignoreFields (e.g. scrapedAt) or compareFields (e.g. only price, inStock) so noise doesn't count as change.

Output (diff mode)

Each output row is the full current row plus:

Field	Meaning
_change	new, changed, removed (or unchanged if requested)
_changedFields	e.g. ["price"]
_previous	previous values of changed fields, e.g. {"price": 20}

A SUMMARY record in the run's key-value store has counts. Real example: listing/output-example.json for listing/input-example.json. Plug the dataset into Slack/email/Make/Zapier integrations or a webhook to alert only on changes.

Pricing (pay per event)

Event	Price
Row processed (input rows of the current data; baseline rows are free)	$0.00001 ($0.01 per 1,000)
Change detected (diff mode, per new/changed/removed row output)	$0.0005
Actor start (Apify standard, once per run)	$0.0003

Example: a daily 10,000-row scrape with 150 changes costs $0.10 + $0.075 = $0.175 per day (plus the $0.0003 Actor start). Dedupe/merge of 10,000 rows costs $0.10. Unchanged rows and the first snapshot run's baseline cost nothing extra. If you set a max charge per run that cannot cover all rows, a diff refuses to start (a partial diff would report false removals); if the budget runs out while outputting changes, the snapshot is not advanced, so no change is lost.

Notes

• Snapshots live in a named key-value store dataset-diff-snapshots in your account (gzip; full rows up to ~8 MB compressed, otherwise fingerprints only, in which case _changedFields is null).
• File URLs: only submit files you have the right to process; robots.txt is respected by default and private network addresses are refused.
• The Actor runs with limited permissions: it can read only the datasets you select.
• Default memory 512 MB. Our tests used small inputs; very large datasets (hundreds of thousands of wide rows) may need more memory (max 8 GB). maxRows default 1,000,000.
• Data: only the datasets and files you provide. No third-party data sources.

No warranty

Provided as is, without warranty of any kind. Results are automated and can be incomplete or wrong; check anything important against the original source before relying on it.