Dataset Change Tracker

Compare two runs of your scraper and get only what changed: new items, changed items (with the changed fields and their old values) and removed items. Timestamps and other fields that change on every run can be ignored.

Typical uses:

• Price and stock monitoring: get only the products whose price or availability changed since yesterday.
• New listings: get only the job ads, real-estate offers or articles that appeared since the last run.
• Delisting: find the items that disappeared.
• Smaller downstream work: send only the changes to your spreadsheet, database or notification step.

Try it in one click

Click Start without choosing a dataset. The Actor compares two small built-in sample datasets and shows you the output format: 1 added, 3 changed and 1 removed item. The sample is not charged per item (only Apify's Actor start event applies, see Pricing).

How it works

1. You choose the current dataset (for example the newest run of your scraper).
2. You choose what to compare it with, in one of two ways:
   • Previous dataset: the older dataset. Gives the most detail (changed fields, old values, the full removed items).
   • Remember the last run in this store (stateStoreName): the Actor keeps a fingerprint of each complete run in a key-value store in your account, so the next run needs only the current dataset. The fingerprint holds 64-bit hashes of each item's key and content, not the items themselves (see Privacy).
3. Items are matched by the key fields you choose (for example url, id or sku). Fields listed in Fields to ignore (for example scrapedAt) never count as a change. Hidden fields (names starting with #) are not compared.
4. Before anything is charged, the Actor checks the first 100 items of each dataset and the memory of the run, and it keeps checking the memory while it loads the previous dataset. If the key fields are missing from most items, if both datasets are the same, or if the run does not have enough memory, it stops with a message that says what to change.
5. The Actor writes one record per change to its dataset and a summary with the counts to the OUTPUT record of its key-value store.

Input

Field	What it does
datasetId	Current dataset. Leave empty to try the sample data.
previousDatasetId	Previous dataset to compare with. Optional if stateStoreName is set.
stateStoreName	Optional name of a key-value store where the Actor remembers the last complete run (hashes only). Letters, digits and hyphens; not default.
keyFields	Fields that identify the same item, for example ["url"] or ["id", "size"]. Dots for nested fields: product.id. Empty = compare whole items (added and removed only).
ignoreFields	Fields that should not count as a change, for example ["scrapedAt", "meta.loadedAt"]. A nested path also applies inside lists: offers.seenAt.
changeTypes	Which changes to output: added, changed, removed (default: all). All are counted in the summary.
ignoreArrayOrder	If true, lists with the same elements in another order are not a change. Default false.

Example input:

{
    "datasetId": "YOUR_CURRENT_DATASET_ID",
    "previousDatasetId": "YOUR_PREVIOUS_DATASET_ID",
    "keyFields": ["url"],
    "ignoreFields": ["scrapedAt"],
    "changeTypes": ["added", "changed", "removed"],
    "ignoreArrayOrder": false
}

Output

Each record in the dataset is one change:

{
    "changeType": "changed",
    "key": "https://example.com/products/2",
    "changedFields": ["stock.quantity", "stock.status"],
    "previousValues": { "stock.quantity": 3, "stock.status": "in_stock" },
    "item": {
        "url": "https://example.com/products/2",
        "name": "Office chair",
        "price": 129,
        "stock": { "status": "out_of_stock", "quantity": 0 },
        "scrapedAt": "2026-10-28T06:00:01.000Z"
    }
}

• changeType: added, changed or removed.
• key: the value of the key fields as text. With several key fields it is a JSON list such as ["A1","red"].
• changedFields and previousValues: only for changed items compared with a previous dataset; otherwise null.
• item: the current item for added and changed, the previous item for removed.
• With the saved fingerprint, the Actor knows only hashes of the last run. Changed items then have changedFields: null, and removed items have key: null, item: null and a keyHash: the first 16 hex characters of the SHA-256 of the key text (the same text that key shows for other items). You can compute it for your own keys to find out which item was removed.

The OUTPUT record of the run's key-value store holds the summary. It says "status": "RUNNING" while the run works (it is updated after every page), then COMPLETE or PARTIAL:

{
    "status": "COMPLETE",
    "mode": "datasets",
    "message": "Compared 6 items: 1 added, 3 changed, 1 removed, 2 unchanged. 5 changes written.",
    "counts": {
        "previousItems": 6, "currentItems": 6, "added": 1, "changed": 3, "unchanged": 2, "removed": 1,
        "duplicateKeysPrevious": 0, "duplicateKeysCurrent": 0, "itemsWithoutKeyPrevious": 0, "itemsWithoutKeyCurrent": 0,
        "unsafeIntegerKeys": 0
    },
    "changesWritten": 5,
    "chargedItems": 6,
    "fingerprintSaved": false
}

Run it after your scraper

1. Open your scraper (or its saved task) in Apify Console and go to its Integrations tab.
2. Add this Actor as an integration that runs when the scraper's run succeeds.
3. In the integration's input, set:

{
    "datasetId": "{{resource.defaultDatasetId}}",
    "stateStoreName": "my-shop-tracker",
    "keyFields": ["url"],
    "ignoreFields": ["scrapedAt"]
}

{{resource.defaultDatasetId}} is replaced with the dataset of the scraper run that just finished. Because of stateStoreName, every run is compared with the previous one without picking datasets by hand. Then add your next step (for example a spreadsheet or notification integration) on this Actor's own Integrations tab, so it receives only the changes.

• Keep datasetId in the integration's input. The integration also passes the scraper run in a payload field, but this Actor runs with limited permissions and cannot read a dataset from there. Without datasetId it stops before charging and asks you to add it.
• The first run with a new stateStoreName reports every item as added, because there is nothing to compare with yet.
• Use one stateStoreName per scraper, and don't let two runs use the same name at the same time. If that happens, the last run to finish wins; if the fingerprint ends up unreadable, the Actor stops (before charging) and asks you to use a new name.

Pricing

This Actor uses pay-per-event pricing. The current prices are on the Pricing tab of this Actor.

Event	Price	When it is charged
Actor start	$0.00005 per run (once more for each GB above 1 GB)	Set by Apify, also for the sample data: once for runs with up to 1 GB of memory, plus once for each extra GB (a 4 GB run is charged 4 times). The default memory of this Actor is 1 GB.
Item compared (item-compared)	$0.00002 per item ($0.02 per 1,000 items)	Once for every item read from the current dataset, including items that are skipped because they have no key value or a duplicate key. The previous dataset, the fingerprint and the written changes are not charged separately.

• The cost of a run is predictable: it grows with the size of the current dataset, not with the number of changes.
• Runs that stop at the checks before comparing (key fields missing from most items, the same dataset twice, not enough memory, a reserved or damaged store) are not charged per item.
• Maximum cost per run: if you set one, the Actor counts how many items it can afford before each page and stops cleanly when the limit is reached. The changes found so far are kept and the summary says "status": "PARTIAL"; it is saved before the charge that reaches your limit, because Apify stops a run as soon as its limit is reached. A partial run cannot know which items were removed, so it skips removed items and does not update the fingerprint. The log tells you at the start if your limit does not cover the whole dataset.
• If a run is interrupted (for example moved to another server by Apify), it continues from the last saved page; an item is never charged twice.

Limits

Memory. The previous dataset is held in memory while the current one is read in pages of at most about 4 MB.

• Measured locally (Node.js 24, items of about 350 bytes as JSON, comparison in memory only, without reading through the API):
  • 100,000 items take about 2 seconds and about 65 MB of JavaScript heap.
  • Two datasets, with the JavaScript heap limited to 128, 192 or 256 MB (heap sizes set for the test, not the memory of a run): 150,000, 250,000 or 300,000 items fit (the whole process then used about 400 MB).
  • stateStoreName without a previous dataset: 1,000,000 items fit in a 96 MB heap, plus about 95-130 MB of compact arrays outside the heap; the whole process used about 300-330 MB.
• Measured on Apify (items of about 350 bytes, 1024 MB memory unless noted):
  • A previous dataset of 100,000 items was loaded in about 7 seconds, and the whole run used at most about 150 MB. With 512 MB it worked the same way.
  • Reading and comparing 100,000 current items took about 13 seconds. Most of the time is spent reading the datasets through the API (about 8,000-14,000 items per second).
  • With 256 MB, the run stopped before loading a previous dataset of 100,000 items and asked for 512 MB.
• Estimated from those numbers (not measured on Apify):
  • Two datasets of 100,000 items take about 20 seconds, of 300,000 items about 1 minute, of 500,000 items about 2 minutes.
  • The memory check accepts up to about 90,000 items of this size in a run with 256 MB of memory, 230,000 with 512 MB, 500,000 with the default 1024 MB and 1,100,000 with 2048 MB. Above that, it stops before charging and tells you how much memory to choose.
• At the start, after every page of the previous dataset and once more after it is loaded, the Actor estimates the memory it still needs, both on the JavaScript heap and outside it, and compares it with the memory the run has left. If it is not enough, it stops before any item is charged and tells you how much memory to choose. Bigger items need proportionally more memory.
• One case the check cannot foresee: items that suddenly become much bigger far into a dataset (after thousands of small ones). The Actor reads up to 5,000 items at a time, so one such page can still be too big for the memory, and the run fails. If that happens in the current dataset, the changes of the pages compared before are kept, and those pages are charged.

Partial runs (maximum cost, timeout, abort). A run can end with "status": "PARTIAL" in OUTPUT:

• when it reaches your maximum cost per run (see Pricing);
• 30 seconds before the run's timeout: the Actor stops itself so that it can still save the changes found so far and the summary;
• when you abort the run;
• rarely, when Apify moves a run to another server and the restarted run no longer fits in its memory: the changes found before the move are kept and charged, and the run fails with a message that suggests more memory.

At the maximum cost and at the timeout, the run itself ends as succeeded (an aborted run is shown as aborted). So:

• an integration that starts on "run succeeded" (for example a spreadsheet or notification step) also starts after a partial run, and works with the changes found up to that point; check status in OUTPUT if your next step needs the full result;
• removed items are not reported in a partial run;
• with stateStoreName, a partial run does not save the fingerprint, so the next run compares with the same older fingerprint and reports the same changes again. Give the run a longer timeout or a higher maximum cost to finish.

Matching.

• Key values are compared as text, so 123 and "123" are the same key. Integer IDs above 2^53 (for example 12345678901234567890) cannot be kept exactly by JSON numbers, so different IDs may look the same; they are counted in unsafeIntegerKeys. Use a text field as the key if you can.
• Items with a missing, null or empty key field are skipped and counted (itemsWithoutKeyCurrent, itemsWithoutKeyPrevious). If a key appears more than once, the first item is used and the others are counted (duplicateKeysCurrent, duplicateKeysPrevious). Both are read and charged; if fewer than half of the first 100 items have a key, the run stops before charging.
• A field set to null and a missing field are different values.
• Lists are compared as a whole: changedFields names the list, not the position inside it.
• A field literally named a.b and a nested field a → b have the same dotted path; avoid keys and ignore rules that depend on telling them apart.
• The fingerprint is tied to the key fields, ignored fields and list-order setting. If you change them, use a new stateStoreName; the Actor stops with a clear message instead of reporting false changes.
• The Actor only reads datasets that you choose in the input (it runs with limited permissions).

Privacy

• The Actor does not keep your data. It reads the datasets you choose, writes the changes to its own run dataset in your account, and stops.
• The optional fingerprint holds 64-bit hashes of each item's key and content, plus the settings and counts. It is stored in your own account, in the key-value store you name; delete the store to delete it. The items cannot be read back from the hashes, but the hashes are not salted: someone who can open the store can confirm a key that is easy to guess (for example a URL pattern or a sequential ID) by hashing guesses.
• The Actor does not contact any website or service outside your Apify account.

Example tasks

Ready-made inputs for common jobs. Open one, then click Try for free to copy it to your account and change the input:

• Get new, changed and removed items between two scraper runs
• Track price and stock changes between two product scrapes
• Get only the new items from a scheduled scraper
• Find listings that disappeared between two scraper runs

More from Quiet Works Lab

• Image Converter & Compressor: convert images to WebP, AVIF, JPEG or PNG, resize them and remove EXIF and GPS metadata.
• Japanese Text Normalizer & RAG Chunker: clean Japanese text before an LLM or RAG and split it into sentence-aware chunks.

Support

This Actor is developed and maintained with the help of AI (Claude Code). Issues are answered within a few days.

Please open an issue on the Issues tab with the run ID and, if possible, a few example items (remove anything private first).