# Dataset Change Tracker (`quietworkslab/dataset-change-tracker`) Actor

Compare two runs of your scraper and get only the new, changed and removed items. Ignores fields you choose, shows which fields changed and their old values, and can remember the last run by storing only hashes of the items.

- **URL**: https://apify.com/quietworkslab/dataset-change-tracker.md
- **Developed by:** [Quiet Works Lab](https://apify.com/quietworkslab) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.01 / 1,000 item compareds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Change Tracker

Compare two runs of your scraper and get **only what changed**: new items, changed items (with the changed fields and their old values) and removed items. Timestamps and other fields that change on every run can be ignored.

Typical uses:

- Price and stock monitoring: get only the products whose price or availability changed since yesterday.
- New listings: get only the job ads, real-estate offers or articles that appeared since the last run.
- Delisting: find the items that disappeared.
- Smaller downstream work: send only the changes to your spreadsheet, database or notification step.

### Try it in one click

Click **Start** without choosing a dataset. The Actor compares two small built-in sample datasets and shows you the output format: 1 added, 3 changed and 1 removed item. The sample is not charged per item (only Apify's Actor start event applies, see Pricing).

### How it works

1. You choose the **current dataset** (for example the newest run of your scraper).
2. You choose what to compare it with, in one of two ways:
   - **Previous dataset**: the older dataset. Gives the most detail (changed fields, old values, the full removed items).
   - **Remember the last run in this store** (`stateStoreName`): the Actor keeps a fingerprint of each complete run in a key-value store in your account, so the next run needs only the current dataset. The fingerprint holds 64-bit hashes of each item's key and content, not the items themselves (see Privacy).
3. Items are matched by the **key fields** you choose (for example `url`, `id` or `sku`). Fields listed in **Fields to ignore** (for example `scrapedAt`) never count as a change. Hidden fields (names starting with `#`) are not compared.
4. Before anything is charged, the Actor checks the first 100 items of each dataset and the memory of the run, and it keeps checking the memory while it loads the previous dataset. If the key fields are missing from most items, if both datasets are the same, or if the run does not have enough memory, it stops with a message that says what to change.
5. The Actor writes one record per change to its dataset and a summary with the counts to the `OUTPUT` record of its key-value store.

### Input

| Field | What it does |
|---|---|
| `datasetId` | Current dataset. Leave empty to try the sample data. |
| `previousDatasetId` | Previous dataset to compare with. Optional if `stateStoreName` is set. |
| `stateStoreName` | Optional name of a key-value store where the Actor remembers the last complete run (hashes only). Letters, digits and hyphens; not `default`. |
| `keyFields` | Fields that identify the same item, for example `["url"]` or `["id", "size"]`. Dots for nested fields: `product.id`. Empty = compare whole items (added and removed only). |
| `ignoreFields` | Fields that should not count as a change, for example `["scrapedAt", "meta.loadedAt"]`. A nested path also applies inside lists: `offers.seenAt`. |
| `changeTypes` | Which changes to output: `added`, `changed`, `removed` (default: all). All are counted in the summary. |
| `ignoreArrayOrder` | If `true`, lists with the same elements in another order are not a change. Default `false`. |

Example input:

```json
{
    "datasetId": "YOUR_CURRENT_DATASET_ID",
    "previousDatasetId": "YOUR_PREVIOUS_DATASET_ID",
    "keyFields": ["url"],
    "ignoreFields": ["scrapedAt"],
    "changeTypes": ["added", "changed", "removed"],
    "ignoreArrayOrder": false
}
```

### Output

Each record in the dataset is one change:

```json
{
    "changeType": "changed",
    "key": "https://example.com/products/2",
    "changedFields": ["stock.quantity", "stock.status"],
    "previousValues": { "stock.quantity": 3, "stock.status": "in_stock" },
    "item": {
        "url": "https://example.com/products/2",
        "name": "Office chair",
        "price": 129,
        "stock": { "status": "out_of_stock", "quantity": 0 },
        "scrapedAt": "2026-10-28T06:00:01.000Z"
    }
}
```

- `changeType`: `added`, `changed` or `removed`.
- `key`: the value of the key fields as text. With several key fields it is a JSON list such as `["A1","red"]`.
- `changedFields` and `previousValues`: only for changed items compared with a previous dataset; otherwise `null`.
- `item`: the current item for `added` and `changed`, the previous item for `removed`.
- With the saved fingerprint, the Actor knows only hashes of the last run. Changed items then have `changedFields: null`, and removed items have `key: null`, `item: null` and a `keyHash`: the first 16 hex characters of the SHA-256 of the key text (the same text that `key` shows for other items). You can compute it for your own keys to find out which item was removed.

The `OUTPUT` record of the run's key-value store holds the summary. It says `"status": "RUNNING"` while the run works (it is updated after every page), then `COMPLETE` or `PARTIAL`:

```json
{
    "status": "COMPLETE",
    "mode": "datasets",
    "message": "Compared 6 items: 1 added, 3 changed, 1 removed, 2 unchanged. 5 changes written.",
    "counts": {
        "previousItems": 6, "currentItems": 6, "added": 1, "changed": 3, "unchanged": 2, "removed": 1,
        "duplicateKeysPrevious": 0, "duplicateKeysCurrent": 0, "itemsWithoutKeyPrevious": 0, "itemsWithoutKeyCurrent": 0,
        "unsafeIntegerKeys": 0
    },
    "changesWritten": 5,
    "chargedItems": 6,
    "fingerprintSaved": false
}
```

### Run it after your scraper

1. Open your scraper (or its saved task) in Apify Console and go to its **Integrations** tab.
2. Add this Actor as an integration that runs when the scraper's run succeeds.
3. In the integration's input, set:

```json
{
    "datasetId": "{{resource.defaultDatasetId}}",
    "stateStoreName": "my-shop-tracker",
    "keyFields": ["url"],
    "ignoreFields": ["scrapedAt"]
}
```

`{{resource.defaultDatasetId}}` is replaced with the dataset of the scraper run that just finished. Because of `stateStoreName`, every run is compared with the previous one without picking datasets by hand. Then add your next step (for example a spreadsheet or notification integration) on this Actor's own **Integrations** tab, so it receives only the changes.

- Keep `datasetId` in the integration's input. The integration also passes the scraper run in a `payload` field, but this Actor runs with limited permissions and cannot read a dataset from there. Without `datasetId` it stops before charging and asks you to add it.
- The first run with a new `stateStoreName` reports every item as `added`, because there is nothing to compare with yet.
- Use one `stateStoreName` per scraper, and don't let two runs use the same name at the same time. If that happens, the last run to finish wins; if the fingerprint ends up unreadable, the Actor stops (before charging) and asks you to use a new name.

### Pricing

This Actor uses pay-per-event pricing. The current prices are on the **Pricing** tab of this Actor.

| Event | Price | When it is charged |
|---|---|---|
| Actor start | $0.00005 per run (once more for each GB above 1 GB) | Set by Apify, also for the sample data: once for runs with up to 1 GB of memory, plus once for each extra GB (a 4 GB run is charged 4 times). The default memory of this Actor is 1 GB. |
| Item compared (`item-compared`) | $0.00002 per item ($0.02 per 1,000 items) | Once for every item read from the current dataset, including items that are skipped because they have no key value or a duplicate key. The previous dataset, the fingerprint and the written changes are not charged separately. |

- The cost of a run is predictable: it grows with the size of the current dataset, not with the number of changes.
- Runs that stop at the checks before comparing (key fields missing from most items, the same dataset twice, not enough memory, a reserved or damaged store) are not charged per item.
- **Maximum cost per run**: if you set one, the Actor counts how many items it can afford before each page and stops cleanly when the limit is reached. The changes found so far are kept and the summary says `"status": "PARTIAL"`; it is saved before the charge that reaches your limit, because Apify stops a run as soon as its limit is reached. A partial run cannot know which items were removed, so it skips removed items and does not update the fingerprint. The log tells you at the start if your limit does not cover the whole dataset.
- If a run is interrupted (for example moved to another server by Apify), it continues from the last saved page; an item is never charged twice.

### Limits

**Memory.** The previous dataset is held in memory while the current one is read in pages of at most about 4 MB.

- Measured locally (Node.js 24, items of about 350 bytes as JSON, comparison in memory only, without reading through the API):
  - 100,000 items take about 2 seconds and about 65 MB of JavaScript heap.
  - Two datasets, with the JavaScript heap limited to 128, 192 or 256 MB (heap sizes set for the test, not the memory of a run): 150,000, 250,000 or 300,000 items fit (the whole process then used about 400 MB).
  - `stateStoreName` without a previous dataset: 1,000,000 items fit in a 96 MB heap, plus about 95-130 MB of compact arrays outside the heap; the whole process used about 300-330 MB.
- Measured on Apify (items of about 350 bytes, 1024 MB memory unless noted):
  - A previous dataset of 100,000 items was loaded in about 7 seconds, and the whole run used at most about 150 MB. With 512 MB it worked the same way.
  - Reading and comparing 100,000 current items took about 13 seconds. Most of the time is spent reading the datasets through the API (about 8,000-14,000 items per second).
  - With 256 MB, the run stopped before loading a previous dataset of 100,000 items and asked for 512 MB.
- Estimated from those numbers (not measured on Apify):
  - Two datasets of 100,000 items take about 20 seconds, of 300,000 items about 1 minute, of 500,000 items about 2 minutes.
  - The memory check accepts up to about 90,000 items of this size in a run with 256 MB of memory, 230,000 with 512 MB, 500,000 with the default 1024 MB and 1,100,000 with 2048 MB. Above that, it stops before charging and tells you how much memory to choose.
- At the start, after every page of the previous dataset and once more after it is loaded, the Actor estimates the memory it still needs, both on the JavaScript heap and outside it, and compares it with the memory the run has left. If it is not enough, it stops before any item is charged and tells you how much memory to choose. Bigger items need proportionally more memory.
- One case the check cannot foresee: items that suddenly become much bigger far into a dataset (after thousands of small ones). The Actor reads up to 5,000 items at a time, so one such page can still be too big for the memory, and the run fails. If that happens in the current dataset, the changes of the pages compared before are kept, and those pages are charged.

**Partial runs (maximum cost, timeout, abort).** A run can end with `"status": "PARTIAL"` in `OUTPUT`:

- when it reaches your maximum cost per run (see Pricing);
- 30 seconds before the run's timeout: the Actor stops itself so that it can still save the changes found so far and the summary;
- when you abort the run;
- rarely, when Apify moves a run to another server and the restarted run no longer fits in its memory: the changes found before the move are kept and charged, and the run fails with a message that suggests more memory.

At the maximum cost and at the timeout, the run itself ends as **succeeded** (an aborted run is shown as aborted). So:

- an integration that starts on "run succeeded" (for example a spreadsheet or notification step) also starts after a partial run, and works with the changes found up to that point; check `status` in `OUTPUT` if your next step needs the full result;
- removed items are not reported in a partial run;
- with `stateStoreName`, a partial run does not save the fingerprint, so the next run compares with the same older fingerprint and reports the same changes again. Give the run a longer timeout or a higher maximum cost to finish.

**Matching.**

- Key values are compared as text, so `123` and `"123"` are the same key. Integer IDs above 2^53 (for example `12345678901234567890`) cannot be kept exactly by JSON numbers, so different IDs may look the same; they are counted in `unsafeIntegerKeys`. Use a text field as the key if you can.
- Items with a missing, `null` or empty key field are skipped and counted (`itemsWithoutKeyCurrent`, `itemsWithoutKeyPrevious`). If a key appears more than once, the first item is used and the others are counted (`duplicateKeysCurrent`, `duplicateKeysPrevious`). Both are read and charged; if fewer than half of the first 100 items have a key, the run stops before charging.
- A field set to `null` and a missing field are different values.
- Lists are compared as a whole: `changedFields` names the list, not the position inside it.
- A field literally named `a.b` and a nested field `a` → `b` have the same dotted path; avoid keys and ignore rules that depend on telling them apart.
- The fingerprint is tied to the key fields, ignored fields and list-order setting. If you change them, use a new `stateStoreName`; the Actor stops with a clear message instead of reporting false changes.
- The Actor only reads datasets that you choose in the input (it runs with limited permissions).

### Privacy

- The Actor does not keep your data. It reads the datasets you choose, writes the changes to its own run dataset in your account, and stops.
- The optional fingerprint holds 64-bit hashes of each item's key and content, plus the settings and counts. It is stored in your own account, in the key-value store you name; delete the store to delete it. The items cannot be read back from the hashes, but the hashes are not salted: someone who can open the store can confirm a key that is easy to guess (for example a URL pattern or a sequential ID) by hashing guesses.
- The Actor does not contact any website or service outside your Apify account.

### Example tasks

Ready-made inputs for common jobs. Open one, then click **Try for free** to copy it to your account and change the input:

- [Get new, changed and removed items between two scraper runs](https://apify.com/quietworkslab/dataset-change-tracker/examples/new-changed-removed-items-between-scraper-runs)
- [Track price and stock changes between two product scrapes](https://apify.com/quietworkslab/dataset-change-tracker/examples/track-price-and-stock-changes)
- [Get only the new items from a scheduled scraper](https://apify.com/quietworkslab/dataset-change-tracker/examples/only-new-items-from-scheduled-scraper)
- [Find listings that disappeared between two scraper runs](https://apify.com/quietworkslab/dataset-change-tracker/examples/find-removed-listings-between-runs)

### More from Quiet Works Lab

- [Image Converter & Compressor](https://apify.com/quietworkslab/image-converter): convert images to WebP, AVIF, JPEG or PNG, resize them and remove EXIF and GPS metadata.
- [Japanese Text Normalizer & RAG Chunker](https://apify.com/quietworkslab/japanese-text-normalizer): clean Japanese text before an LLM or RAG and split it into sentence-aware chunks.

### Support

This Actor is developed and maintained with the help of AI (Claude Code). Issues are answered within a few days.

Please open an issue on the **Issues** tab with the run ID and, if possible, a few example items (remove anything private first).

# Changelog

This Actor's version history is a separate document: https://apify.com/quietworkslab/dataset-change-tracker/changelog.md

# Actor input Schema

## `datasetId` (type: `string`):

The dataset to check, usually the newest run of your scraper. Leave it empty to try the Actor on built-in sample data. When this Actor runs after your scraper from the scraper's Integrations tab, set it to {{resource.defaultDatasetId}}. Hidden fields (names starting with #) are not compared.

## `previousDatasetId` (type: `string`):

The older dataset to compare with. You can leave it empty if you set "Remember the last run in this store" below: the Actor then compares with the fingerprint saved by its previous run.

## `stateStoreName` (type: `string`):

Optional. Name of a key-value store in your account, for example my-shop-tracker (not "default"). After each complete run the Actor saves a fingerprint there: 64-bit hashes of each item's key and content, not the items themselves. Note that a key that is easy to guess, such as a URL pattern or a sequential ID, can be confirmed by hashing guesses. The next run compares with the fingerprint, so you don't have to pick the previous dataset. The first run with a new name reports every item as added. Do not use the same name in two runs at the same time. Letters, digits and hyphens only.

## `keyFields` (type: `array`):

Fields that identify the same item in both datasets, for example url, id or sku. Use dots for nested fields (product.id). With several fields, the combination is the key. If fewer than half of the first 100 items have a value in the key fields, the run stops before charging and lists the fields it found. If empty, whole items are compared and only added and removed items are reported.

## `ignoreFields` (type: `array`):

Fields that change on every run and should not count as a change, for example scrapedAt or a session ID. Use dots for nested fields (meta.loadedAt); a nested path also applies to every element of a list (offers.seenAt).

## `changeTypes` (type: `array`):

Which changes to write to the dataset. All of them are still counted in the summary (OUTPUT).

## `ignoreArrayOrder` (type: `boolean`):

If checked, a list with the same elements in a different order (for example tags or image URLs) does not count as a change.

## Actor input object example

```json
{
  "keyFields": [
    "url"
  ],
  "ignoreFields": [
    "scrapedAt"
  ],
  "changeTypes": [
    "added",
    "changed",
    "removed"
  ],
  "ignoreArrayOrder": false
}
```

# Actor output Schema

## `changes` (type: `string`):

One record for each added, changed or removed item.

## `summary` (type: `string`):

Status, settings and counts of the comparison (record OUTPUT).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keyFields": [
        "url"
    ],
    "ignoreFields": [
        "scrapedAt"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("quietworkslab/dataset-change-tracker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "keyFields": ["url"],
    "ignoreFields": ["scrapedAt"],
}

# Run the Actor and wait for it to finish
run = client.actor("quietworkslab/dataset-change-tracker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keyFields": [
    "url"
  ],
  "ignoreFields": [
    "scrapedAt"
  ]
}' |
apify call quietworkslab/dataset-change-tracker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,quietworkslab/dataset-change-tracker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3ELDazGgnvu4Rhp22/builds/5kyhLcx7trjipZ6fp/openapi.json