Dataset Changes Since Last Run: New, Changed, Removed Rows

Dataset Changes Since Last Run: New, Changed, Removed Rows

This tool compares your Apify datasets with the last time you ran it and tells you only what is different. Give it one or more datasets, for example the results of your latest scraper run, and choose the field that identifies each row, such as url, id or sku. It merges the datasets, removes duplicate rows, and outputs one row for each record that is new, changed or removed since the previous run. For a changed record, you see which fields changed, with the old and new values. Run it after each scrape and you get a short list of changes instead of the full data every time. You pay $0.00002 for each row compared ($0.02 per 1,000 rows, with no rounding) and $0.001 for each change found.

Who it's for

• Anyone running a scraper on a schedule who only wants to act on what moved, such as new listings, price changes or products that were taken down.
• Teams that merge results from several scrapers or several runs into one list without duplicates.
• Developers building alerts, syncs or reports that need changes since last time rather than a full export.
• Analysts tracking catalogues, job boards, directories or price lists over time.

Why use it

• Only what changed: each output row is marked new, changed or removed. Rows that did not change are not output.
• Field-level detail: a changed row lists every field that changed, with its value before and after, plus a line-by-line diff.
• You choose the key: compare rows by any field, including a nested field such as owner.id.
• Merge and remove duplicates: several datasets become one list. When two rows share a key, the first one seen is kept.
• Ignore noisy fields: fields that change on every run, such as scrapedAt, can be left out so they never count as a change.
• Limited permissions: it only reads the datasets you pick. It never changes them and never creates datasets in your account. The only storage it creates is its own key-value store for the previous rows.
• Clear failures: an unreadable dataset, a missing key field or empty data stops the run straight away with a plain-English message, instead of finishing with an empty result.
• No partial bills: before charging anything, the run works out the cost of the whole run. If your maximum total charge can't cover all of it, or the run is about to run out of time, it stops with a message, charges nothing, writes nothing and keeps the saved rows, so no change is lost and no change is billed twice.
• Safe restarts: if Apify restarts a run part-way through, it skips the output it already wrote and charged.

How to use it

1. Open the Actor in Apify Console. Under Datasets to compare, use the dataset picker to choose one or more datasets, for example the dataset from your scraper's latest run. Picking them this way gives the Actor permission to read them.
2. Set the Key field, the field that identifies a row, such as url or id. Use dots for nested fields, for example owner.id.
3. Give the comparison a Monitor name, for example shop-prices. Keep the same name every time you run this comparison.
4. Optionally add Fields to ignore, for example scrapedAt.
5. Click Start. The first run saves a baseline (the starting version of your data) and reports no changes. Tick Output baseline on first run if you also want every row output once on that run.
6. Run it again after your next scrape. Open the Output tab to see one row for each new, changed or removed record, and export them as JSON, CSV or Excel.
7. To automate this, add it as an integration on your scraper (so it runs after each scraper run, with that run's dataset), or create a schedule under Schedules. Add a webhook or integration such as Slack, Google Sheets, Zapier or Make to send the changes on.

Pricing

You pay $0.00002 for each row compared ($0.02 per 1,000 rows). Every row read in a run is charged once, with no rounding, including on the first run and including duplicate rows and rows without a key. You also pay $0.001 for each change found, meaning each new, changed or removed row in the output. Baseline rows aren't charged as changes. Each run also has a tiny Apify start fee of $0.00005. Before charging anything, the run adds up what the whole run will cost. If your maximum total charge can't cover all of it, the run fails, charges nothing and writes nothing. Runs that fail, for example because a dataset can't be read, the key field is missing or there are too many rows, are not charged.

Example: comparing 10,000 rows and finding 50 changes costs $0.20 for the rows plus $0.05 for the changes, so $0.25. Comparing 100,000 rows where 1% changed (1,000 changes) costs $2.00 plus $1.00, so $3.00.

Scheduled example: comparing a 20,000-row dataset once a day, with about 100 changes a day, costs $0.40 plus $0.10, so $0.50 a day, or about $15.00 over 30 days. The first run, which saves the baseline, costs only the $0.40 for the rows.

API and reference

This section is for developers and AI agents. It lists every input and output field, a sample record and code examples.

Summary

Reads one or more existing Apify datasets by ID or name, merges their rows by a key field, removes duplicate keys (the first row seen wins), and compares the result with the snapshot saved by the previous run with the same monitorName. Writes one dataset row per new, changed or removed key. The snapshot is kept in a named key-value store, dataset-changes-<monitorName>, in the user's account. It is the only storage the Actor creates. Input datasets are only read. The Actor runs with limited permissions and can read the datasets the user selects with the dataset picker (resourceType: "dataset", resourcePermissions: ["READ"]).

Input

Field	Type	Default	Description
datasetIds	string[]	required	Datasets to merge and compare, by ID or name. A bare name is looked up in the user's account, and username~name also works. Datasets are read in the order given and repeated entries are removed. Pick them with the dataset picker so the Actor has read permission. A dataset that can't be read fails the run.
keyField	string	required	Field that identifies a row. Use dots for nested fields (owner.id). A top-level field whose name contains dots takes priority. Values are compared as text. Rows where the value is missing, null, empty, an object or an array are counted in rowsWithoutKey and not compared.
monitorName	string	default	Name of this comparison. It names the snapshot store dataset-changes-<monitorName>. The name is lowercased, and characters other than a-z, 0-9 and - become -. Keep it the same between runs.
ignoreFields	string[]	[]	Top-level fields left out of the comparison and of data in the output. Must not include keyField.
outputBaseline	boolean	false	On the first run, also write every row once with changeType: "baseline". Baseline rows are not charged as changes.
resetBaseline	boolean	false	Ignore the saved snapshot and record a new baseline from this run's data.
maxRows	integer	20000	1 to 100000, counting all datasets together. More rows than this fail the run, so a partial read can't report false removed rows.

Changing keyField or ignoreFields for an existing monitorName records a new baseline on the next run, and the SUMMARY record's note says so.

Output

One dataset row per new, changed or removed key. Rows are ordered new, then changed, then removed, and sorted by key within each group. Unchanged rows are not written. The fields from changedFields to removedCount appear only on changed rows.

Field	Type	Description
changeType	string	new, changed, removed or baseline (first run only, when outputBaseline is true)
key	string	Value of the key field, as text
data	object	The current row, without the ignored fields. For removed rows, the last version that was saved
changedFields	object[]	Every top-level field that changed, sorted by field name
changedFields[].field	string	Field name
changedFields[].before	any	Previous value, null if the field was not there before
changedFields[].after	any	Current value, null if the field was removed
diff	string	Unified diff of the row, one field: value line per field, cut to 4,000 characters
diffTruncated	boolean	true if diff was cut short
addedLines	string[]	Lines added (up to 200, each up to 500 characters)
removedLines	string[]	Lines removed (up to 200, each up to 500 characters)
addedCount	integer	Total number of lines added
removedCount	integer	Total number of lines removed
monitorName	string	Monitor name from the input
detectedAt	string	ISO 8601 UTC time of the run, for example 2026-10-21T06:00:12+00:00

Example changed record from a real run on 21 October 2026, with keyField: "id" and no ignored fields. Key values are always text, so the numeric id 3 gives "key": "3":

{
  "changeType": "changed",
  "key": "3",
  "data": {
    "id": 3,
    "sku": "SKU-003",
    "price": 99,
    "title": "Item 3"
  },
  "changedFields": [
    {"field": "price", "before": 3, "after": 99}
  ],
  "diff": "--- previous\n+++ current\n@@ -1,3 +1,3 @@\n id: 3\n-price: 3\n+price: 99\n sku: \"SKU-003\"",
  "diffTruncated": false,
  "addedLines": ["price: 99"],
  "removedLines": ["price: 3"],
  "addedCount": 1,
  "removedCount": 1,
  "monitorName": "precheck-018",
  "detectedAt": "2026-10-21T17:03:45+00:00"
}

new, removed and baseline records have only changeType, key, data, monitorName and detectedAt.

The run's default key-value store gets a SUMMARY record with these fields: monitorName (string), stateStore (string, the snapshot store name), mode (baseline or compare), keyField (string), finishedAt (string), rowsRead (integer, every row read, including duplicates and rows without a key), duplicatesRemoved (integer), rowsWithoutKey (integer), uniqueRows (integer), new, changed, removed and unchanged (integers, all 0 on a baseline run), changesWritten (integer, the number of new, changed and removed rows, 0 on a baseline run) and note (string or null, for example when a new baseline was recorded). The run also keeps a PUSHED record in its default key-value store, which it uses to resume output after a restart.

Error handling: these cases fail the run with a plain-English status message. Nothing is charged, no rows are written and the saved snapshot is left as it was:

• A dataset can't be read (wrong ID or name, or no permission): We can't read dataset X. Pick it with the dataset picker, or check the ID.
• The key field is not in any of the first 50 rows of a non-empty dataset. The message lists the fields that were found.
• All datasets are empty, or no row has a usable key value.
• keyField is also listed in ignoreFields.
• The datasets have more than maxRows rows in total.
• The rows read would use too much of the run's memory (more than 12% of the run's memory, measured as the JSON size of the rows). The message says to lower Max rows or give the run more memory.
• The run gets within 20 seconds of its timeout while reading the datasets, or within 60 seconds of its timeout after comparing and before charging.
• The snapshot store can't be opened.
• The maximum total charge left can't cover the whole run. The cost is one row-compared charge per row read plus one change-detected charge per change. The message gives that cost and the amount left, in the form This run would cost $X (N rows compared, M changes), but only $Y is left of your maximum total charge. Nothing was charged or written and the saved snapshot was left as it was, so no change is lost. Raise the maximum total charge and run again. Amounts show up to 5 decimal places: the cost is rounded up and the amount left is rounded down, for example would cost $0.09 (2500 rows compared, 40 changes), but only $0.089 is left.

Charge limit: before the first charge, the run adds up the price of every event it still has to charge: one row-compared per row read and one change-detected per change. Charges already made by an earlier attempt of the same run are not counted again. If the total is more than what is left of the maximum total charge, the run fails as above. Otherwise, everything is written and charged, and the snapshot is saved.

Pricing details

Pay per event:

• row-compared at $0.00002 ($0.02 per 1,000 rows): charged once per run for every row read across all datasets, including on the baseline run. Rows read include duplicates and rows without a key.
• change-detected at $0.001: charged for each new, changed or removed row written. baseline rows are not charged as changes.

These are the only two charges, plus the Apify Actor start event ($0.00005).

Nothing is charged until the datasets are read, compared and the whole cost fits within the maximum total charge. Every charge carries an idempotency key made from the run ID, the event and the batch start, with no time in it, so a retried charge or a restarted run is not billed twice. A restarted run does not charge row-compared again and skips the output batches it already wrote and charged. Runs that fail are not charged.

Usage

Apify API (synchronous run, returns dataset items):

curl -X POST "https://api.apify.com/v2/acts/opendata-tools~dataset-changes-since-last-run/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "datasetIds": ["YOUR_DATASET_ID"],
    "keyField": "url",
    "monitorName": "shop-prices",
    "ignoreFields": ["scrapedAt"]
  }'

Python (pip install apify-client):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("opendata-tools/dataset-changes-since-last-run").call(run_input={
    "datasetIds": ["YOUR_DATASET_ID", "my-other-dataset"],
    "keyField": "url",
    "monitorName": "shop-prices",
    "ignoreFields": ["scrapedAt"],
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["changeType"] == "changed":
        for change in item["changedFields"]:
            print(item["key"], change["field"], change["before"], "->", change["after"])
    else:
        print(item["changeType"], item["key"])

JavaScript (npm install apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('opendata-tools/dataset-changes-since-last-run').call({
    datasetIds: ['YOUR_DATASET_ID'],
    keyField: 'url',
    monitorName: 'shop-prices',
    ignoreFields: ['scrapedAt'],
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) console.log(item.changeType, item.key, item.changedFields ?? []);
const summary = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('SUMMARY');
console.log(summary.value);

When the Actor runs with limited permissions, it can only read a dataset passed by API if the token's user owns it or has access to it.

For a change feed, add this Actor as an integration on your scraper, or create an Apify schedule with a fixed input, and attach a webhook on ACTOR.RUN.SUCCEEDED to receive the results.

Limits

• Up to 100,000 rows per run across all datasets (maxRows, default 20,000).
• The previous snapshot, the current rows and the diffs are all held in memory. The default run uses 1,024 MB of memory and a 600-second timeout. The run stops cleanly when the rows read pass 12% of its memory (about 128 MB of row JSON at 1,024 MB), so very wide rows can hit this before the row limit. Give the run more memory or lower maxRows.
• changedFields and ignoreFields work on top-level fields. A change inside a nested object shows as the whole object before and after.
• Only rows with a usable key value are compared. The rest are counted in rowsWithoutKey.
• The snapshot holds a copy of your rows (without the ignored fields) and is stored only in your account, in dataset-changes-<monitorName>.

FAQ

Why did my first run return nothing?

The first run for a monitor name saves a baseline and has nothing to compare against yet, so it outputs no rows. Tick Output baseline on first run (outputBaseline) to get every row once, marked baseline. Baseline rows aren't charged as changes.

Why does it say "We can't read dataset X"?

The Actor can only read datasets you have given it access to. Choose the dataset with the dataset picker under Datasets to compare, or check that the ID or name is correct and that the dataset still exists.

Does it create anything in my account?

Only one named key-value store per monitor name, dataset-changes-<monitorName>, which holds the rows from the last run. It never creates datasets and never changes your input datasets.

Which row is kept when two rows have the same key?

The first one seen. Datasets are read in the order you list them, and each dataset from top to bottom. The SUMMARY record shows how many duplicates were removed.

What happens when my maximum total charge is reached?

The run works out the cost of the whole run before charging anything. If what is left of your maximum total charge can't cover all of it, the run fails with a message giving the cost and the amount left. Nothing is charged or written, and the saved snapshot is kept, so the next run with a higher limit reports the same changes. You are never charged for part of a run's changes.

How do I start again from scratch?

Tick Reset baseline (resetBaseline) for one run, or use a new monitor name. Changing the key field or the ignored fields also records a new baseline.

Issues and feature requests: use the Issues tab.