# Dataset Changes Since Last Run: New, Changed, Removed Rows (`opendata-tools/dataset-changes-since-last-run`) Actor Merges one or more Apify datasets, removes duplicates by a key field you choose, and outputs only the rows that are new, changed or removed since the last run. Pay per event. - **URL**: https://apify.com/opendata-tools/dataset-changes-since-last-run.md - **Developed by:** [Open Data Tools](https://apify.com/opendata-tools) (community) - **Categories:** Developer tools, Automation - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.02 / 1,000 compared rows This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### Dataset Changes Since Last Run: New, Changed, Removed Rows This tool compares your Apify datasets with the last time you ran it and tells you only what is different. Give it one or more datasets, for example the results of your latest scraper run, and choose the field that identifies each row, such as `url`, `id` or `sku`. It merges the datasets, removes duplicate rows, and outputs one row for each record that is new, changed or removed since the previous run. For a changed record, you see which fields changed, with the old and new values. Run it after each scrape and you get a short list of changes instead of the full data every time. You pay $0.00002 for each row compared ($0.02 per 1,000 rows, with no rounding) and $0.001 for each change found. #### Who it's for - Anyone running a scraper on a schedule who only wants to act on what moved, such as new listings, price changes or products that were taken down. - Teams that merge results from several scrapers or several runs into one list without duplicates. - Developers building alerts, syncs or reports that need changes since last time rather than a full export. - Analysts tracking catalogues, job boards, directories or price lists over time. #### Why use it - **Only what changed**: each output row is marked `new`, `changed` or `removed`. Rows that did not change are not output. - **Field-level detail**: a changed row lists every field that changed, with its value before and after, plus a line-by-line diff. - **You choose the key**: compare rows by any field, including a nested field such as `owner.id`. - **Merge and remove duplicates**: several datasets become one list. When two rows share a key, the first one seen is kept. - **Ignore noisy fields**: fields that change on every run, such as `scrapedAt`, can be left out so they never count as a change. - **Limited permissions**: it only reads the datasets you pick. It never changes them and never creates datasets in your account. The only storage it creates is its own key-value store for the previous rows. - **Clear failures**: an unreadable dataset, a missing key field or empty data stops the run straight away with a plain-English message, instead of finishing with an empty result. - **No partial bills**: before charging anything, the run works out the cost of the whole run. If your maximum total charge can't cover all of it, or the run is about to run out of time, it stops with a message, charges nothing, writes nothing and keeps the saved rows, so no change is lost and no change is billed twice. - **Safe restarts**: if Apify restarts a run part-way through, it skips the output it already wrote and charged. #### How to use it 1. Open the Actor in Apify Console. Under **Datasets to compare**, use the dataset picker to choose one or more datasets, for example the dataset from your scraper's latest run. Picking them this way gives the Actor permission to read them. 2. Set the **Key field**, the field that identifies a row, such as `url` or `id`. Use dots for nested fields, for example `owner.id`. 3. Give the comparison a **Monitor name**, for example `shop-prices`. Keep the same name every time you run this comparison. 4. Optionally add **Fields to ignore**, for example `scrapedAt`. 5. Click **Start**. The first run saves a baseline (the starting version of your data) and reports no changes. Tick **Output baseline on first run** if you also want every row output once on that run. 6. Run it again after your next scrape. Open the **Output** tab to see one row for each new, changed or removed record, and export them as JSON, CSV or Excel. 7. To automate this, add it as an integration on your scraper (so it runs after each scraper run, with that run's dataset), or create a schedule under **Schedules**. Add a webhook or integration such as Slack, Google Sheets, Zapier or Make to send the changes on. #### Pricing You pay $0.00002 for each row compared ($0.02 per 1,000 rows). Every row read in a run is charged once, with no rounding, including on the first run and including duplicate rows and rows without a key. You also pay $0.001 for each change found, meaning each new, changed or removed row in the output. Baseline rows aren't charged as changes. Each run also has a tiny Apify start fee of $0.00005. Before charging anything, the run adds up what the whole run will cost. If your maximum total charge can't cover all of it, the run fails, charges nothing and writes nothing. Runs that fail, for example because a dataset can't be read, the key field is missing or there are too many rows, are not charged. Example: comparing 10,000 rows and finding 50 changes costs $0.20 for the rows plus $0.05 for the changes, so $0.25. Comparing 100,000 rows where 1% changed (1,000 changes) costs $2.00 plus $1.00, so $3.00. Scheduled example: comparing a 20,000-row dataset once a day, with about 100 changes a day, costs $0.40 plus $0.10, so $0.50 a day, or about $15.00 over 30 days. The first run, which saves the baseline, costs only the $0.40 for the rows. ### API and reference This section is for developers and AI agents. It lists every input and output field, a sample record and code examples. #### Summary Reads one or more existing Apify datasets by ID or name, merges their rows by a key field, removes duplicate keys (the first row seen wins), and compares the result with the snapshot saved by the previous run with the same `monitorName`. Writes one dataset row per new, changed or removed key. The snapshot is kept in a named key-value store, `dataset-changes-`, in the user's account. It is the only storage the Actor creates. Input datasets are only read. The Actor runs with limited permissions and can read the datasets the user selects with the dataset picker (`resourceType: "dataset"`, `resourcePermissions: ["READ"]`). #### Input | Field | Type | Default | Description | |---|---|---|---| | `datasetIds` | string\[] | required | Datasets to merge and compare, by ID or name. A bare name is looked up in the user's account, and `username~name` also works. Datasets are read in the order given and repeated entries are removed. Pick them with the dataset picker so the Actor has read permission. A dataset that can't be read fails the run. | | `keyField` | string | required | Field that identifies a row. Use dots for nested fields (`owner.id`). A top-level field whose name contains dots takes priority. Values are compared as text. Rows where the value is missing, `null`, empty, an object or an array are counted in `rowsWithoutKey` and not compared. | | `monitorName` | string | `default` | Name of this comparison. It names the snapshot store `dataset-changes-`. The name is lowercased, and characters other than `a-z`, `0-9` and `-` become `-`. Keep it the same between runs. | | `ignoreFields` | string\[] | `[]` | Top-level fields left out of the comparison and of `data` in the output. Must not include `keyField`. | | `outputBaseline` | boolean | `false` | On the first run, also write every row once with `changeType: "baseline"`. Baseline rows are not charged as changes. | | `resetBaseline` | boolean | `false` | Ignore the saved snapshot and record a new baseline from this run's data. | | `maxRows` | integer | `20000` | 1 to 100000, counting all datasets together. More rows than this fail the run, so a partial read can't report false `removed` rows. | Changing `keyField` or `ignoreFields` for an existing `monitorName` records a new baseline on the next run, and the `SUMMARY` record's `note` says so. #### Output One dataset row per new, changed or removed key. Rows are ordered `new`, then `changed`, then `removed`, and sorted by `key` within each group. Unchanged rows are not written. The fields from `changedFields` to `removedCount` appear only on `changed` rows. | Field | Type | Description | |---|---|---| | `changeType` | string | `new`, `changed`, `removed` or `baseline` (first run only, when `outputBaseline` is `true`) | | `key` | string | Value of the key field, as text | | `data` | object | The current row, without the ignored fields. For `removed` rows, the last version that was saved | | `changedFields` | object\[] | Every top-level field that changed, sorted by field name | | `changedFields[].field` | string | Field name | | `changedFields[].before` | any | Previous value, `null` if the field was not there before | | `changedFields[].after` | any | Current value, `null` if the field was removed | | `diff` | string | Unified diff of the row, one `field: value` line per field, cut to 4,000 characters | | `diffTruncated` | boolean | `true` if `diff` was cut short | | `addedLines` | string\[] | Lines added (up to 200, each up to 500 characters) | | `removedLines` | string\[] | Lines removed (up to 200, each up to 500 characters) | | `addedCount` | integer | Total number of lines added | | `removedCount` | integer | Total number of lines removed | | `monitorName` | string | Monitor name from the input | | `detectedAt` | string | ISO 8601 UTC time of the run, for example `2026-10-21T06:00:12+00:00` | Example `changed` record from a real run on 21 October 2026, with `keyField: "id"` and no ignored fields. Key values are always text, so the numeric `id` 3 gives `"key": "3"`: ```json { "changeType": "changed", "key": "3", "data": { "id": 3, "sku": "SKU-003", "price": 99, "title": "Item 3" }, "changedFields": [ {"field": "price", "before": 3, "after": 99} ], "diff": "--- previous\n+++ current\n@@ -1,3 +1,3 @@\n id: 3\n-price: 3\n+price: 99\n sku: \"SKU-003\"", "diffTruncated": false, "addedLines": ["price: 99"], "removedLines": ["price: 3"], "addedCount": 1, "removedCount": 1, "monitorName": "precheck-018", "detectedAt": "2026-10-21T17:03:45+00:00" } ``` `new`, `removed` and `baseline` records have only `changeType`, `key`, `data`, `monitorName` and `detectedAt`. The run's default key-value store gets a `SUMMARY` record with these fields: `monitorName` (string), `stateStore` (string, the snapshot store name), `mode` (`baseline` or `compare`), `keyField` (string), `finishedAt` (string), `rowsRead` (integer, every row read, including duplicates and rows without a key), `duplicatesRemoved` (integer), `rowsWithoutKey` (integer), `uniqueRows` (integer), `new`, `changed`, `removed` and `unchanged` (integers, all `0` on a baseline run), `changesWritten` (integer, the number of `new`, `changed` and `removed` rows, `0` on a baseline run) and `note` (string or null, for example when a new baseline was recorded). The run also keeps a `PUSHED` record in its default key-value store, which it uses to resume output after a restart. Error handling: these cases fail the run with a plain-English status message. Nothing is charged, no rows are written and the saved snapshot is left as it was: - A dataset can't be read (wrong ID or name, or no permission): `We can't read dataset X. Pick it with the dataset picker, or check the ID.` - The key field is not in any of the first 50 rows of a non-empty dataset. The message lists the fields that were found. - All datasets are empty, or no row has a usable key value. - `keyField` is also listed in `ignoreFields`. - The datasets have more than `maxRows` rows in total. - The rows read would use too much of the run's memory (more than 12% of the run's memory, measured as the JSON size of the rows). The message says to lower **Max rows** or give the run more memory. - The run gets within 20 seconds of its timeout while reading the datasets, or within 60 seconds of its timeout after comparing and before charging. - The snapshot store can't be opened. - The maximum total charge left can't cover the whole run. The cost is one `row-compared` charge per row read plus one `change-detected` charge per change. The message gives that cost and the amount left, in the form `This run would cost $X (N rows compared, M changes), but only $Y is left of your maximum total charge. Nothing was charged or written and the saved snapshot was left as it was, so no change is lost. Raise the maximum total charge and run again.` Amounts show up to 5 decimal places: the cost is rounded up and the amount left is rounded down, for example `would cost $0.09 (2500 rows compared, 40 changes), but only $0.089 is left`. Charge limit: before the first charge, the run adds up the price of every event it still has to charge: one `row-compared` per row read and one `change-detected` per change. Charges already made by an earlier attempt of the same run are not counted again. If the total is more than what is left of the maximum total charge, the run fails as above. Otherwise, everything is written and charged, and the snapshot is saved. #### Pricing details Pay per event: - `row-compared` at $0.00002 ($0.02 per 1,000 rows): charged once per run for every row read across all datasets, including on the baseline run. Rows read include duplicates and rows without a key. - `change-detected` at $0.001: charged for each `new`, `changed` or `removed` row written. `baseline` rows are not charged as changes. These are the only two charges, plus the Apify Actor start event ($0.00005). Nothing is charged until the datasets are read, compared and the whole cost fits within the maximum total charge. Every charge carries an idempotency key made from the run ID, the event and the batch start, with no time in it, so a retried charge or a restarted run is not billed twice. A restarted run does not charge `row-compared` again and skips the output batches it already wrote and charged. Runs that fail are not charged. #### Usage Apify API (synchronous run, returns dataset items): ```bash curl -X POST "https://api.apify.com/v2/acts/opendata-tools~dataset-changes-since-last-run/run-sync-get-dataset-items" \ -H "Authorization: Bearer $APIFY_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "datasetIds": ["YOUR_DATASET_ID"], "keyField": "url", "monitorName": "shop-prices", "ignoreFields": ["scrapedAt"] }' ``` Python (`pip install apify-client`): ```python import os from apify_client import ApifyClient client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor("opendata-tools/dataset-changes-since-last-run").call(run_input={ "datasetIds": ["YOUR_DATASET_ID", "my-other-dataset"], "keyField": "url", "monitorName": "shop-prices", "ignoreFields": ["scrapedAt"], }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): if item["changeType"] == "changed": for change in item["changedFields"]: print(item["key"], change["field"], change["before"], "->", change["after"]) else: print(item["changeType"], item["key"]) ``` JavaScript (`npm install apify-client`): ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); const run = await client.actor('opendata-tools/dataset-changes-since-last-run').call({ datasetIds: ['YOUR_DATASET_ID'], keyField: 'url', monitorName: 'shop-prices', ignoreFields: ['scrapedAt'], }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); for (const item of items) console.log(item.changeType, item.key, item.changedFields ?? []); const summary = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('SUMMARY'); console.log(summary.value); ``` When the Actor runs with limited permissions, it can only read a dataset passed by API if the token's user owns it or has access to it. For a change feed, add this Actor as an integration on your scraper, or create an Apify schedule with a fixed input, and attach a webhook on `ACTOR.RUN.SUCCEEDED` to receive the results. #### Limits - Up to 100,000 rows per run across all datasets (`maxRows`, default 20,000). - The previous snapshot, the current rows and the diffs are all held in memory. The default run uses 1,024 MB of memory and a 600-second timeout. The run stops cleanly when the rows read pass 12% of its memory (about 128 MB of row JSON at 1,024 MB), so very wide rows can hit this before the row limit. Give the run more memory or lower `maxRows`. - `changedFields` and `ignoreFields` work on top-level fields. A change inside a nested object shows as the whole object before and after. - Only rows with a usable key value are compared. The rest are counted in `rowsWithoutKey`. - The snapshot holds a copy of your rows (without the ignored fields) and is stored only in your account, in `dataset-changes-`. ### FAQ #### Why did my first run return nothing? The first run for a monitor name saves a baseline and has nothing to compare against yet, so it outputs no rows. Tick **Output baseline on first run** (`outputBaseline`) to get every row once, marked `baseline`. Baseline rows aren't charged as changes. #### Why does it say "We can't read dataset X"? The Actor can only read datasets you have given it access to. Choose the dataset with the dataset picker under **Datasets to compare**, or check that the ID or name is correct and that the dataset still exists. #### Does it create anything in my account? Only one named key-value store per monitor name, `dataset-changes-`, which holds the rows from the last run. It never creates datasets and never changes your input datasets. #### Which row is kept when two rows have the same key? The first one seen. Datasets are read in the order you list them, and each dataset from top to bottom. The `SUMMARY` record shows how many duplicates were removed. #### What happens when my maximum total charge is reached? The run works out the cost of the whole run before charging anything. If what is left of your maximum total charge can't cover all of it, the run fails with a message giving the cost and the amount left. Nothing is charged or written, and the saved snapshot is kept, so the next run with a higher limit reports the same changes. You are never charged for part of a run's changes. #### How do I start again from scratch? Tick **Reset baseline** (`resetBaseline`) for one run, or use a new monitor name. Changing the key field or the ignored fields also records a new baseline. Issues and feature requests: use the Issues tab. # Actor input Schema ## `datasetIds` (type: `array`): The datasets to merge and compare, from the Storage tab or a previous run. Use the latest run's dataset each time (for example via the 'last run dataset' option in a task or an integration). ## `keyField` (type: `string`): The field that identifies a row, for example url, id or sku. Use dots for nested fields, e.g. owner.id. The run stops straight away if this field is not in your data. ## `monitorName` (type: `string`): Name of this comparison. The previous rows are kept in a named key-value store in your account (dataset-changes-). Keep it the same between runs. ## `ignoreFields` (type: `array`): Optional. Fields that change on every run and should not count as a change, e.g. scrapedAt or position. ## `outputBaseline` (type: `boolean`): On the first run nothing is 'new' yet. Tick this to also write all rows once, marked baseline (free). ## `resetBaseline` (type: `boolean`): Forget the saved rows and start again from this run's data. ## `maxRows` (type: `integer`): 1 to 100000, counting all datasets together. If there are more rows, the run stops with a message instead of comparing only part of them. ## Actor input object example ```json { "datasetIds": [ "YOUR_DATASET_ID" ], "keyField": "url", "monitorName": "default", "ignoreFields": [], "outputBaseline": false, "resetBaseline": false, "maxRows": 20000 } ``` # Actor output Schema ## `changes` (type: `string`): One row per new, changed or removed row. ## `summary` (type: `string`): Counts of new, changed, removed and unchanged rows. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "keyField": "url", "monitorName": "default" }; // Run the Actor and wait for it to finish const run = await client.actor("opendata-tools/dataset-changes-since-last-run").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "keyField": "url", "monitorName": "default", } # Run the Actor and wait for it to finish run = client.actor("opendata-tools/dataset-changes-since-last-run").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "keyField": "url", "monitorName": "default" }' | apify call opendata-tools/dataset-changes-since-last-run --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,opendata-tools/dataset-changes-since-last-run" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Gb0H9hUTr2Pr3Oizd/builds/mANQnn3ciiuyHAzzp/openapi.json