# Dataset Diff: Compare Datasets, Get New & Changed Rows (`humble-echidna/dataset-diff`) Actor Compare two Apify datasets or CSV, JSON or Excel files, or each run of a scraper with its last run: get only the new, changed and removed rows, with which fields changed and their old values. Chain it after any actor to monitor prices, stock, listings or jobs. Pay per change. - **URL**: https://apify.com/humble-echidna/dataset-diff.md - **Developed by:** [Michael Costa](https://apify.com/humble-echidna) (community) - **Categories:** Developer tools, Automation, Integrations - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.70 / 1,000 changes This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Dataset Diff do? **Dataset Diff** compares **two Apify datasets or CSV, JSON or Excel files**, or **each run of a scraper with its last run**, and returns only the **new, changed and removed rows**, with which fields changed and what they were before. Chain it after any actor to watch prices, stock, listings, jobs or any other data for changes. It's the "what's different since last time?" step between a scheduled scraper and your alerts, spreadsheet or database: only the 12 products whose price moved, not all 5,000 again. [Input fields](https://apify.com/humble-echidna/dataset-diff/input-schema) · [API](https://apify.com/humble-echidna/dataset-diff/api) · [Use it from Claude, ChatGPT or Cursor (MCP)](#can-i-use-dataset-diff-from-an-ai-agent-mcp) **Jump to:** [Monitor a scraper](#monitor-any-scraper-get-only-what-changed-since-its-last-run) · [Fields](#what-data-does-dataset-diff-return) · [Price](#how-much-does-it-cost-to-compare-datasets) · [How to use](#how-to-compare-two-datasets-or-files) · [Input](#input) · [Output](#output) · [AI agents (MCP)](#can-i-use-dataset-diff-from-an-ai-agent-mcp) · [Limits](#limits) · [FAQ](#faq) **Try it in one click:** the input comes pre-filled with the US Treasury's official exchange rates at two dates (2025-09-30 and 2025-12-31, public CSV files), compared by currency on the rate: 120 rates changed, 1 currency is new and 1 is gone, so 122 rows, about $0.12 (122 × $0.001, plus 167 rows compared × $0.00001 and $0.00005 for the run start). **Then pick your own datasets or files.** ### Monitor any scraper: get only what changed since its last run Leave **Compare with** empty and each run compares with what the same comparison saw on its previous run, remembered in your own Apify account. A run where nothing changed writes no rows and costs only the rows compared ($0.01 per 1,000) and the $0.00005 start fee. 1. Open the scraper (or its saved task), go to the **Integrations** tab and add **Apify actor** → **Dataset Diff**, triggered when a run **succeeds**. 2. Set the input to `{"datasetId": "{{resource.defaultDatasetId}}", "keyFields": ["url"], "ignoreFields": ["scrapedAt"]}`: Apify fills in the finished run's dataset, `keyFields` says what identifies a row (a URL, an id, or `name, city`), and `ignoreFields` lists what the scraper stamps on every run, so it doesn't count as a change. 3. Every successful scrape now produces a dataset with only the new, changed and removed rows. The first run reports every row as new (set **First run with no earlier data** to "Only remember the rows" to skip that). 4. To get them in **Slack**, **email** or a **webhook**, open the Integrations tab of Dataset Diff (or of its saved task) and add one, triggered when a run succeeds: Slack with `{{resource.statusMessage}}` and a link to `https://console.apify.com/storage/datasets/{{resource.defaultDatasetId}}`, Gmail with the dataset attached, or an HTTP webhook whose `resource.defaultDatasetId` is the dataset of changes. The memory follows the actor that wrote the dataset (so each scraper run's new dataset id doesn't matter), or the saved task running Dataset Diff, or **Comparison name** when you set one. Changes cut by **Max rows per run** or your maximum cost per run aren't lost: the next run reports them. ### What data does Dataset Diff return? Your own rows, with three fields added. From the example: | Field | Example | Notes | |---|---|---| | *(your fields)* | `"country": "Afghanistan"`, `"exchange_rate": "65.96"` | The new row as it is; for a removed row, the old one. | | `changeType` | `changed` | `new`, `changed`, `removed` (or `unchanged`, if you ask for those too). | | `changedFields` | `["exchange_rate"]` | The compared fields that differ, for a changed row; empty otherwise. | | `previousValues` | `{"exchange_rate": "67.33"}` | What the changed fields held before; `null` for new and removed rows. | The run's `RUN_STATS` record counts the rows read, compared, new, changed, removed and unchanged, rows without a key and repeated keys. More under [Output](#output). ### How much does it cost to compare datasets? You pay for each row written (a change) and for the rows compared: **$1.00 per 1,000 changes** and **$0.01 per 1,000 rows compared**, plus $0.00005 each time a run starts. **Changes are cheaper on paid Apify plans:** $0.90 per 1,000 on Starter, $0.80 on Scale and $0.70 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show. - **The example:** 122 changes × $0.001 + 167 rows compared × $0.00001 = about $0.12, plus the start fee. - **A month, for example:** a daily product scraper of 5,000 items where 50 change a day: 30 × (5,000 × $0.00001 + 50 × $0.001) = **$3.00**, plus 30 starts ($0.0015). The same with nothing changing: $1.50. - **Caps:** **Max rows per run** in the input, and **Maximum cost per run** in the run options. The run stops cleanly at whichever comes first, and what it didn't write is reported next run. **Never charged:** unchanged rows (unless you ask for them), rows without a key, repeated keys, and the older data you compare with. ### How to compare two datasets or files 1. Open Dataset Diff and click **Try for free** (or **Start** if you're signed in). 2. Pick the newer data in **New data: Apify dataset** (or paste a link in **Or: new data file URL**). 3. Pick the older data in **Compare with** (a dataset or a file link), or leave it empty to compare with the last run. 4. Set **Key fields** (what identifies a row, e.g. `url`), and if you like **Fields to compare** (e.g. `price`, `stock`) or **Fields to ignore** (e.g. `scrapedAt`). 5. Click **Start**, then open the **Output** tab and export as JSON, CSV or Excel. ### Example: exchange rates that changed in a quarter The pre-filled input (two public CSV files, compared by currency on the rate): ```json {"fileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300", "previousFileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-09-30&page[size]=300", "keyFields": ["country_currency_desc"], "compareFields": ["exchange_rate"]} ``` It compared 167 currencies (7 listed twice were counted once) and wrote 122 rows. One of each kind (real output from a local run on 2026-10-03; the Treasury's bookkeeping columns left out): ```json [ {"country": "Afghanistan", "currency": "Afghani", "exchange_rate": "65.96", "record_date": "2025-12-31", "changeType": "changed", "changedFields": ["exchange_rate"], "previousValues": {"exchange_rate": "67.33"}}, {"country": "Cyprus", "currency": "Euro", "exchange_rate": "0.851", "record_date": "2025-12-31", "changeType": "new", "changedFields": [], "previousValues": null}, {"country": "Belarus", "currency": "New Ruble", "exchange_rate": "3.025", "record_date": "2025-09-30", "changeType": "removed", "changedFields": [], "previousValues": null} ] ``` ### Ready-to-run examples Each example opens with the input already filled in. Run it as it is, or change the input first. - **[Compare two CSV files and get the rows that changed](https://apify.com/humble-echidna/dataset-diff/examples/compare-two-csv-files)**: Two versions of the same table, compared by a key: only the new, changed and removed rows, with the old values. - **[Monitor a CSV feed and get only new or updated rows](https://apify.com/humble-echidna/dataset-diff/examples/monitor-csv-feed-new-rows)**: Run it on a schedule against a feed (here USGS earthquakes) and get only what's new or re-reviewed since the last run. ### Input | Field | What it does | |---|---| | **New data: Apify dataset** | The latest data: one of your datasets, picked from the list (or its id or name). Read-only. | | **Or: new data file URL** | A direct link to a CSV/TSV, JSON, JSON Lines or Excel (.xlsx, first sheet) file, up to 200 MB. | | **Compare with** (dataset or file URL) | The older data. Leave both empty to compare with the last run. | | Key fields | What identifies a row: `url`, `id`, or several (`name, city`). Dots reach nested fields (`product.sku`). Empty: whole rows. | | Fields to compare | Only these make a row changed (e.g. `price`, `stock`). Empty: every top-level field. | | Fields to ignore | Never make a row changed (e.g. `scrapedAt`). | | Report new / changed / removed / unchanged rows | Which kinds to write (new, changed and removed by default). | | First run with no earlier data | Report every row as new (default), or only remember them. | | Comparison name | Keeps a comparison's memory apart from others (or starts afresh). | | Max rows per run | Cap the rows written; the rest is reported next run. | Values compare as they read: `12` and `"12"` are equal (so a CSV export and a dataset compare fine), and so are `null`, `""`, `[]`, `{}` and a missing field; text that differs only in how an accent is encoded is the same. Objects and lists compare as a whole. A full example, for a product scraper chained into Dataset Diff: ```json { "datasetId": "{{resource.defaultDatasetId}}", "keyFields": ["url"], "compareFields": ["price", "inStock"], "includeRemoved": true, "firstRun": "baseline" } ``` ### Output One dataset row per change: the row's own fields plus `changeType`, `changedFields` and `previousValues`. A field of yours with one of those three names is overwritten. The `RUN_STATS` record has the counts. The dataset can be downloaded as CSV, JSON, Excel, XML or HTML at any time, or passed on to the next actor. ### Run it on a schedule, or from your own code Chained after a scraper (above) is the usual set-up. Otherwise, save your input as a [task](https://docs.apify.com/platform/actors/running/tasks) and add it to a [schedule](https://docs.apify.com/platform/schedules), for example daily after a CSV feed you watch is updated. Collect the changes from the [API](https://apify.com/humble-echidna/dataset-diff/api) (`GET https://api.apify.com/v2/actor-tasks//runs/last/dataset/items?status=SUCCEEDED`, with your API token), a [webhook](https://docs.apify.com/platform/integrations/webhooks), or Make, Zapier and n8n through [Apify's integrations](https://docs.apify.com/platform/integrations). #### Can I use Dataset Diff from an AI agent (MCP)? Yes, through [Apify's MCP server](https://docs.apify.com/platform/integrations/mcp), from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=humble-echidna/dataset-diff" } } } ``` To use an [Apify API token](https://console.apify.com/settings/integrations) instead of signing in, add `"headers": {"Authorization": "Bearer "}` next to `url`. An agent that has run a scraper twice can pass both datasets, e.g. `{"datasetId": "", "previousDatasetId": "", "keyFields": ["url"]}`, and get back only what changed. ### Who it's for Anyone who runs a scraper on a schedule and only cares about the difference: new listings or jobs, price and stock changes, products that disappeared, companies that changed a field. Also for comparing two exports (CSV, Excel) of the same table. ### Why this one? - **Works with any actor's data.** Your own datasets or CSV/TSV, JSON, JSON Lines and Excel files, with any key, nested fields included. No scraper of its own, so nothing on a website can break it. - **Says what changed.** Each changed row lists the fields that differ and their old values; removed rows come back as they were. - **Never loses a change.** The memory only moves on for rows you actually got, so a run cut by your limits reports the rest next time. - **Cheap when nothing happens.** A quiet run costs $0.01 per 1,000 rows compared; changes $1.00 per 1,000 ($0.90 on Starter), with no compute, proxy or storage bill on top. - **Tells you what happened.** The status counts new, changed, removed and unchanged rows, and warns about field names no row has (a typo in a key field is the most common mistake). - **Polite and safe with files.** It identifies itself honestly (User-Agent `HumbleEchidnaApify`), follows each site's robots.txt, and only requests public web addresses on the standard ports (80 and 443). ### Limits - Up to 100,000 rows on each side per run; the status says so if the new data has more (filter it first with [Dataset Transformer](https://apify.com/humble-echidna/dataset-transform)). Very wide or long rows can stop a run earlier at the default 1 GB memory; the status says to give the run more memory. - Files: up to 200 MB (after decompression), public web addresses on ports 80 and 443; Excel's first sheet only. - A key appearing twice in the same data counts once (the first row); rows with every key field empty are skipped. - Removed rows are only checked when every row of the new data was read. - The last run's snapshot is kept in a key-value store in your account (`dataset-diff-memory`); rows over 20 KB keep only their key, so their old values aren't shown. - A single output row can be up to 5 MB; larger ones are skipped and counted. ### FAQ #### What if my rows have no single id? Use several key fields (`name, city`), or leave **Key fields** empty to compare whole rows: then a row that changed shows up as one removed and one new row. #### Why is every row "new"? It's the comparison's first run, or its memory belongs to another set-up: the key, compared and ignored fields, and the comparison's name are part of what's remembered. Changing them starts a new comparison. #### Why are rows "changed" when nothing important moved? The scraper writes something new on every run (a timestamp, a position, a session id). Add it to **Fields to ignore**, or list only the fields you care about in **Fields to compare**. #### Can it compare datasets from different actors? Yes: put one in **New data** and the other in **Compare with**. Both are read with your own account's access. #### Is it legal to compare datasets and files with it? It only processes data you choose: your own Apify datasets, and files at addresses you supply, fetched politely (robots.txt honoured, honest User-Agent). Make sure you're allowed to use the files you point it at. The example data is the US Treasury's Fiscal Data API, which is "offered free, without restriction" for commercial and non-commercial use. #### Something that used to work now fails. Why? The run log and the status say what went wrong. Please open an issue with the input you used. ### Related actors | Actor | Use it when | |---|---| | [Dataset Transformer](https://apify.com/humble-echidna/dataset-transform) | You want to filter, dedupe or reshape the changes (or the data before comparing it), or save them as CSV. | | [Join Datasets](https://apify.com/humble-echidna/dataset-join) | You want to add fields from another dataset to each row, matched by a key (VLOOKUP for datasets). | | [Website Change Monitor](https://apify.com/humble-echidna/website-change-monitor) | You want to watch web pages themselves for changes, with a diff of the text. | ### Feedback and support Found a bug, or need a comparison that isn't here? Open an issue on the **Issues** tab with the input you used. ### Versions Current version: **0.1**. See the Changelog tab for what changed in each version. # Changelog This Actor's version history is a separate document: https://apify.com/humble-echidna/dataset-diff/changelog.md # Actor input Schema ## `datasetId` (type: `string`): The latest data: the id or name of one of your own Apify datasets, e.g. the defaultDatasetId of a scraper's latest run (or pick one in the form). To run this actor after every run of a scraper in an integration, `{{resource.defaultDatasetId}}`. Read-only, with your own account's access. When it's set, fileUrl is ignored. ## `fileUrl` (type: `string`): Used only when datasetId is empty: a direct link to a CSV/TSV, JSON (a list of objects), JSON Lines or Excel (.xlsx, first sheet) file on a public web address (http/https, ports 80 and 443), up to 200 MB. The default is the US Treasury's official exchange rates for 2025-12-31 (174 currencies), the example's new data. ## `previousDatasetId` (type: `string`): The older data to compare with, e.g. an earlier run's dataset. Leave this and previousFileUrl empty to compare with what this comparison saw on its last run (the usual set-up for a schedule: each run reports what's new since the one before). When it's set, previousFileUrl is ignored. ## `previousFileUrl` (type: `string`): Used only when previousDatasetId is empty: a direct link to the older CSV, JSON, JSONL or Excel file. Leave empty (and previousDatasetId too) to compare with the last run. The default is the same Treasury table for 2025-09-30, the example's older data; it's only used together with the example's new data, never with your own. ## `keyFields` (type: `array`): The field or fields that identify the same row in both, one per line or comma-separated: e.g. `url`, `id`, or `name, city`; dots reach nested fields (`product.sku`). Two rows with the same key are compared field by field (changed or unchanged); a key only in the new data is new, only in the older data removed. Leave empty to compare whole rows: then any difference reads as one removed and one new row. ## `compareFields` (type: `array`): Only differences in these fields make a row changed, one per line (e.g. `price`, `stock`). Leave empty (the default) to compare every top-level field except the ignored ones. Values compare as they read: 12 and "12" are equal, and so are null, an empty text and a missing field. ## `ignoreFields` (type: `array`): Fields that never make a row changed, one per line: typically what a scraper stamps on every run, like `scrapedAt`, `crawledAt` or a position in the results. Default: none. ## `includeNew` (type: `boolean`): Default true: write a row for each key that wasn't in the older data (changeType `new`). ## `includeChanged` (type: `boolean`): Default true: write the new version of each row whose compared fields differ (changeType `changed`), with changedFields and the old values in previousValues. ## `includeRemoved` (type: `boolean`): Default true: write the old version of each key that is gone from the new data (changeType `removed`). ## `includeUnchanged` (type: `boolean`): Default false. When true, also write rows that didn't change (changeType `unchanged`), e.g. to get the full current data with a change label on every row. Each is a charged row. ## `firstRun` (type: `string`): Only when comparing with the last run, on the very first run (nothing remembered yet). `all` (the default): every row is reported as new. `baseline`: the rows are remembered and nothing is reported, so a schedule only reports what's new from its second run on. ## `monitorName` (type: `string`): Only when comparing with the last run: a name that keeps this comparison's memory apart from others, e.g. `shop-a-prices`. Leave empty (the default) to name it after the saved task running it, or else the actor that wrote the dataset, or else the file URL. Set it when one scraper feeds two separate comparisons, or to start afresh (a new name has no memory). ## `maxResults` (type: `integer`): Stop after writing this many rows, e.g. 100. Minimum 1; leave empty (the default) for no limit. The run also stops cleanly at the maximum cost per run you set in the run options. Changes not written because of either limit aren't forgotten: the next run reports them. ## Actor input object example ```json { "fileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300", "previousFileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-09-30&page[size]=300", "keyFields": [ "country_currency_desc" ], "compareFields": [ "exchange_rate" ], "includeNew": true, "includeChanged": true, "includeRemoved": true, "includeUnchanged": false, "firstRun": "all" } ``` # Actor output Schema ## `results` (type: `string`): No description ## `runStats` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "fileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300", "previousFileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-09-30&page[size]=300", "keyFields": [ "country_currency_desc" ], "compareFields": [ "exchange_rate" ] }; // Run the Actor and wait for it to finish const run = await client.actor("humble-echidna/dataset-diff").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "fileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300", "previousFileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-09-30&page[size]=300", "keyFields": ["country_currency_desc"], "compareFields": ["exchange_rate"], } # Run the Actor and wait for it to finish run = client.actor("humble-echidna/dataset-diff").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "fileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300", "previousFileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-09-30&page[size]=300", "keyFields": [ "country_currency_desc" ], "compareFields": [ "exchange_rate" ] }' | apify call humble-echidna/dataset-diff --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,humble-echidna/dataset-diff" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/z2f9uCrOg2J1IaBOE/builds/axfqv2vbBSIe5j00r/openapi.json