# Apify Dataset Diff API - Find New, Removed & Changed Rows (`vincesoft/dataset-diff-change`) Actor Compare Apify datasets and get a clean change feed showing only new, removed, and modified records. Perfect for scheduled scrapers, price monitors, lead feeds, SEO checks, and automation workflows in Make, Zapier, n8n, AI agents, and custom APIs. - **URL**: https://apify.com/vincesoft/dataset-diff-change.md - **Developed by:** [Vincent Pestana](https://apify.com/vincesoft) (community) - **Categories:** Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.75 / 1,000 compared rows This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### What does Apify Dataset Diff API do? **Apify Dataset Diff API** compares two [Apify datasets](https://docs.apify.com/platform/storage/dataset) and returns a clean feed of only the records that were added, removed, or changed. You can compare a new dataset against a previous dataset ID, or run it repeatedly with `saveSnapshot=true` so each run compares the latest dataset against the last saved snapshot. This Actor is built for scheduled scrapers, data pipelines, price monitors, lead feeds, SEO checks, and automation tools. On Apify, you can run it from the UI, call it through the API, schedule it after another Actor, monitor failures, and send the JSON output into Make, Zapier, n8n, Slack, or a custom webhook. ### Why use Apify Dataset Diff API? Most scrapers output the full dataset every time. That is expensive and noisy when your real question is: "what changed since the last run?" This Actor turns full scraper outputs into an automation-ready change feed. Use it to detect new leads, removed products, price changes, stock changes, SERP movement, policy text updates, listing edits, or any repeated dataset where rows have stable identifiers. It supports one or more key fields, dot-notation fields such as `product.id`, changed field names, before/after values, unchanged counts, and snapshot storage for hands-off scheduled runs. ### How to use Apify Dataset Diff API 1. Run the scraper or Actor that produces your latest dataset. 2. Open this Actor and paste the latest dataset ID into `newDatasetId`. 3. Add one or more `keyFields`, such as `id`, `url`, `sku`, or `product.id`. 4. Optionally paste a `previousDatasetId`. If you leave it empty, the Actor compares against the saved snapshot. 5. Keep `saveSnapshot=true` when you want the current dataset to become the baseline for the next run. 6. Start the Actor and open the Output tab to inspect changed rows and the summary JSON. 7. Use the API, webhooks, or integrations to route the change feed into your workflow. ### Input See the Input tab for the full configuration. The main fields are: ```json { "newDatasetId": "abc123", "previousDatasetId": "xyz789", "keyFields": ["url"], "compareFields": ["price", "availability"], "ignoreFields": ["scrapedAt"], "saveSnapshot": true, "freeSampleMode": false } ```` Leave `compareFields` empty to compare all fields except keys and ignored fields. Use `freeSampleMode=true` to cap the run to a small number of rows while testing. ### Output The default dataset contains one item per added, removed, or changed row. The default key-value store contains `OUTPUT`, a webhook-ready JSON envelope with counts and grouped changes. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. ```json { "counts": { "previousRows": 2, "currentRows": 2, "added": 1, "removed": 1, "changed": 1, "unchanged": 0 }, "changes": { "changed": [ { "type": "changed", "key": { "url": "https://example.com/product/1" }, "changedFields": ["price"], "changes": { "price": { "before": 19.99, "after": 17.99 } } } ] } } ``` ### Data table | Field | Type | Description | | --------------- | -------------- | ----------------------------------------------------------- | | `type` | string | `added`, `removed`, or `changed`. | | `key` | object | The key field values that identify the row. | | `changedFields` | array | Field names that changed. Empty for added and removed rows. | | `changes` | object | Before/after values for changed fields. | | `before` | object or null | Previous row for removed and changed records. | | `after` | object or null | Current row for added and changed records. | | `previousIndex` | number or null | Zero-based row index in the previous source. | | `currentIndex` | number or null | Zero-based row index in the current source. | ### Pricing / Cost estimation How much does it cost to compare Apify datasets? The recommended pricing model is pay per event: $0.01 per Actor start, $0.75 per 1,000 rows compared, and optionally $0.10 per webhook delivery batch. The goal is to keep runs cheap enough to attach this Actor to every scheduled scraper. For trials, keep free sample mode enabled. It compares only a small row sample, which is useful for confirming keys and output shape before running against full datasets. ### Tips or Advanced options Choose stable keys. URLs, product IDs, listing IDs, and composite keys like `["storeId", "sku"]` work well. Avoid keys that change between runs, such as scrape timestamps or row numbers. Ignore volatile fields such as `scrapedAt`, run IDs, ranking timestamps, and debug metadata. If nested objects are noisy, compare only the fields you care about with `compareFields`. For scheduled workflows, run your scraper first, pass its dataset ID into `newDatasetId`, leave `previousDatasetId` empty, and keep `saveSnapshot=true`. The first run creates the baseline; later runs produce the change feed. ### FAQ, disclaimers, and support #### What happens on the first snapshot run? If no `previousDatasetId` is supplied and no snapshot exists yet, the Actor treats the current dataset as the first snapshot. It saves the snapshot when `saveSnapshot=true` and reports the current rows as added against an empty baseline. #### Can I send results to Make, Zapier, n8n, or Slack? Yes. Use the `OUTPUT` key-value store record as webhook-ready JSON, or provide `webhookUrl` to POST batches directly to your automation endpoint. #### Is it legal to compare datasets? This Actor compares datasets you provide. Your source datasets may contain public or personal data depending on the scraper that created them. You should only process personal data when you have a legitimate reason and comply with GDPR and other applicable privacy regulations. #### Where can I get support? Use the Issues tab on the Actor page for bugs, feature requests, and examples of datasets that do not compare as expected. Custom data normalization, schema mapping, and companion Actors can be built for specialized pipelines. # Actor input Schema ## `newDatasetId` (type: `string`): Dataset ID or name containing the latest records to compare. ## `previousDatasetId` (type: `string`): Optional dataset ID or name containing the previous records. Leave empty to compare against the saved snapshot. ## `keyFields` (type: `array`): One or more field names that uniquely identify a row. Dot notation is supported, for example product.id. ## `compareFields` (type: `array`): Optional list of fields to compare. Leave empty to compare all fields except key fields and ignored fields. ## `ignoreFields` (type: `array`): Fields to ignore during comparison. Dot notation ignores nested paths. ## `saveSnapshot` (type: `boolean`): Save the new dataset as the snapshot for the next run. ## `snapshotKey` (type: `string`): Key-value store key used for saved snapshots when no previous dataset is provided. ## `freeSampleMode` (type: `boolean`): Cap compared rows to a small sample for quick trials. ## `freeSampleRowLimit` (type: `integer`): Maximum rows loaded from each source when free sample mode is enabled. ## `maxRows` (type: `integer`): Optional hard limit for rows loaded from each source. Use 0 for unlimited unless free sample mode is enabled. ## `webhookUrl` (type: `string`): Optional HTTP or HTTPS URL to receive the diff payload in webhook-ready JSON batches. ## `webhookBatchSize` (type: `integer`): Maximum number of change records per webhook request. ## Actor input object example ```json { "newDatasetId": "NEW_DATASET_ID", "keyFields": [ "id" ], "compareFields": [], "ignoreFields": [], "saveSnapshot": true, "snapshotKey": "LAST_SNAPSHOT", "freeSampleMode": true, "freeSampleRowLimit": 100, "maxRows": 0, "webhookBatchSize": 100 } ``` # Actor output Schema ## `changes` (type: `string`): No description ## `summary` (type: `string`): No description ## `snapshot` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "newDatasetId": "NEW_DATASET_ID", "keyFields": [ "id" ] }; // Run the Actor and wait for it to finish const run = await client.actor("vincesoft/dataset-diff-change").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "newDatasetId": "NEW_DATASET_ID", "keyFields": ["id"], } # Run the Actor and wait for it to finish run = client.actor("vincesoft/dataset-diff-change").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "newDatasetId": "NEW_DATASET_ID", "keyFields": [ "id" ] }' | apify call vincesoft/dataset-diff-change --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=vincesoft/dataset-diff-change", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Apify Dataset Diff API - Find New, Removed & Changed Rows", "description": "Compare Apify datasets and get a clean change feed showing only new, removed, and modified records. Perfect for scheduled scrapers, price monitors, lead feeds, SEO checks, and automation workflows in Make, Zapier, n8n, AI agents, and custom APIs.", "version": "0.1", "x-build-id": "pOrl2rv4b0cACBuXC" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/vincesoft~dataset-diff-change/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-vincesoft-dataset-diff-change", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/vincesoft~dataset-diff-change/runs": { "post": { "operationId": "runs-sync-vincesoft-dataset-diff-change", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/vincesoft~dataset-diff-change/run-sync": { "post": { "operationId": "run-sync-vincesoft-dataset-diff-change", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "newDatasetId", "keyFields" ], "properties": { "newDatasetId": { "title": "New dataset ID", "type": "string", "description": "Dataset ID or name containing the latest records to compare." }, "previousDatasetId": { "title": "Previous dataset ID", "type": "string", "description": "Optional dataset ID or name containing the previous records. Leave empty to compare against the saved snapshot." }, "keyFields": { "title": "Key fields", "type": "array", "description": "One or more field names that uniquely identify a row. Dot notation is supported, for example product.id.", "default": [ "id" ], "items": { "type": "string" } }, "compareFields": { "title": "Compare fields", "type": "array", "description": "Optional list of fields to compare. Leave empty to compare all fields except key fields and ignored fields.", "default": [], "items": { "type": "string" } }, "ignoreFields": { "title": "Ignore fields", "type": "array", "description": "Fields to ignore during comparison. Dot notation ignores nested paths.", "default": [], "items": { "type": "string" } }, "saveSnapshot": { "title": "Save snapshot", "type": "boolean", "description": "Save the new dataset as the snapshot for the next run.", "default": true }, "snapshotKey": { "title": "Snapshot key", "type": "string", "description": "Key-value store key used for saved snapshots when no previous dataset is provided.", "default": "LAST_SNAPSHOT" }, "freeSampleMode": { "title": "Free sample mode", "type": "boolean", "description": "Cap compared rows to a small sample for quick trials.", "default": true }, "freeSampleRowLimit": { "title": "Free sample row limit", "minimum": 1, "type": "integer", "description": "Maximum rows loaded from each source when free sample mode is enabled.", "default": 100 }, "maxRows": { "title": "Maximum rows", "minimum": 0, "type": "integer", "description": "Optional hard limit for rows loaded from each source. Use 0 for unlimited unless free sample mode is enabled.", "default": 0 }, "webhookUrl": { "title": "Webhook URL", "type": "string", "description": "Optional HTTP or HTTPS URL to receive the diff payload in webhook-ready JSON batches." }, "webhookBatchSize": { "title": "Webhook batch size", "minimum": 1, "type": "integer", "description": "Maximum number of change records per webhook request.", "default": 100 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```