Apify Dataset Diff API - Find New, Removed & Changed Rows

What does Apify Dataset Diff API do?

Apify Dataset Diff API compares two Apify datasets and returns a clean feed of only the records that were added, removed, or changed. You can compare a new dataset against a previous dataset ID, or run it repeatedly with saveSnapshot=true so each run compares the latest dataset against the last saved snapshot.

This Actor is built for scheduled scrapers, data pipelines, price monitors, lead feeds, SEO checks, and automation tools. On Apify, you can run it from the UI, call it through the API, schedule it after another Actor, monitor failures, and send the JSON output into Make, Zapier, n8n, Slack, or a custom webhook.

Why use Apify Dataset Diff API?

Most scrapers output the full dataset every time. That is expensive and noisy when your real question is: "what changed since the last run?" This Actor turns full scraper outputs into an automation-ready change feed.

Use it to detect new leads, removed products, price changes, stock changes, SERP movement, policy text updates, listing edits, or any repeated dataset where rows have stable identifiers. It supports one or more key fields, dot-notation fields such as product.id, changed field names, before/after values, unchanged counts, and snapshot storage for hands-off scheduled runs.

How to use Apify Dataset Diff API

1. Run the scraper or Actor that produces your latest dataset.
2. Open this Actor and paste the latest dataset ID into newDatasetId.
3. Add one or more keyFields, such as id, url, sku, or product.id.
4. Optionally paste a previousDatasetId. If you leave it empty, the Actor compares against the saved snapshot.
5. Keep saveSnapshot=true when you want the current dataset to become the baseline for the next run.
6. Start the Actor and open the Output tab to inspect changed rows and the summary JSON.
7. Use the API, webhooks, or integrations to route the change feed into your workflow.

Input

See the Input tab for the full configuration. The main fields are:

{
    "newDatasetId": "abc123",
    "previousDatasetId": "xyz789",
    "keyFields": ["url"],
    "compareFields": ["price", "availability"],
    "ignoreFields": ["scrapedAt"],
    "saveSnapshot": true,
    "freeSampleMode": false
}

Leave compareFields empty to compare all fields except keys and ignored fields. Use freeSampleMode=true to cap the run to a small number of rows while testing.

Output

The default dataset contains one item per added, removed, or changed row. The default key-value store contains OUTPUT, a webhook-ready JSON envelope with counts and grouped changes. You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

{
    "counts": {
        "previousRows": 2,
        "currentRows": 2,
        "added": 1,
        "removed": 1,
        "changed": 1,
        "unchanged": 0
    },
    "changes": {
        "changed": [
            {
                "type": "changed",
                "key": { "url": "https://example.com/product/1" },
                "changedFields": ["price"],
                "changes": {
                    "price": { "before": 19.99, "after": 17.99 }
                }
            }
        ]
    }
}

Data table

Field	Type	Description
type	string	added, removed, or changed.
key	object	The key field values that identify the row.
changedFields	array	Field names that changed. Empty for added and removed rows.
changes	object	Before/after values for changed fields.
before	object or null	Previous row for removed and changed records.
after	object or null	Current row for added and changed records.
previousIndex	number or null	Zero-based row index in the previous source.
currentIndex	number or null	Zero-based row index in the current source.

Pricing / Cost estimation

How much does it cost to compare Apify datasets? The recommended pricing model is pay per event: $0.01 per Actor start, $0.75 per 1,000 rows compared, and optionally $0.10 per webhook delivery batch. The goal is to keep runs cheap enough to attach this Actor to every scheduled scraper.

For trials, keep free sample mode enabled. It compares only a small row sample, which is useful for confirming keys and output shape before running against full datasets.

Tips or Advanced options

Choose stable keys. URLs, product IDs, listing IDs, and composite keys like ["storeId", "sku"] work well. Avoid keys that change between runs, such as scrape timestamps or row numbers.

Ignore volatile fields such as scrapedAt, run IDs, ranking timestamps, and debug metadata. If nested objects are noisy, compare only the fields you care about with compareFields.

For scheduled workflows, run your scraper first, pass its dataset ID into newDatasetId, leave previousDatasetId empty, and keep saveSnapshot=true. The first run creates the baseline; later runs produce the change feed.

FAQ, disclaimers, and support

What happens on the first snapshot run?

If no previousDatasetId is supplied and no snapshot exists yet, the Actor treats the current dataset as the first snapshot. It saves the snapshot when saveSnapshot=true and reports the current rows as added against an empty baseline.

Can I send results to Make, Zapier, n8n, or Slack?

Yes. Use the OUTPUT key-value store record as webhook-ready JSON, or provide webhookUrl to POST batches directly to your automation endpoint.

Is it legal to compare datasets?

This Actor compares datasets you provide. Your source datasets may contain public or personal data depending on the scraper that created them. You should only process personal data when you have a legitimate reason and comply with GDPR and other applicable privacy regulations.

Where can I get support?

Use the Issues tab on the Actor page for bugs, feature requests, and examples of datasets that do not compare as expected. Custom data normalization, schema mapping, and companion Actors can be built for specialized pipelines.