Dataset Delta

Turn full scraper outputs into a small, actionable change feed.

Dataset Delta compares any two Apify datasets—or tracks one dataset between scheduled runs—and returns only records that were added, removed, or changed. It is designed for price monitoring, job alerts, inventory tracking, lead-list updates, regulatory feeds, and any recurring Actor workflow where users care about what changed, not another complete export.

What it produces

Each output row contains:

• changeType: added, removed, changed, or optionally unchanged
• key: the unique field or composite key used to match the record
• changedFields: exact dot paths that changed
• before: the old record
• after: the new record

The run's OUTPUT record contains totals and source metadata.

Modes

Compare two datasets

Provide an older baselineDatasetId and a newer currentDatasetId. This mode is stateless and supports up to 50,000 records per dataset.

Track one dataset

Provide the current dataset and a stable stateKey. Dataset Delta saves a private snapshot in a named key-value store. By default, the first run only initializes the snapshot and emits no changes; enable emitInitialSnapshot if every existing record should be emitted as added. Later runs report the delta. Use a different stateKey for each scheduled source.

If a user's maximum charge is reached before every change is emitted, the snapshot is deliberately not advanced. The next run can therefore recover the unprocessed changes.

Tracking snapshots are capped at 8 MB. Use Compare mode for larger datasets.

Example input

{
  "mode": "compare",
  "baselineDatasetId": "OLDER_DATASET_ID",
  "currentDatasetId": "NEWER_DATASET_ID",
  "keyFields": ["url"],
  "ignoreFields": ["scrapedAt", "metadata.requestId"],
  "maxItems": 10000
}

For records that are only unique in combination, use a composite key:

{
  "keyFields": ["company.id", "locationId"]
}

Duplicate or missing keys fail clearly instead of silently producing incorrect changes.

Pricing

Dataset Delta uses pay-per-event pricing:

• Actor start: Apify's standard low-cost start event
• delta-record: $0.002 per emitted change

Unchanged records do not create result charges unless you explicitly enable includeUnchanged.

Requirements and limits

• Both datasets must be accessible to the account running Dataset Delta.
• Every record must contain the configured unique key fields.
• Duplicate keys fail the run instead of producing an ambiguous comparison.
• Compare mode supports at most 50,000 records per dataset.
• Tracking snapshots are limited to 8 MB; use Compare mode for larger datasets.
• No external API key, proxy, browser, LLM, or third-party service is required.

Local development

Requires Node.js 20 or newer.

npm install
npm test
npm start

The checked-in local storage fixture compares two tiny datasets and ignores scrapedAt.