Dataset Drift & QA Monitor

Stop finding out your scrapers broke three days late. Point this Actor at any Apify dataset or JSON endpoint and it watches the data itself — schema, row counts, null rates, numeric distributions and row-level changes — not just whether the run exited zero.

What it does

A scraper that returns 200 OK and an empty price field on every row is broken, but every monitoring tool built around run status will call it healthy. This Actor profiles the output data and compares each run's fingerprint against the previous one.

It alerts on the failures that actually matter in production:

• New or missing fields — the site changed its markup and your selector silently stopped matching.
• Row-count deltas — yesterday 5,000 rows, today 12.
• Null-rate jumps — the field is still there but is now empty.
• Numeric distribution shifts — prices suddenly 200% higher, or parsed as strings.
• New-value bursts — a categorical field filling with unexpected values.
• Row-level changes — matched by your identity keys, so you see which records changed.

Alerts route to a webhook, Slack, Discord or email.

Features

• Watches data, not run status — catches silent breakage that status monitors miss.
• Six alert classes — new/missing fields, row-count delta, null-rate delta, numeric-stats delta, new-value burst, row-level changes.
• Tunable sensitivity — every threshold is a percentage you set in alertOn.
• Any JSON source — an Apify datasetId or any URL returning a JSON array.
• Row-level matching — set identity keys and get per-record change detection.
• Slack and Discord built in, plus generic signed webhooks and email.
• Field scoping — fieldsToWatch limits profiling to the columns that matter.
• Isolated state — storeName keeps separate monitors from colliding.

What people use it for

• Monitor your own scheduled scrapers for silent breakage.
• Data-quality gates in an ETL pipeline before loading a warehouse.
• Vendor API monitoring — catch upstream schema changes before they hit production.
• Regression detection after deploying a scraper change.
• SLA reporting — a durable record of data health over time.

Tuning alertOn

Every threshold is a percentage, so you can set sensitivity per signal:

{
  "alertOn": {
    "newFields": true,
    "missingFields": true,
    "rowCountDeltaPct": 20,
    "nullRateDeltaPct": 25,
    "newValueBurstPct": 40,
    "numericStatsDeltaPct": 30,
    "rowLevelChangesPct": 15
  }
}

Start permissive, watch a week of real runs, then tighten. Data that is naturally volatile (prices, inventory) needs looser numeric thresholds than a reference dataset that should barely move.

Row-level change detection

Set keys to the fields that identify a row — ["id"], ["productId"], ["sku", "region"]. The Actor then matches rows across runs and reports which specific records changed, rather than just telling you the aggregate moved. Without keys, monitoring is aggregate-only.

The first run is a baseline

There's no previous fingerprint on run one, so it establishes the baseline and won't alert. Real comparisons begin on the second run.

Input

Supply either datasetId (an Apify dataset) or datasetUrl (any JSON array endpoint). Set keys if you want row-level change detection.

Field	Type	Default	What it does
datasetId	string	—	ID of an Apify dataset to monitor (e.g. the dataset produced by one of your scheduled actors).
datasetUrl	string	https://jsonplaceholder.typicode.com/users	OR a URL returning a JSON array (another monitored source).
fieldsToWatch	array	[]	Subset of fields to profile. Empty = all top-level fields.
keys	array	[]	For object datasets: stable identity keys for cross-run row matching (e.g. ['id'] or ['productId']).
alertOn	object	{"newFields": true, "missingFields": true, "rowCountDeltaPct": 20, "nullRateDeltaPct": 25, "newValueBurstPct": 40, "numericStatsDeltaPct": 30, "rowLevelChangesPct": 15}	Toggle individual alerts and set sensitivity (percent deltas).
storeName	string	dataset-drift-qa-state	Named key-value store holding the previous fingerprint.
webhookUrl	string	—	POST alert payloads to this URL on every run (or only when alerts fire).
webhookEvents	array	["on_alert"]	When to fire webhook: 'always' (every run), 'on_alert' (only when status=ALERT), 'on_change' (schema/rowcount changed).
webhookSecret	string	—	HMAC-SHA256 secret for verifying webhook payloads. If set, X-Signature header is included.
slackWebhookUrl	string	—	Incoming webhook URL for Slack. Sends formatted message on alert.
discordWebhookUrl	string	—	Discord webhook URL. Sends embed on alert.
emailAlert	object	{"enabled": false, "to": [], "subjectPrefix": "[Dataset Drift Alert]"}	Email notification settings. Requires APIFY_TOKEN with email scope or external SMTP service.

Input example

{
  "datasetUrl": "https://api.apify.com/v2/key-value-stores/dcDAdW7LfWrrUKAIt/records/fixture?REDACTED",
  "keys": [
    "id"
  ],
  "alertOn": {
    "newFields": true,
    "missingFields": true,
    "rowCountDeltaPct": 20,
    "nullRateDeltaPct": 25,
    "newValueBurstPct": 40,
    "numericStatsDeltaPct": 30,
    "rowLevelChangesPct": 15
  },
  "storeName": "ddqa-proof-final",
  "webhookEvents": [
    "on_alert"
  ],
  "emailAlert": {
    "enabled": false,
    "to": [],
    "subjectPrefix": "[Dataset Drift Alert]"
  }
}

Output

One record per check: when it ran, the source, row and field counts, an overall status (OK or ALERT), the alert count, and the individual alerts with previous versus current values and the percentage change.

Every dataset record contains: checkedAt, source, rowCount, fieldCount, alertCount, status, alerts, fingerprint, deliveredTo.

Output example

A real record from a run of this Actor:

{
  "checkedAt": "2026-08-21T00:34:24.659Z",
  "source": "https://api.apify.com/v2/key-value-stores/dcDAdW7LfWrrUKAIt/records/fixture?REDACTED",
  "rowCount": 3,
  "fieldCount": 3,
  "alertCount": 3,
  "status": "ALERT",
  "alerts": [
    {
      "type": "numeric_stats_delta",
      "field": "price",
      "stat": "max",
      "prev": 30,
      "now": 99,
      "pct": 230
    },
    {
      "type": "numeric_stats_delta",
      "field": "price",
      "stat": "mean",
      "prev": 20,
      "now": 46.3333,
      "pct": 131.7
    },
    {
      "type": "row_level_changes",
      "changed": 1,
      "added": 0,
      "removed": 0,
      "pct": 33.3
    }
  ],
  "fingerprint": {
    "rowCount": 3,
    "fields": [
      "id",
      "name",
      "price"
    ],
    "nullRate": {
      "id": 0,
      "name": 0,
      "price": 0
    },
    "valueSets": {
      "id": [
        "1",
        "2",
        "3"
      ],
      "name": [
        "BETA-CHANGED",
        "alpha",
        "gamma"
      ],
      "price": [
        "10",
        "30",
        "99"
      ]
    },
    "numericStats": {
      "id": {
        "min": 1,
        "max": 3,
        "mean": 2
      },
      "price": {
        "min": 10,
        "max": 99,
        "mean": 46.3333
      }
    },
    "rowHashes": {
      "1": [
        "98a18e525b5ddb38"
      ],
      "2": [
        "59dd4a5192244817"
      ],
      "3": [
        "7a0f703b21de0954"
      ]
    }
  },
  "deliveredTo": []
}

Export the dataset as JSON, CSV, Excel, XML or JSONL from the Console, or pull it programmatically through the Apify API and any of the official clients.

How to use it

1. Click Try for free (or Start if you already have an Apify account).
2. Fill in the input fields described above — the defaults already produce a working run.
3. Press Start and watch the log; results stream into the dataset as they are found.
4. When the run finishes, open the Output/Storage tab and export as JSON, CSV or Excel.

Runs can be scheduled (hourly, daily, weekly) and wired into Slack, Google Sheets, Zapier, Make, webhooks or your own backend through Apify integrations. Everything the Console does is also available over the Apify API.

Pricing

This Actor is billed on Apify's pay-per-event model: a small charge when a run starts, plus a charge for each result written to the dataset. You only pay for records you actually receive — a run that finds nothing costs only the start event. Current rates are always shown on the Pricing tab of this page, and the run log prints your usage as it goes.

Free-plan credits from Apify cover a large amount of light usage, so you can evaluate the Actor before committing to anything.

FAQ

Why isn't run status enough?

Because the worst scraper failures succeed. A site changes its markup, your selector stops matching, and the run finishes cleanly with empty fields. Only profiling the data catches that.

What can it monitor?

Any Apify dataset by datasetId, or any URL returning a JSON array via datasetUrl — including your own API endpoints and third-party APIs.

How do I connect it to Slack?

Paste an incoming-webhook URL into slackWebhookUrl. Discord works the same way via discordWebhookUrl. Both send formatted alerts.

I'm getting too many alerts. What should I change?

Raise the percentage thresholds in alertOn, narrow fieldsToWatch to the columns that matter, and switch webhook events to on_alert so healthy runs stay quiet.

Does it store my data?

It stores a statistical fingerprint — schema, counts, null rates, numeric summaries and row hashes — in the named key-value store, not your full dataset.

Can one Actor watch several datasets?

Run one task per dataset with its own storeName. That keeps each baseline clean and makes alerts unambiguous.

Other Actors by cynix_dev

Actor	What it does
Website to RAG Chunks	Crawl any website and turn its pages into clean, chunked, metadata-rich Markdown records ready for RAG pipelines, vector stores, …
Page Change Monitor	Monitor web pages for content changes. Diffs each run against the previous snapshot and emits structured change records with …
OpenStreetMap Geocoder	Forward and reverse geocoding via the free Komoot Photon / OpenStreetMap service. No API key, no scraping, ODbL data.
Page Change Monitor	Monitor web pages for content changes. Diffs each run against the previous snapshot and emits structured change records with …
Website to RAG Chunks	Crawl any website and turn its pages into clean, chunked, metadata-rich Markdown records ready for RAG pipelines, vector stores, …

Legal and responsible use

This Actor collects only publicly available information. You are responsible for how you use the data, including compliance with the target site's Terms of Service, robots directives, copyright, and data protection law such as GDPR and CCPA. Do not use it to gather personal data without a lawful basis.

Support and feedback

Found a bug, hit a site change, or need an extra field? Open a ticket on the Issues tab of this Actor — issues are read and fixed. Feature requests and custom-scraper enquiries are welcome through the same channel.