Scraper Freshness Watchdog

A lightweight Apify Actor for teams whose scheduled scrapers silently return stale, empty, duplicated, or sharply reduced datasets.

It accepts either inline rows or a permission-respecting public HTTPS JSON endpoint and emits one deterministic health record. It does not log in, bypass challenges, use proxies, or scrape private data.

Follow the 60-second setup tutorial and inspect a deterministic failure example.

Checks

• newest timestamp age against maxAgeMinutes
• minimum row count
• duplicate IDs
• row-count regression of 25% or more against either previousRowCount or the automatically stored prior run
• automatic per-feed snapshot persistence in a named key-value store; public source URLs identify the feed automatically, while inline data can use monitorKey
• missing or invalid timestamps
• optional failOnIssues mode that saves the full health result, then marks an unhealthy run failed so existing Apify task failure notifications or webhooks can alert the owner without this Actor handling notification credentials

Example input

{
  "rows": [
    { "id": "a", "updatedAt": "2026-07-14T08:00:00Z" },
    { "id": "a", "updatedAt": "2026-07-14T09:00:00Z" }
  ],
  "timestampField": "updatedAt",
  "idField": "id",
  "maxAgeMinutes": 60,
  "minimumRows": 1,
  "autoBaseline": true,
  "failOnIssues": true,
  "monitorKey": "example-orders-feed",
  "now": "2026-07-15T12:00:00Z"
}

Example output

{
  "source": "inline",
  "baselineSource": "stored",
  "automaticBaseline": true,
  "status": "STALE",
  "rowCount": 2,
  "previousRowCount": 10,
  "rowDeltaPercent": -80,
  "newestTimestamp": "2026-07-14T09:00:00.000Z",
  "ageMinutes": 1620,
  "duplicateIdCount": 1,
  "reasons": ["NEWEST_ROW_TOO_OLD", "DUPLICATE_IDS", "ROW_COUNT_DROP"]
}

Local verification

npm install
npm test
APIFY_LOCAL_STORAGE_DIR="$PWD/storage" npm start

The first run for a feed reports baselineSource: "none" and saves its row count. The next run reports baselineSource: "stored" and automatically checks for a 25% drop. A supplied previousRowCount still overrides stored history for one run. Set autoBaseline: false to disable persistence.

Set failOnIssues: true for scheduled monitors. The Actor writes the dataset row and OUTPUT record first, then fails the run when the status is not HEALTHY. This turns Apify's existing task-failure notifications or webhooks into the alert destination. It is disabled by default for backward compatibility.

Pricing

The Actor is currently free because creator payout billing/KYC has not yet been completed. The planned monetized model is $0.01 per completed health-check. A run invokes at most one event, and only after the health result has been saved; on the current free listing Apify safely ignores the charge call. Invalid input, source/network errors, local runs, and failures before output persistence do not invoke a charge event. Normal Apify platform usage may still apply.

Safety and scope

The Actor only accepts inline JSON rows or a public HTTPS JSON endpoint. It rejects local/private-network targets, does not log in, does not use proxies, and does not bypass access controls. Use it only with data you are allowed to monitor.