# Scraper Freshness Watchdog — Dataset Health Monitor (`convenient_yarn/scraper-freshness-watchdog`) Actor

Detect stale, empty, duplicated, invalid, or sharply reduced scraper output from inline rows or a public JSON feed. Save one deterministic health record and optionally fail unhealthy scheduled runs for native alerts.

- **URL**: https://apify.com/convenient\_yarn/scraper-freshness-watchdog.md
- **Developed by:** [Travis Berman](https://apify.com/convenient_yarn) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Scraper Freshness Watchdog

A lightweight Apify Actor for teams whose scheduled scrapers silently return stale, empty, duplicated, or sharply reduced datasets.

It accepts either inline rows or a permission-respecting public HTTPS JSON endpoint and emits one deterministic health record. It does not log in, bypass challenges, use proxies, or scrape private data.

[Follow the 60-second setup tutorial and inspect a deterministic failure example](https://github.com/kaijanokovsky/scraper-freshness-watchdog?utm_source=apify-store\&utm_medium=readme\&utm_campaign=freshness-watchdog-launch-2026).

### Checks

- newest timestamp age against `maxAgeMinutes`
- minimum row count
- duplicate IDs
- row-count regression of 25% or more against either `previousRowCount` or the automatically stored prior run
- automatic per-feed snapshot persistence in a named key-value store; public source URLs identify the feed automatically, while inline data can use `monitorKey`
- missing or invalid timestamps
- optional `failOnIssues` mode that saves the full health result, then marks an unhealthy run failed so existing Apify task failure notifications or webhooks can alert the owner without this Actor handling notification credentials

### Example input

```json
{
  "rows": [
    { "id": "a", "updatedAt": "2026-07-14T08:00:00Z" },
    { "id": "a", "updatedAt": "2026-07-14T09:00:00Z" }
  ],
  "timestampField": "updatedAt",
  "idField": "id",
  "maxAgeMinutes": 60,
  "minimumRows": 1,
  "autoBaseline": true,
  "failOnIssues": true,
  "monitorKey": "example-orders-feed",
  "now": "2026-07-15T12:00:00Z"
}
```

### Example output

```json
{
  "source": "inline",
  "baselineSource": "stored",
  "automaticBaseline": true,
  "status": "STALE",
  "rowCount": 2,
  "previousRowCount": 10,
  "rowDeltaPercent": -80,
  "newestTimestamp": "2026-07-14T09:00:00.000Z",
  "ageMinutes": 1620,
  "duplicateIdCount": 1,
  "reasons": ["NEWEST_ROW_TOO_OLD", "DUPLICATE_IDS", "ROW_COUNT_DROP"]
}
```

### Local verification

```bash
npm install
npm test
APIFY_LOCAL_STORAGE_DIR="$PWD/storage" npm start
```

The first run for a feed reports `baselineSource: "none"` and saves its row count. The next run reports `baselineSource: "stored"` and automatically checks for a 25% drop. A supplied `previousRowCount` still overrides stored history for one run. Set `autoBaseline: false` to disable persistence.

Set `failOnIssues: true` for scheduled monitors. The Actor writes the dataset row and `OUTPUT` record first, then fails the run when the status is not `HEALTHY`. This turns Apify's existing task-failure notifications or webhooks into the alert destination. It is disabled by default for backward compatibility.

### Pricing

The Actor is currently **free** because creator payout billing/KYC has not yet been completed. The planned monetized model is **$0.01 per completed `health-check`**. A run invokes at most one event, and only after the health result has been saved; on the current free listing Apify safely ignores the charge call. Invalid input, source/network errors, local runs, and failures before output persistence do not invoke a charge event. Normal Apify platform usage may still apply.

### Safety and scope

The Actor only accepts inline JSON rows or a public HTTPS JSON endpoint. It rejects local/private-network targets, does not log in, does not use proxies, and does not bypass access controls. Use it only with data you are allowed to monitor.

# Actor input Schema

## `sourceUrl` (type: `string`):

HTTPS URL returning public JSON. Local and private-network hosts are rejected.

## `rows` (type: `array`):

Use inline rows instead of sourceUrl.

## `rowsPath` (type: `string`):

Dot path to the array inside the JSON response, for example data.items.

## `timestampField` (type: `string`):

Dot path to each row's ISO timestamp.

## `idField` (type: `string`):

Optional dot path used to count duplicate IDs.

## `maxAgeMinutes` (type: `integer`):

Flag the dataset when its newest valid timestamp is older than this many minutes.

## `minimumRows` (type: `integer`):

Fail the check when the selected JSON array contains fewer rows than this value.

## `previousRowCount` (type: `integer`):

Optional manual baseline. When supplied it overrides stored history for this run.

## `autoBaseline` (type: `boolean`):

Stores this run's row count in a named key-value store and compares it on the next run.

## `failOnIssues` (type: `boolean`):

Marks the Actor run failed after saving its output when stale, missing, duplicate, invalid, or sharply reduced data is detected. Use Apify task failure notifications or webhooks as the alert destination.

## `monitorKey` (type: `string`):

Stable identity for inline data. Public source URLs are used automatically when this is omitted.

## Actor input object example

```json
{
  "timestampField": "updatedAt",
  "maxAgeMinutes": 1440,
  "minimumRows": 1,
  "autoBaseline": true,
  "failOnIssues": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `keyValueStore` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("convenient_yarn/scraper-freshness-watchdog").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("convenient_yarn/scraper-freshness-watchdog").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call convenient_yarn/scraper-freshness-watchdog --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=convenient_yarn/scraper-freshness-watchdog",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LX2LbCm7Gbp1MszgB/builds/LZZZcVRVyyJ3wfHBJ/openapi.json