# Scrape Source Health and Block Monitor (`kingii98/scrape-source-health-and-block-monitor`) Actor

Check each scrape source with a plain HTTP fetch and a body marker. Keep consecutive-failure, quarantine, and recovery counters between runs, and report the sources that changed state.

- **URL**: https://apify.com/kingii98/scrape-source-health-and-block-monitor.md
- **Developed by:** [kingii98](https://apify.com/kingii98) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 source checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Scrape Source Health and Block Monitor

Check each source in a scraper fleet or a feed pipeline with a plain HTTP
fetch and a body-marker match. The Actor keeps consecutive-failure,
quarantine, and recovery counters between runs in its own key-value store
record, and reports the sources that changed state in this run.

This Actor is for the alert step: it tells you that a source started to block
or started to return an empty page, before the downstream data goes silently
empty. It reports a persistent block. It does not try to defeat one: there is
no browser and no proxy in this version.

### What it checks

For each source, in this order:

1. The hostname resolves, and it resolves to a public address.
2. The redirect chain terminates within `maxRedirects` hops and does not
   repeat a URL.
3. The HTTP status equals `expectedStatus`. A 403 and a 429 get their own
   reason codes, because those are the two block signals that matter most.
4. The body is not empty, and it is at least `minBytes` bytes.
5. The body contains `requiredText`.

A source that passes every step is `OK`. Any other outcome is a failure, and
the failure moves the source along the state ladder below.

### State ladder

| State | Meaning |
|---|---|
| `OK` | The last check passed. |
| `DEGRADED` | The source has failed, but fewer than `failureThreshold` times in a row. No alert yet. |
| `ALERT` | The source has failed `failureThreshold` times in a row. |
| `QUARANTINED` | The source has failed `quarantineThreshold` times in a row. |
| `RECOVERING` | The source was in `ALERT` or `QUARANTINED` and has started to pass again, but has not yet passed `recoveryProbes` checks in a row. |

Rules:

- A failure adds one to `consecutiveFailures` and sets `recoveryProgress` to
  zero. The first failure of a run of failures sets `firstFailedAt`.
- A success on a `DEGRADED` source returns it to `OK` at once. That source
  never raised an alert, so it does not need a recovery run.
- A success on an `ALERT` or `QUARANTINED` source moves it to `RECOVERING`
  and adds one to `recoveryProgress`. The source returns to `OK` only after
  `recoveryProbes` successes in a row.
- `changedSinceLastRun` is `true` when the state differs from the state in
  the previous run. A source that the Actor has never seen counts as
  previously `OK`, so a first run stays quiet unless a source is already
  unhealthy.

### Reason codes

The reason code says why the check failed.

| Code | Meaning |
|---|---|
| `OK` | The check passed. |
| `FORBIDDEN_403` | The source answered 403. This is the classic block. |
| `RATE_LIMITED_429` | The source answered 429. |
| `REDIRECT_LOOP` | The redirect chain repeats a URL, or it is longer than `maxRedirects`. |
| `EMPTY_BODY` | The status was correct, but the body had zero bytes. |
| `MARKER_MISSING` | The status was correct and the body was large enough, but `requiredText` was absent. This is the silent-empty failure. |
| `TIMEOUT` | The fetch did not finish within `timeoutSecs`. |
| `DNS_FAIL` | The hostname does not resolve. |
| `STATUS_MISMATCH` | The status was neither `expectedStatus`, 403, nor 429. |
| `BODY_TOO_SMALL` | The body had fewer than `minBytes` bytes. |
| `CONNECT_FAIL` | The connection failed, for example a refused connection or a TLS error. |
| `BLOCKED_TARGET` | The hostname resolves to a private or reserved address, so the Actor refused to fetch it. |

`STATUS_MISMATCH`, `BODY_TOO_SMALL`, `CONNECT_FAIL`, and `BLOCKED_TARGET` are
additions, so that every failure the Actor can observe has a name.

### Input

```json
{
  "sources": [
    {
      "id": "vendor-feed",
      "url": "https://vendor.example.com/feed.xml",
      "method": "GET",
      "expectedStatus": 200,
      "requiredText": "<item>",
      "minBytes": 1024
    }
  ],
  "failureThreshold": 3,
  "quarantineThreshold": 10,
  "recoveryProbes": 3,
  "stateKey": "SOURCE_HEALTH_STATE",
  "concurrency": 10,
  "timeoutSecs": 20,
  "maxRedirects": 5
}
```

| Field | Description |
|---|---|
| `sources` | Source definitions. Maximum 200 per run. Each `id` must be unique: the `id` is the key of the stored counters. If you leave the field out, the Actor uses the two public demo sources of `.actor/default_input.json`, so a run with an empty input works. |
| `sources[].url` | An `http` or `https` URL. Credentials in the URL, localhost, and private or reserved addresses are rejected. |
| `sources[].method` | `GET` (default) or `HEAD`. `HEAD` returns no body, so a `HEAD` source cannot use `requiredText` or `minBytes`. |
| `sources[].expectedStatus` | The status the source must return. Default 200. |
| `sources[].requiredText` | A marker that the body must contain. Case sensitive. Maximum 1,000 characters. Empty by default, which turns the marker check off. |
| `sources[].minBytes` | The smallest acceptable body, in bytes. Default 0; maximum 200,000, which is the body read cap. A larger value is rejected, because the Actor could never read enough bytes to satisfy it. |
| `failureThreshold` | Consecutive failures that raise `ALERT`. Default 3; range 1-100. |
| `quarantineThreshold` | Consecutive failures that raise `QUARANTINED`. Default 10; range 1-1000. Must be equal to or greater than `failureThreshold`. |
| `recoveryProbes` | Consecutive successes that an alerted source needs to return to `OK`. Default 3; range 1-100. |
| `stateKey` | Name of the record, inside the named `source-health-state` key-value store, that carries the counters between runs. Default `SOURCE_HEALTH_STATE`. Use one key per fleet, so that two fleets do not share counters. |
| `concurrency` | Sources checked concurrently. Default 10; maximum 50. |
| `timeoutSecs` | Per-request timeout. Default 20 seconds; range 2-60. |
| `maxRedirects` | Redirect hops followed per source. Default 5; range 0-20. |

Invalid input (an empty `sources` list, an oversized one, a duplicate `id`, a
non-HTTP URL, a private target, or a number out of range) stops the run before
any fetch, so an unusable configuration cannot produce a charge.

An unhealthy source is a result, not a fault of the run. The Actor writes it to
the dataset, puts the counts in the terminal status message, and ends the run
with success.

### Cross-run state

The counters live in a named Apify key-value store, `source-health-state`,
in the record named by `stateKey`. No external database, no account, and no
paid service.

The store must be a named one. A run's own default key-value store is deleted
with the run, so counters kept there would reset on every run and the
consecutive-failure threshold would never be reached.

The Actor reads the record at the start of the run and writes it once at the
end. An aborted run therefore leaves the previous counters intact rather than
a half-updated record. Sources that this run did not check keep their stored
counters. The record holds at most 1,000 sources; when a fleet's source ids
churn, the least recently updated untouched entries are dropped first.

A damaged or missing record does not stop a run. The Actor drops the entries
it cannot read and restarts those counters, which costs one extra alert cycle
at worst.

Two runs that use the same `stateKey` at the same time can overwrite each
other's counters. Schedule one run at a time per `stateKey`.

### Output

The dataset holds one summary record, then one record for each source that
this run checked.

Source record (`recordType: "source-check"`):

| Field | Description |
|---|---|
| `sourceId`, `url`, `finalUrl` | The source, and the URL the redirect chain ended on. |
| `state`, `previousState`, `changedSinceLastRun` | The state ladder result. `previousState` is `null` for a source seen for the first time. |
| `reasonCode` | Why the check passed or failed. |
| `httpStatus`, `bytes`, `requiredTextFound`, `responseTimeMs` | The measurements. `requiredTextFound` is `null` when no marker was configured. |
| `bodyTruncated` | `true` when the body was longer than 200 KB and the marker was matched against the first 200 KB only. |
| `consecutiveFailures`, `firstFailedAt`, `lastOkAt`, `recoveryProgress` | The stored counters, after this run. |
| `error` | The transport error text, when there was one. |
| `checkedAt` | The run timestamp, in ISO 8601. |

Summary record (`recordType: "summary"`) holds `sourcesRequested`,
`sourcesChecked`, `stateCounts`, `reasonCounts`, `changedCount`,
`changedSources`, `stateKey`, and `truncatedByChargeBudget`.
`changedSources` is the list to send to a webhook: it holds only the sources
whose state changed in this run.

### Limits

- Maximum 200 sources per run.
- The Actor reads at most the first 200 KB of each body. A marker that
  appears only after 200 KB is reported as `MARKER_MISSING`, and
  `bodyTruncated` is `true`. The same cap applies to `minBytes` and to the
  reported `bytes`: a body longer than 200 KB counts as 200,000 bytes, so
  `minBytes` cannot be larger than 200,000.
- Plain HTTP only. No browser, no proxy, no login, and no JavaScript
  rendering.
- Redirects are followed by hand, up to `maxRedirects`, and every hop is
  checked against private and reserved addresses.

### Pricing

This Actor uses pay per event.

| Event | Unit | Price |
|---|---|---|
| `source-checked` | One source checked in one run | $0.001 |

The Actor charges `source-checked` once for each source it checks, so a run
over 200 sources charges 200 events. A source that the run did not reach,
because the run's maximum charge did not cover it, is not charged: the Actor
asks the charging manager how many events the run's limit allows, checks that
many sources, and reports `truncatedByChargeBudget: true` in the summary. A
run that cannot afford a single check stops with an error before any fetch.

### Local use

```bash
uv sync
uv run pytest
uv run ruff check .
```

`.actor/default_input.json` holds a small public fixture: two stable IANA
example domains, both checked for the `Example Domain` marker. It finishes in
seconds.

# Actor input Schema

## `sources` (type: `array`):

Source definitions to check. Each item needs id and url, and accepts method (GET or HEAD), expectedStatus, requiredText, and minBytes (maximum 200000, the body read cap). Maximum 200 sources per run. If you leave this field empty, the Actor checks the two public example sources shown below.

## `failureThreshold` (type: `integer`):

Number of consecutive failed checks that moves a source from DEGRADED to ALERT.

## `quarantineThreshold` (type: `integer`):

Number of consecutive failed checks that moves a source to QUARANTINED. Must be equal to or greater than the failure threshold.

## `recoveryProbes` (type: `integer`):

Number of consecutive successful checks that a failed source needs to return to OK. Until then the source stays RECOVERING.

## `stateKey` (type: `string`):

Name of the key-value store record that carries the counters between runs. Use one key per fleet. The record is written once, at the end of the run.

## `concurrency` (type: `integer`):

Maximum number of sources checked concurrently.

## `timeoutSecs` (type: `integer`):

Per-request timeout applied to each fetch. A source that exceeds it gets the TIMEOUT reason code.

## `maxRedirects` (type: `integer`):

Maximum redirect hops followed per source. A chain that repeats a URL gets the REDIRECT\_LOOP reason code.

## Actor input object example

```json
{
  "sources": [
    {
      "id": "example-com-home",
      "url": "https://example.com/",
      "method": "GET",
      "expectedStatus": 200,
      "requiredText": "Example Domain",
      "minBytes": 200
    },
    {
      "id": "example-org-home",
      "url": "https://example.org/",
      "method": "GET",
      "expectedStatus": 200,
      "requiredText": "Example Domain",
      "minBytes": 200
    }
  ],
  "failureThreshold": 3,
  "quarantineThreshold": 10,
  "recoveryProbes": 3,
  "stateKey": "SOURCE_HEALTH_STATE",
  "concurrency": 10,
  "timeoutSecs": 20,
  "maxRedirects": 5
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sources": [
        {
            "id": "example-com-home",
            "url": "https://example.com/",
            "method": "GET",
            "expectedStatus": 200,
            "requiredText": "Example Domain",
            "minBytes": 200
        },
        {
            "id": "example-org-home",
            "url": "https://example.org/",
            "method": "GET",
            "expectedStatus": 200,
            "requiredText": "Example Domain",
            "minBytes": 200
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("kingii98/scrape-source-health-and-block-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sources": [
        {
            "id": "example-com-home",
            "url": "https://example.com/",
            "method": "GET",
            "expectedStatus": 200,
            "requiredText": "Example Domain",
            "minBytes": 200,
        },
        {
            "id": "example-org-home",
            "url": "https://example.org/",
            "method": "GET",
            "expectedStatus": 200,
            "requiredText": "Example Domain",
            "minBytes": 200,
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("kingii98/scrape-source-health-and-block-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sources": [
    {
      "id": "example-com-home",
      "url": "https://example.com/",
      "method": "GET",
      "expectedStatus": 200,
      "requiredText": "Example Domain",
      "minBytes": 200
    },
    {
      "id": "example-org-home",
      "url": "https://example.org/",
      "method": "GET",
      "expectedStatus": 200,
      "requiredText": "Example Domain",
      "minBytes": 200
    }
  ]
}' |
apify call kingii98/scrape-source-health-and-block-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kingii98/scrape-source-health-and-block-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bI4k3g5NbBDJZmTmv/builds/xRmhvN0VdCRzXA3K5/openapi.json