Scrape Source Health and Block Monitor

Check each source in a scraper fleet or a feed pipeline with a plain HTTP fetch and a body-marker match. The Actor keeps consecutive-failure, quarantine, and recovery counters between runs in its own key-value store record, and reports the sources that changed state in this run.

This Actor is for the alert step: it tells you that a source started to block or started to return an empty page, before the downstream data goes silently empty. It reports a persistent block. It does not try to defeat one: there is no browser and no proxy in this version.

What it checks

For each source, in this order:

1. The hostname resolves, and it resolves to a public address.
2. The redirect chain terminates within maxRedirects hops and does not repeat a URL.
3. The HTTP status equals expectedStatus. A 403 and a 429 get their own reason codes, because those are the two block signals that matter most.
4. The body is not empty, and it is at least minBytes bytes.
5. The body contains requiredText.

A source that passes every step is OK. Any other outcome is a failure, and the failure moves the source along the state ladder below.

State ladder

State	Meaning
OK	The last check passed.
DEGRADED	The source has failed, but fewer than failureThreshold times in a row. No alert yet.
ALERT	The source has failed failureThreshold times in a row.
QUARANTINED	The source has failed quarantineThreshold times in a row.
RECOVERING	The source was in ALERT or QUARANTINED and has started to pass again, but has not yet passed recoveryProbes checks in a row.

Rules:

• A failure adds one to consecutiveFailures and sets recoveryProgress to zero. The first failure of a run of failures sets firstFailedAt.
• A success on a DEGRADED source returns it to OK at once. That source never raised an alert, so it does not need a recovery run.
• A success on an ALERT or QUARANTINED source moves it to RECOVERING and adds one to recoveryProgress. The source returns to OK only after recoveryProbes successes in a row.
• changedSinceLastRun is true when the state differs from the state in the previous run. A source that the Actor has never seen counts as previously OK, so a first run stays quiet unless a source is already unhealthy.

Reason codes

The reason code says why the check failed.

Code	Meaning
OK	The check passed.
FORBIDDEN_403	The source answered 403. This is the classic block.
RATE_LIMITED_429	The source answered 429.
REDIRECT_LOOP	The redirect chain repeats a URL, or it is longer than maxRedirects.
EMPTY_BODY	The status was correct, but the body had zero bytes.
MARKER_MISSING	The status was correct and the body was large enough, but requiredText was absent. This is the silent-empty failure.
TIMEOUT	The fetch did not finish within timeoutSecs.
DNS_FAIL	The hostname does not resolve.
STATUS_MISMATCH	The status was neither expectedStatus, 403, nor 429.
BODY_TOO_SMALL	The body had fewer than minBytes bytes.
CONNECT_FAIL	The connection failed, for example a refused connection or a TLS error.
BLOCKED_TARGET	The hostname resolves to a private or reserved address, so the Actor refused to fetch it.

STATUS_MISMATCH, BODY_TOO_SMALL, CONNECT_FAIL, and BLOCKED_TARGET are additions, so that every failure the Actor can observe has a name.

Input

{
  "sources": [
    {
      "id": "vendor-feed",
      "url": "https://vendor.example.com/feed.xml",
      "method": "GET",
      "expectedStatus": 200,
      "requiredText": "<item>",
      "minBytes": 1024
    }
  ],
  "failureThreshold": 3,
  "quarantineThreshold": 10,
  "recoveryProbes": 3,
  "stateKey": "SOURCE_HEALTH_STATE",
  "concurrency": 10,
  "timeoutSecs": 20,
  "maxRedirects": 5
}

Field	Description
sources	Source definitions. Maximum 200 per run. Each id must be unique: the id is the key of the stored counters. If you leave the field out, the Actor uses the two public demo sources of .actor/default_input.json, so a run with an empty input works.
sources[].url	An http or https URL. Credentials in the URL, localhost, and private or reserved addresses are rejected.
sources[].method	GET (default) or HEAD. HEAD returns no body, so a HEAD source cannot use requiredText or minBytes.
sources[].expectedStatus	The status the source must return. Default 200.
sources[].requiredText	A marker that the body must contain. Case sensitive. Maximum 1,000 characters. Empty by default, which turns the marker check off.
sources[].minBytes	The smallest acceptable body, in bytes. Default 0; maximum 200,000, which is the body read cap. A larger value is rejected, because the Actor could never read enough bytes to satisfy it.
failureThreshold	Consecutive failures that raise ALERT. Default 3; range 1-100.
quarantineThreshold	Consecutive failures that raise QUARANTINED. Default 10; range 1-1000. Must be equal to or greater than failureThreshold.
recoveryProbes	Consecutive successes that an alerted source needs to return to OK. Default 3; range 1-100.
stateKey	Name of the record, inside the named source-health-state key-value store, that carries the counters between runs. Default SOURCE_HEALTH_STATE. Use one key per fleet, so that two fleets do not share counters.
concurrency	Sources checked concurrently. Default 10; maximum 50.
timeoutSecs	Per-request timeout. Default 20 seconds; range 2-60.
maxRedirects	Redirect hops followed per source. Default 5; range 0-20.

Invalid input (an empty sources list, an oversized one, a duplicate id, a non-HTTP URL, a private target, or a number out of range) stops the run before any fetch, so an unusable configuration cannot produce a charge.

An unhealthy source is a result, not a fault of the run. The Actor writes it to the dataset, puts the counts in the terminal status message, and ends the run with success.

Cross-run state

The counters live in a named Apify key-value store, source-health-state, in the record named by stateKey. No external database, no account, and no paid service.

The store must be a named one. A run's own default key-value store is deleted with the run, so counters kept there would reset on every run and the consecutive-failure threshold would never be reached.

The Actor reads the record at the start of the run and writes it once at the end. An aborted run therefore leaves the previous counters intact rather than a half-updated record. Sources that this run did not check keep their stored counters. The record holds at most 1,000 sources; when a fleet's source ids churn, the least recently updated untouched entries are dropped first.

A damaged or missing record does not stop a run. The Actor drops the entries it cannot read and restarts those counters, which costs one extra alert cycle at worst.

Two runs that use the same stateKey at the same time can overwrite each other's counters. Schedule one run at a time per stateKey.

Output

The dataset holds one summary record, then one record for each source that this run checked.

Source record (recordType: "source-check"):

Field	Description
sourceId, url, finalUrl	The source, and the URL the redirect chain ended on.
state, previousState, changedSinceLastRun	The state ladder result. previousState is null for a source seen for the first time.
reasonCode	Why the check passed or failed.
httpStatus, bytes, requiredTextFound, responseTimeMs	The measurements. requiredTextFound is null when no marker was configured.
bodyTruncated	true when the body was longer than 200 KB and the marker was matched against the first 200 KB only.
consecutiveFailures, firstFailedAt, lastOkAt, recoveryProgress	The stored counters, after this run.
error	The transport error text, when there was one.
checkedAt	The run timestamp, in ISO 8601.

Summary record (recordType: "summary") holds sourcesRequested, sourcesChecked, stateCounts, reasonCounts, changedCount, changedSources, stateKey, and truncatedByChargeBudget. changedSources is the list to send to a webhook: it holds only the sources whose state changed in this run.

Limits

• Maximum 200 sources per run.
• The Actor reads at most the first 200 KB of each body. A marker that appears only after 200 KB is reported as MARKER_MISSING, and bodyTruncated is true. The same cap applies to minBytes and to the reported bytes: a body longer than 200 KB counts as 200,000 bytes, so minBytes cannot be larger than 200,000.
• Plain HTTP only. No browser, no proxy, no login, and no JavaScript rendering.
• Redirects are followed by hand, up to maxRedirects, and every hop is checked against private and reserved addresses.

Pricing

This Actor uses pay per event.

Event	Unit	Price
source-checked	One source checked in one run	$0.001

The Actor charges source-checked once for each source it checks, so a run over 200 sources charges 200 events. A source that the run did not reach, because the run's maximum charge did not cover it, is not charged: the Actor asks the charging manager how many events the run's limit allows, checks that many sources, and reports truncatedByChargeBudget: true in the summary. A run that cannot afford a single check stops with an error before any fetch.

Local use

uv sync
uv run pytest
uv run ruff check .

.actor/default_input.json holds a small public fixture: two stable IANA example domains, both checked for the Example Domain marker. It finishes in seconds.