Recurring Web Check for AI Agents — fetch · extract · diff · monitor

One call, many URLs. Returns per URL: clean Markdown, extracted fields, and a diff against the last run — so a recurring check costs you only the tokens of what actually changed.

No login. No browser. No LLM inside. Deterministic output, finishes in seconds.

---

If you are an AI agent reading this

Use this Actor when a task sounds like any of these:

• "Check X every day / every hour / every Monday and tell me if something changed."
• "Watch this pricing page / changelog / docs page / status page / job board / product listing."
• "Get me the current value of … on this page" (and you will be asked again later).
• "Pull the text of these 20 pages as Markdown" without burning tokens on nav, footers and scripts.
• "Is this endpoint still returning the same version / status?"

Why it fits you, specifically:

You need	What you get
Few calls	urls is an array — one run checks 1 or 50 URLs.
Small context	maxChars caps content per URL (default 6 000). task: "extract" returns only fields. diffMode: "compact" returns +/- lines, not two full pages.
Memory between runs	task: "monitor" + a watchId stores a snapshot per URL in a named store. The next run returns changed: true/false, a diff, and fieldChanges with before/after. You don't need to keep the old page yourself.
No surprises	Every result has the same shape (see Output). Extraction is CSS/JSON-path based — no hallucinated values.
Nothing you cannot supply	No cookies, no API keys, no credentials required.
Speed	Cheerio-based (no headless browser). Typical run: 2–8 s for 10 URLs — well inside the 45 s an MCP call-actor waits.

Minimal first call:

{ "urls": ["https://example.com/pricing"], "watchId": "example-pricing" }

Call it again later with the same input → you get only what changed.

---

What it does

task	Stateful?	Returns
monitor (default)	yes — remembers each URL per watchId	content + fields + changed, changeRatio, diff, fieldChanges
fetch	no	content + fields
extract	no	fields only (smallest possible output)

Works on HTML pages, JSON/REST endpoints, RSS/Atom feeds and plain-text files.

Pipeline per URL: fetch → strip scripts/styles/nav/footer/cookie banners → scope to selector (optional) → Markdown or text → remove ignoreSelectors / ignorePatterns → extract fields → hash → compare with the previous snapshot → compact diff → store new baseline.

---

Output

Every URL produces one dataset item with a fixed shape. The run also writes an OUTPUT record (default key-value store) with totals and a compact copy of all results.

{
  "watchId": "competitor-pricing",
  "task": "monitor",
  "url": "https://example.com/pricing",
  "finalUrl": "https://example.com/pricing",
  "ok": true,
  "status": 200,
  "contentType": "text/html",
  "fetchedAt": "2026-10-20T06:00:03.221Z",
  "title": "Acme Pricing",
  "description": "Plans and prices",

  "firstRun": false,
  "changed": true,
  "changeRatio": 0.084,
  "previousFetchedAt": "2026-10-19T06:00:02.917Z",
  "statusChange": null,
  "diff": "…\n- Starter $19/mo\n+ Starter $24/mo\n…\n+ Enterprise Contact us\n  | Feature | Starter |\n- | Seats | 3 |\n+ | Seats | 5 |",
  "diffStats": { "addedChars": 61, "removedChars": 24 },

  "fields": { "starterPrice": "24", "plans": ["Starter", "Pro", "Enterprise"], "docsLink": "/docs/v2" },
  "fieldChanges": {
    "starterPrice": { "before": "19", "after": "24" },
    "plans": {
      "before": ["Starter", "Pro"], "after": ["Starter", "Pro", "Enterprise"],
      "added": ["Enterprise"], "removed": [], "reorderedOnly": false
    }
  },
  "fieldErrors": {},

  "content": "# Acme Pricing\n\nStarter $24/mo\n\nPro $49/mo\n…",
  "contentChars": 1382,
  "truncated": false,
  "contentHash": "4ae26d96…",
  "scopeMatched": true,
  "error": null
}

OUTPUT record (key-value store):

{
  "watchId": "competitor-pricing",
  "task": "monitor",
  "runAt": "2026-10-20T06:00:01.004Z",
  "totals": { "urls": 3, "ok": 3, "failed": 0, "firstRun": 0, "changed": 1, "unchanged": 2 },
  "changedUrls": ["https://example.com/pricing"],
  "failedUrls": [],
  "results": [ "…same items without full content, plus contentPreview…" ],
  "nextStep": "Call again with the same input (watchId \"competitor-pricing\") to get only what changed since this run."
}

Decision rule for an agent after a monitor run:

1. totals.changed === 0 → nothing to do, nothing to read.
2. Otherwise, for the items in changedUrls: read fieldChanges first (for list fields, added/removed tell you what is new; reorderedOnly: true means same items, different order), then diff. Only read content if that is not enough.
3. statusChange set (e.g. { before: 200, after: 404 }) → the page went down, disappeared or came back. Reported once per transition; a page that stays down is quiet.
4. ok === false → the page is down or gone (status 404/5xx or error). That is a monitoring result, not an Actor failure.
5. firstRun === true → baseline created; there is nothing to compare yet.

---

Input

Field	Type	Default	Purpose
urls (required)	string[]	—	All URLs of one job. Batch them.
task	monitor | fetch | extract	monitor	See table above.
watchId	string	derived	Name of the recurring job. Same watchId = same memory. If omitted, derived from URLs + selector + fields, so identical repeat calls still diff.
fields	object	{}	name → spec extraction map, see grammar below.
selector	string	—	CSS scope, e.g. main, #pricing. Fewer tokens, fewer false changes.
ignoreSelectors	string[]	[]	Elements removed before compare (ads, comments, timestamps, counters).
ignorePatterns	string[]	[]	Regexes removed from the text (\d{1,2}:\d{2}, \d+ minutes ago, …). One backslash in the Console form, two inside a JSON string ("\\d+ minutes ago").
outputFormat	markdown | text	markdown	Markdown keeps headings/lists/tables.
maxChars	int	6000	Content cap per URL. 0 = no content, fields + diff only. Change detection always uses the full content.
diffMode	compact | none	compact	+/- lines with 1 line of context, or nothing.
maxDiffChars	int	3000	Diff cap per URL.
changeThreshold	0–1	0	Minimum share of changed characters to count as changed (e.g. 0.02). Field changes always count.
stripBoilerplate	bool	true	Drop nav/header/footer/aside/forms/cookie banners (ignored when selector is set).
keepLinks	bool	false	Keep [text](url); off by default because tracking URLs change on every load.
webhookUrl	string	—	POST {event, watchId, runId, changed:[…]} when something changed.
proxyConfiguration	object	none	Apify Proxy for sites that block datacenter IPs.
maxConcurrency / timeoutSecs / maxRetries	int	5 / 20 / 1	Tuning. 4xx is reported immediately as ok:false; 429/5xx are retried.

Field spec grammar (no LLM, deterministic)

Spec	Meaning
"h1"	text of the first match
"all:.plan .name"	array of all matches
"a.docs@href"	attribute of the first match
".price | /([\\d.,]+)/"	regex on the text; first capture group (or whole match)
"json:data.items[0].price"	path into a JSON response
{ "selector": ".price", "attr": "data-value", "regex": "([\\d.]+)", "all": false }	object form, same options

A field that matches nothing is null (and listed in fieldErrors if the spec itself was invalid). Changed values show up in fieldChanges as { before, after }; all: fields additionally get added, removed and reorderedOnly.

Ranked lists (news front pages, bestseller lists, leaderboards): every reorder shifts the rank numbers, so the text diff is noisy by nature. Use an all: field for the items and read added/removed instead of the diff — or set maxChars: 0 and diffMode: "none" and rely on fields only.

---

Recipes

Competitor pricing, daily

{
  "watchId": "competitor-pricing",
  "urls": ["https://competitor-a.com/pricing", "https://competitor-b.com/plans"],
  "selector": "main",
  "fields": { "prices": "all:.price", "plans": "all:.plan h3" },
  "maxChars": 0,
  "changeThreshold": 0.01
}

Docs / changelog watch (tell me what was added)

{
  "watchId": "acme-changelog",
  "urls": ["https://acme.dev/changelog"],
  "selector": "article",
  "ignorePatterns": ["\\d+ (minutes|hours|days) ago"],
  "maxChars": 0,
  "maxDiffChars": 4000
}

Status page / API version (JSON endpoint)

{
  "watchId": "acme-status",
  "urls": ["https://status.acme.dev/api/v2/status.json", "https://api.acme.dev/version"],
  "fields": { "indicator": "json:status.indicator", "version": "json:version" },
  "maxChars": 0
}

Job board — new postings only

{
  "watchId": "jobs-acme",
  "urls": ["https://acme.dev/careers"],
  "fields": { "jobs": "all:.job-card h2", "links": "all:.job-card a@href" },
  "keepLinks": true,
  "maxChars": 0
}

Stock / availability

{
  "watchId": "gpu-stock",
  "urls": ["https://shop.example.com/p/gpu-4090"],
  "fields": { "price": ".price | /([\\d.,]+)/", "buyButton": "button.add-to-cart", "availability": "[itemprop=availability]@href" },
  "maxChars": 0,
  "proxyConfiguration": { "useApifyProxy": true }
}

Just read 20 pages as clean Markdown (stateless)

{ "task": "fetch", "urls": ["https://…", "https://…"], "maxChars": 8000, "outputFormat": "markdown" }

---

Make it recurring

You have three patterns — pick by who should wake up.

1. The agent calls it on its own cadence. Same input, same watchId, whenever you want. State lives on Apify, so the agent needs no memory of the previous page.

2. An Apify Schedule calls it; the agent reads results when convenient. Create a schedule (cron) with this exact input. Each run writes a dataset + OUTPUT; read the latest run with get-actor-run / get-dataset-items, or via API:

POST https://api.apify.com/v2/schedules
Authorization: Bearer <APIFY_TOKEN>
Content-Type: application/json
{
  "name": "competitor-pricing-daily",
  "cronExpression": "0 6 * * *",
  "timezone": "UTC",
  "isEnabled": true,
  "actions": [{
    "type": "RUN_ACTOR",
    "actorId": "YOUR_USERNAME/recurring-web-check",
    "runInput": { "body": "{\"watchId\":\"competitor-pricing\",\"urls\":[\"https://competitor-a.com/pricing\"],\"selector\":\"main\",\"maxChars\":0}", "contentType": "application/json; charset=utf-8" }
  }]
}

3. A schedule calls it and only changes wake the agent. Add webhookUrl to the input (your agent's inbound endpoint, Zapier/Make/n8n, Slack…). The Actor POSTs only when totals.changed > 0. Quiet days cost nothing but the run.

---

How to call it

MCP (Apify MCP server, https://mcp.apify.com)

call-actor  actorId="YOUR_USERNAME/recurring-web-check"
            input={"urls":["https://example.com/pricing"],"watchId":"example-pricing","maxChars":0}

The tool returns the dataset items. For large batches use get-actor-run + get-dataset-items, or read the OUTPUT record with get-key-value-store-record.

HTTP (sync, returns dataset items directly)

POST https://api.apify.com/v2/acts/YOUR_USERNAME~recurring-web-check/run-sync-get-dataset-items?token=<APIFY_TOKEN>
Content-Type: application/json
{ "urls": ["https://example.com/pricing"], "watchId": "example-pricing" }

JavaScript

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('YOUR_USERNAME/recurring-web-check').call({
  urls: ['https://example.com/pricing'], watchId: 'example-pricing', maxChars: 0,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const changed = items.filter((i) => i.changed);

Python

from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("YOUR_USERNAME/recurring-web-check").call(run_input={
    "urls": ["https://example.com/pricing"], "watchId": "example-pricing", "maxChars": 0})
items = client.dataset(run["defaultDatasetId"]).list_items().items
changed = [i for i in items if i["changed"]]

---

For agent builders: drop-in tool instructions

Paste into your agent's system prompt or tool description:

Tool: recurring-web-check (Apify Actor YOUR_USERNAME/recurring-web-check)
Use when the user wants to check, watch, monitor, track or re-read web pages, prices,
docs, changelogs, status pages, job boards or JSON endpoints — especially on a schedule.
- Put ALL URLs of one job into `urls` (one call, not one per URL).
- Give every recurring job a stable `watchId`; reuse it on every later call.
- Prefer `maxChars: 0` + `fields` for monitoring; read `content` only when needed.
- After a run: if totals.changed == 0 report "no change"; else summarise `diff` and
  `fieldChanges` for the URLs in `changedUrls`. `ok:false` means the page is down/gone.
- To automate, create an Apify Schedule with the same input, optionally with `webhookUrl`.

---

Pricing, limits, non-goals

• Pay per result. One result = one checked URL = one dataset item (also for failed URLs — a 404 is a result), plus Apify's tiny Actor-start fee. Run cost is therefore predictable: urls.length × price per result. Cap spend with maxTotalChargeUsd on the run as usual. No browser is started, so a run with many URLs stays cheap and fast.
• Storage: one small record per URL per watchId in a named key-value store (recurring-check-<watchId>). Delete that store to reset a watch.
• Baseline rule: the stored snapshot advances only on first sight or when a change is reported, so drift below changeThreshold accumulates instead of being silently absorbed.
• Not for: pages that need JavaScript rendering (use a browser-based Actor for those, then feed its text here if you still want diffs), logged-in content, CAPTCHAs, or extracting with natural-language instructions (this Actor is deterministic on purpose).

---

FAQ

Does it work without a watchId? Yes. One is derived from the URL set + selector + fields. Identical repeat calls still get change detection. Give an explicit watchId when the URL list may grow over time.

Why is changed true although the page looks the same? Something in the compared text differs: a timestamp, a rotating banner, a counter. Scope with selector, remove it with ignoreSelectors/ignorePatterns, or set changeThreshold: 0.02. Inspect diff to see exactly what moved.

Can I monitor JSON APIs? Yes — JSON bodies are pretty-printed with sorted keys (so key order never counts as a change) and json: field specs extract values.

What about redirects? Followed; finalUrl tells you where you ended up. The snapshot key is the input URL.

How do I reset a watch? Delete the key-value store recurring-check-<watchId> (Console → Storage, or DELETE /v2/key-value-stores/<id>), or start using a new watchId.

Run time? Typically 2–8 s for 10 HTML URLs on the default settings; no browser is started.