Page Change Monitor

Page Change Monitor — Steadfeed

Version 0.2.0 — watch public pages for text changes, and get a plain-words summary of what actually changed.

Fetch a list of public web pages (pricing, terms, policies, release notes), extract normalized visible text (optionally limited to a CSS selector), compare each page to the previous snapshot stored in a named key-value store, and emit one dataset row per URL with status new, unchanged, changed, or error, a content hash, timestamps, and a capped line diff. New in 0.2: every changed row also carries a summary that says, in plain words, what was added, removed or reworded, which prices, dates and numbers moved (before → after), and which lines were routine noise (timestamps, ads, cookie/footer text) and were ignored. It is deterministic and free: no AI service is involved unless you choose to add your own key.

Designed for scheduled runs: keep the same watchlistId and stateStoreName so baselines persist between Actor runs.

This Actor reports text differences it can see in the fetched HTML. It does not guarantee that every visual or behind-login change is detected, and it does not bypass logins, paywalls, captchas or bot challenges. Respect robots.txt is on by default. A changed status is not legal, compliance or competitive advice.

Meaningful-change summaries (new in 0.2)

Each changed row has a summary object, for example:

{
  "mode": "heuristic",
  "meaningful": true,
  "headline": "Meaningful change: 1 price change, 1 date change, 1 new section.",
  "text": "Price changed: $29 → $39 in “Pro plan $39 per month” Date changed: Nov 20 → Nov 30 in “Sale ends Nov 30” New section added “Enterprise” (3 lines): …",
  "counts": { "priceChanges": 1, "dateChanges": 1, "sectionsAdded": 1, "noiseIgnored": 4 },
  "changes": [ { "type": "price", "from": "$29", "to": "$39", "text": "…" } ]
}

• What it detects: price changes (currency amounts), date changes, number and percentage changes, reworded passages (word-level before → after), single lines added or removed, and whole sections added or removed.
• What it ignores as noise: lines labelled "last updated / modified / generated / as of …" with a date, bare dates and clock times, relative times ("3 hours ago"), copyright years, ad and "sponsored" labels, cookie/consent and sign-in/footer boilerplate, view/visitor counters, and long random tokens. summary.counts.noiseIgnored and summary.noiseReasons show what was dropped. A page whose only changes are noise gets summary.meaningful: false.
• Quiet mode: set skipNoiseOnlyChanges to true and noise-only changes are reported as unchanged (the snapshot is still refreshed), so schedules and alerts fire only on real changes. Off by default, so status behavior matches 0.1.x.
• Heuristics, not magic: the summary works on the extracted text, line by line. It can mislabel an edit as a removal plus an addition, and cannot judge intent or legal significance. The raw diff stays on every row.
• Turn it off: summarizeChanges: false restores 0.1.x rows (summary is null).

Optional: AI summary with your own key

Everything above works with no AI. If you want a short AI-written paragraph on top, add aiApiKey (and optionally aiBaseUrl, aiModel) for any OpenAI-compatible chat-completions API. You pay your AI provider directly; Steadfeed supplies no key and charges nothing extra. Only the already-found change list (not the full page) is sent, only for meaningful changes, at most maxAiSummaries times per run (default 25). The key is a secret input, is never logged or written to output, and if the AI call fails the row still carries the free summary (summary.ai.error explains why). Default model is gpt-6-luna; set aiModel for any other.

Input example

{
  "pages": [
    { "url": "https://example.com/", "label": "Example homepage" },
    {
      "url": "https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document",
      "label": "W3C software notice"
    }
  ],
  "ignorePatterns": ["Last updated"],
  "watchlistId": "default",
  "stateStoreName": "steadfeed-page-change-monitor-state",
  "emitOnFirstRun": true,
  "emitUnchanged": true,
  "respectRobotsTxt": true,
  "requestIntervalMs": 1000,
  "maxDiffLines": 40
}

Fields

Field	Default	Notes
pages	(required)	1–50 entries. Each is a URL string or { url, selector?, label? }.
ignorePatterns	[]	Case-insensitive substrings, or /regex/ / /regex/i, stripped before hash/diff.
watchlistId	default	Separates state when several schedules share one store.
stateStoreName	steadfeed-page-change-monitor-state	Named Apify key-value store for snapshots.
emitOnFirstRun	true	Emit new rows on the first successful snapshot.
emitUnchanged	true	Emit rows when the hash matches.
maxDiffLines	40	Cap on listed added+removed lines (counts are full).
requestIntervalMs	1000	Minimum delay between requests to the same host.
maxRetries	2	Retries for network errors and 408/429/5xx.
timeoutMs	20000	Per-request timeout.
maxBytes	2000000	Response body size cap before parse.
respectRobotsTxt	true	Skip URLs disallowed for this Actor's user-agent.
maxPages	50	Hard cap on pages.length.
summarizeChanges	true	Add the plain-words summary to changed rows.
skipNoiseOnlyChanges	false	Report noise-only changes (timestamps, ads, counters, footer text) as unchanged.
maxSummaryChanges	25	Max items in summary.changes (1–100).
aiApiKey	empty	Optional, secret. Your own OpenAI-compatible key for an extra AI summary.
aiBaseUrl	https://api.openai.com/v1	https only (http allowed for localhost).
aiModel	gpt-6-luna	Model id your provider accepts.
maxAiSummaries	25	Max AI calls per run (1–200).

Invalid input fails before network requests: unknown keys, bad URLs, duplicate url+selector pairs, and out-of-range numbers.

How it works

1. Optionally fetch /robots.txt once per origin and evaluate Allow/Disallow for Steadfeed-Page-Change-Monitor (RFC 9309 longest-match). Missing robots (4xx) is treated as allow-all; unreachable or 5xx robots skips the page.
2. Fetch each page with polite per-host spacing, timeouts, bounded redirects and retries. No authentication, cookies, proxies or captcha bypass.
3. Parse HTML, drop script/style/etc., extract visible text (whole main/article/body, or a CSS selector region via Cheerio).
4. Apply ignore patterns, normalize whitespace, and compute a SHA-256 content hash.
5. Load the previous snapshot from the named store (watchlistId + url/selector key). Compare hashes; on change, produce a capped line diff and the plain-words summary.
6. Save the new snapshot and push a dataset row.

Light heuristics may set accessGate when the response looks like a login form or paywall copy, or when HTTP 401/402/403 is returned. The Actor still does not attempt to log in.

Output

Default dataset: one recordType: "page" row per checked URL (subject to emit flags). Fields include status, url/finalUrl, label, selector, httpStatus, robots decision, title, contentHash, previous hash, textLength, timestamps, optional diff (addedLines, removedLines, counts, truncated), optional summary (see above), and error details.

SUMMARY in the default key-value store holds run totals. Snapshots live in the named state store, not the run's default store, so schedules can reuse them.

Formal schemas: .actor/record_schema.json, .actor/dataset_schema.json, .actor/input_schema.json.

Limits and responsible use

• Public HTTP(S) HTML/text only. No login, paywall bypass, cookie consent clicking, or browser rendering of heavily JavaScript-built pages (content not present in the initial HTML will be missed).
• CSS selectors must match server-rendered markup. A missing selector yields selector_no_match.
• Diffs are line-oriented on normalized text, not a visual or DOM diff. Large pages may use a simplified set-based summary; listed lines are capped.
• Ignore patterns reduce noise from dates and counters but can also hide real changes if overused.
• Sites may block or rate-limit cloud IPs. Errors are visible in the dataset; previous state is kept on fetch failures so a transient outage does not look like a content wipe.
• At most 50 pages by default (100 hard max). Bodies truncated at maxBytes.
• No secrets, credentials or private pages should be placed in input. Do not put API keys in the Actor. Pricing is an account setting, never changed by Actor code at runtime. When pay-per-event is configured, the code emits only the explicit page-checked event for successful new/unchanged/changed checks; error rows are uncharged.

Local development

Requires Node.js 22+.

npm ci
npm test
apify validate-schema
apify run --input-file inputs/smoke.json

Unit tests do not call the network (HTTP is mocked). Live cloud smoke tests use a tiny public page list and a firm spending limit.

Pay-per-event billing (when configured)

Proposed rate (not activated by this README alone): $0.002 per successfully checked page via page-checked (status new, unchanged, or changed). Error rows and skipped robots-disallowed pages are uncharged. Platform Actor Start is $0.00005 per GB of memory (minimum one), matching other Steadfeed Actors. No automatic per-dataset-item event. The Actor refuses to run under PPE if rates or the synthetic dataset-item event disagree with these rules.

The remaining page-checked capacity is checked before each page and again before charging. Runs stop cleanly at the user's maximum total charge.

Deployment safeguards

Visibility and pricing are managed through the official Apify API/Console, not by this Actor at runtime. Publication and live pricing changes require explicit owner approval.

Credits and version

Version 0.2.0. Polite HTTP, robots evaluation and Steadfeed packaging patterns adapted from the AI Crawler Access Auditor and ATS Job Board Monitor. Standalone Actor; does not share state with those tools.