# Page Change Monitor (`steadfeed/page-change-monitor`) Actor

Watch public web pages for text changes. Snapshot pricing, terms or policy pages, compare on a schedule, and get concise diffs. Respects robots.txt. Does not log in or bypass paywalls. By Steadfeed.

- **URL**: https://apify.com/steadfeed/page-change-monitor.md
- **Developed by:** [Steadfeed](https://apify.com/steadfeed) (community)
- **Categories:** Automation, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page successfully checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Page Change Monitor — Steadfeed

**Version 0.2.0 — watch public pages for text changes, and get a plain-words summary of what actually changed.**

Fetch a list of public web pages (pricing, terms, policies, release notes), extract normalized visible text (optionally limited to a CSS selector), compare each page to the previous snapshot stored in a named key-value store, and emit one dataset row per URL with status `new`, `unchanged`, `changed`, or `error`, a content hash, timestamps, and a capped line diff. **New in 0.2:** every `changed` row also carries a `summary` that says, in plain words, what was added, removed or reworded, which prices, dates and numbers moved (before → after), and which lines were routine noise (timestamps, ads, cookie/footer text) and were ignored. It is deterministic and free: no AI service is involved unless you choose to add your own key.

Designed for **scheduled runs**: keep the same `watchlistId` and `stateStoreName` so baselines persist between Actor runs.

> This Actor reports **text differences it can see in the fetched HTML**. It does not guarantee that every visual or behind-login change is detected, and it does not bypass logins, paywalls, captchas or bot challenges. Respect robots.txt is on by default. A `changed` status is not legal, compliance or competitive advice.

### Meaningful-change summaries (new in 0.2)

Each `changed` row has a `summary` object, for example:

```json
{
  "mode": "heuristic",
  "meaningful": true,
  "headline": "Meaningful change: 1 price change, 1 date change, 1 new section.",
  "text": "Price changed: $29 → $39 in “Pro plan $39 per month” Date changed: Nov 20 → Nov 30 in “Sale ends Nov 30” New section added “Enterprise” (3 lines): …",
  "counts": { "priceChanges": 1, "dateChanges": 1, "sectionsAdded": 1, "noiseIgnored": 4 },
  "changes": [ { "type": "price", "from": "$29", "to": "$39", "text": "…" } ]
}
```

- **What it detects:** price changes (currency amounts), date changes, number and percentage changes, reworded passages (word-level before → after), single lines added or removed, and whole sections added or removed.
- **What it ignores as noise:** lines labelled "last updated / modified / generated / as of …" with a date, bare dates and clock times, relative times ("3 hours ago"), copyright years, ad and "sponsored" labels, cookie/consent and sign-in/footer boilerplate, view/visitor counters, and long random tokens. `summary.counts.noiseIgnored` and `summary.noiseReasons` show what was dropped. A page whose only changes are noise gets `summary.meaningful: false`.
- **Quiet mode:** set `skipNoiseOnlyChanges` to `true` and noise-only changes are reported as `unchanged` (the snapshot is still refreshed), so schedules and alerts fire only on real changes. Off by default, so status behavior matches 0.1.x.
- **Heuristics, not magic:** the summary works on the extracted text, line by line. It can mislabel an edit as a removal plus an addition, and cannot judge intent or legal significance. The raw `diff` stays on every row.
- **Turn it off:** `summarizeChanges: false` restores 0.1.x rows (`summary` is `null`).

#### Optional: AI summary with your own key

Everything above works with no AI. If you want a short AI-written paragraph on top, add `aiApiKey` (and optionally `aiBaseUrl`, `aiModel`) for any OpenAI-compatible chat-completions API. You pay your AI provider directly; Steadfeed supplies no key and charges nothing extra. Only the already-found change list (not the full page) is sent, only for meaningful changes, at most `maxAiSummaries` times per run (default 25). The key is a secret input, is never logged or written to output, and if the AI call fails the row still carries the free summary (`summary.ai.error` explains why). Default model is `gpt-6-luna`; set `aiModel` for any other.

### Input example

```json
{
  "pages": [
    { "url": "https://example.com/", "label": "Example homepage" },
    {
      "url": "https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document",
      "label": "W3C software notice"
    }
  ],
  "ignorePatterns": ["Last updated"],
  "watchlistId": "default",
  "stateStoreName": "steadfeed-page-change-monitor-state",
  "emitOnFirstRun": true,
  "emitUnchanged": true,
  "respectRobotsTxt": true,
  "requestIntervalMs": 1000,
  "maxDiffLines": 40
}
```

#### Fields

| Field | Default | Notes |
|---|---|---|
| `pages` | (required) | 1–50 entries. Each is a URL string or `{ url, selector?, label? }`. |
| `ignorePatterns` | `[]` | Case-insensitive substrings, or `/regex/` / `/regex/i`, stripped before hash/diff. |
| `watchlistId` | `default` | Separates state when several schedules share one store. |
| `stateStoreName` | `steadfeed-page-change-monitor-state` | Named Apify key-value store for snapshots. |
| `emitOnFirstRun` | `true` | Emit `new` rows on the first successful snapshot. |
| `emitUnchanged` | `true` | Emit rows when the hash matches. |
| `maxDiffLines` | `40` | Cap on listed added+removed lines (counts are full). |
| `requestIntervalMs` | `1000` | Minimum delay between requests to the same host. |
| `maxRetries` | `2` | Retries for network errors and 408/429/5xx. |
| `timeoutMs` | `20000` | Per-request timeout. |
| `maxBytes` | `2000000` | Response body size cap before parse. |
| `respectRobotsTxt` | `true` | Skip URLs disallowed for this Actor's user-agent. |
| `maxPages` | `50` | Hard cap on `pages.length`. |
| `summarizeChanges` | `true` | Add the plain-words `summary` to changed rows. |
| `skipNoiseOnlyChanges` | `false` | Report noise-only changes (timestamps, ads, counters, footer text) as `unchanged`. |
| `maxSummaryChanges` | `25` | Max items in `summary.changes` (1–100). |
| `aiApiKey` | empty | Optional, secret. Your own OpenAI-compatible key for an extra AI summary. |
| `aiBaseUrl` | `https://api.openai.com/v1` | https only (http allowed for localhost). |
| `aiModel` | `gpt-6-luna` | Model id your provider accepts. |
| `maxAiSummaries` | `25` | Max AI calls per run (1–200). |

Invalid input fails before network requests: unknown keys, bad URLs, duplicate url+selector pairs, and out-of-range numbers.

### How it works

1. Optionally fetch `/robots.txt` once per origin and evaluate Allow/Disallow for `Steadfeed-Page-Change-Monitor` (RFC 9309 longest-match). Missing robots (4xx) is treated as allow-all; unreachable or 5xx robots skips the page.
2. Fetch each page with polite per-host spacing, timeouts, bounded redirects and retries. No authentication, cookies, proxies or captcha bypass.
3. Parse HTML, drop script/style/etc., extract visible text (whole `main`/`article`/`body`, or a CSS selector region via Cheerio).
4. Apply ignore patterns, normalize whitespace, and compute a SHA-256 content hash.
5. Load the previous snapshot from the named store (`watchlistId` + url/selector key). Compare hashes; on change, produce a capped line diff and the plain-words summary.
6. Save the new snapshot and push a dataset row.

Light heuristics may set `accessGate` when the response looks like a login form or paywall copy, or when HTTP 401/402/403 is returned. The Actor still does not attempt to log in.

### Output

Default dataset: one `recordType: "page"` row per checked URL (subject to emit flags). Fields include status, url/finalUrl, label, selector, httpStatus, robots decision, title, contentHash, previous hash, textLength, timestamps, optional `diff` (`addedLines`, `removedLines`, counts, `truncated`), optional `summary` (see above), and error details.

`SUMMARY` in the default key-value store holds run totals. Snapshots live in the **named** state store, not the run's default store, so schedules can reuse them.

Formal schemas: `.actor/record_schema.json`, `.actor/dataset_schema.json`, `.actor/input_schema.json`.

### Limits and responsible use

- Public HTTP(S) HTML/text only. No login, paywall bypass, cookie consent clicking, or browser rendering of heavily JavaScript-built pages (content not present in the initial HTML will be missed).
- CSS selectors must match server-rendered markup. A missing selector yields `selector_no_match`.
- Diffs are line-oriented on normalized text, not a visual or DOM diff. Large pages may use a simplified set-based summary; listed lines are capped.
- Ignore patterns reduce noise from dates and counters but can also hide real changes if overused.
- Sites may block or rate-limit cloud IPs. Errors are visible in the dataset; previous state is kept on fetch failures so a transient outage does not look like a content wipe.
- At most 50 pages by default (100 hard max). Bodies truncated at `maxBytes`.
- No secrets, credentials or private pages should be placed in input. Do not put API keys in the Actor. Pricing is an account setting, never changed by Actor code at runtime. When pay-per-event is configured, the code emits only the explicit `page-checked` event for successful new/unchanged/changed checks; error rows are uncharged.

### Local development

Requires Node.js 22+.

```sh
npm ci
npm test
apify validate-schema
apify run --input-file inputs/smoke.json
```

Unit tests do not call the network (HTTP is mocked). Live cloud smoke tests use a tiny public page list and a firm spending limit.

### Pay-per-event billing (when configured)

Proposed rate (not activated by this README alone): **$0.002 per successfully checked page** via `page-checked` (status `new`, `unchanged`, or `changed`). Error rows and skipped robots-disallowed pages are uncharged. Platform **Actor Start** is $0.00005 per GB of memory (minimum one), matching other Steadfeed Actors. No automatic per-dataset-item event. The Actor refuses to run under PPE if rates or the synthetic dataset-item event disagree with these rules.

The remaining `page-checked` capacity is checked before each page and again before charging. Runs stop cleanly at the user's maximum total charge.

### Deployment safeguards

Visibility and pricing are managed through the official Apify API/Console, not by this Actor at runtime. Publication and live pricing changes require explicit owner approval.

### Credits and version

Version 0.2.0. Polite HTTP, robots evaluation and Steadfeed packaging patterns adapted from the AI Crawler Access Auditor and ATS Job Board Monitor. Standalone Actor; does not share state with those tools.

# Changelog

This Actor's version history is a separate document: https://apify.com/steadfeed/page-change-monitor/changelog.md

# Actor input Schema

## `pages` (type: `array`):

1–50 public http(s) page objects. Each needs a url; optional CSS selector limits the compared region; optional label is a short name. Duplicate url+selector pairs are rejected. Does not log in or bypass paywalls.

## `ignorePatterns` (type: `array`):

Optional substrings (case-insensitive) or /regex/ /regex/i patterns stripped before hashing and diffing. Use for rotating timestamps, view counters, or cookie banners. At most 30 patterns.

## `summarizeChanges` (type: `boolean`):

On changed pages, add a `summary` with what was added, removed or reworded, price/date/number changes, and routine noise (timestamps, ads, cookie/footer text) filtered out. Deterministic and free: no AI is used unless you add your own key below.

## `skipNoiseOnlyChanges` (type: `boolean`):

When the only differences are timestamps, ads, counters or footer text, report the page as `unchanged` instead of `changed` (the stored snapshot is still refreshed). Off by default so behavior matches earlier versions.

## `maxSummaryChanges` (type: `integer`):

Cap on items in `summary.changes` (1–100).

## `watchlistId` (type: `string`):

Separates snapshot state when you run several schedules against the same named store. Letters, digits, \_, -, . only; max 60.

## `stateStoreName` (type: `string`):

Named Apify key-value store that holds page snapshots between scheduled runs. 3–63 characters.

## `emitOnFirstRun` (type: `boolean`):

When true (default), the first successful fetch of a page outputs a status=new row and saves the baseline. When false, only the baseline is saved.

## `emitUnchanged` (type: `boolean`):

When true (default), pages whose content hash matches the previous snapshot still produce a dataset row. Turn off to only receive new/changed/error rows.

## `maxDiffLines` (type: `integer`):

Cap on added+removed lines included in the diff summary (5–200). Counts always reflect the full diff.

## `requestIntervalMs` (type: `integer`):

Polite per-host spacing. 200–30000; default 1000.

## `maxRetries` (type: `integer`):

Retries for network errors and 408/429/5xx. 0–5; default 2.

## `timeoutMs` (type: `integer`):

Per-request timeout. 3000–60000; default 20000.

## `maxBytes` (type: `integer`):

Bodies larger than this are truncated before parsing. 50000–5000000; default 2000000.

## `respectRobotsTxt` (type: `boolean`):

When true (default), fetches /robots.txt per origin and skips URLs disallowed for this Actor's user-agent. Unreachable or 5xx robots.txt also skips the page.

## `maxPages` (type: `integer`):

Hard cap on pages array length for this run. 1–100; default 50.

## `aiApiKey` (type: `string`):

Optional. Bring your own key for an OpenAI-compatible API to add a short AI-written summary on top of the built-in one. You pay your AI provider directly; Steadfeed never supplies or stores a key. Leave empty for free deterministic summaries only.

## `aiBaseUrl` (type: `string`):

OpenAI-compatible base URL (https). Default https://api.openai.com/v1.

## `aiModel` (type: `string`):

Model id your provider accepts. Default gpt-6-luna.

## `maxAiSummaries` (type: `integer`):

Caps how many AI calls (billed by your provider) one run may make (1–200).

## Actor input object example

```json
{
  "pages": [
    {
      "url": "https://example.com/",
      "label": "Example homepage"
    },
    {
      "url": "https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document",
      "label": "W3C software notice"
    }
  ],
  "summarizeChanges": true,
  "skipNoiseOnlyChanges": false,
  "maxSummaryChanges": 25,
  "watchlistId": "default",
  "stateStoreName": "steadfeed-page-change-monitor-state",
  "emitOnFirstRun": true,
  "emitUnchanged": true,
  "maxDiffLines": 40,
  "requestIntervalMs": 1000,
  "maxRetries": 2,
  "timeoutMs": 20000,
  "maxBytes": 2000000,
  "respectRobotsTxt": true,
  "maxPages": 50,
  "aiBaseUrl": "https://api.openai.com/v1",
  "aiModel": "gpt-6-luna",
  "maxAiSummaries": 25
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `changes` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "pages": [
        {
            "url": "https://example.com/",
            "label": "Example homepage"
        },
        {
            "url": "https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document",
            "label": "W3C software notice"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("steadfeed/page-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "pages": [
        {
            "url": "https://example.com/",
            "label": "Example homepage",
        },
        {
            "url": "https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document",
            "label": "W3C software notice",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("steadfeed/page-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "pages": [
    {
      "url": "https://example.com/",
      "label": "Example homepage"
    },
    {
      "url": "https://www.w3.org/Consortium/Legal/2015/copyright-software-and-document",
      "label": "W3C software notice"
    }
  ]
}' |
apify call steadfeed/page-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,steadfeed/page-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/13ovpGhcOIEXIB2DL/builds/h32ujaxglrsFWnYWc/openapi.json