# Website Change Monitor — Sitemap & Page Content Diff (`owlabee/website-change-monitor`) Actor

Track website changes via sitemaps: finds new, removed and updated pages (lastmod) and detects page content changes with normalized hashes and word-level diff previews. Pay per page checked ($0.0005) and per change detected ($0.002).

- **URL**: https://apify.com/owlabee/website-change-monitor.md
- **Developed by:** [Owlabee](https://apify.com/owlabee) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 page checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Change Monitor — Sitemap & Page Content Diff

### What it does

Website Change Monitor discovers every page of a site through its sitemaps and tells you what changed since the last run. Give it site URLs or sitemap URLs; it follows `robots.txt` `Sitemap:` lines, sitemap indexes, plain and gzipped sitemaps (`.xml`, `.xml.gz`, `.txt`), and falls back to `/sitemap.xml` and `/sitemap_index.xml`. In **discover** mode it lists every page URL with `lastmod`, `changefreq` and `priority`. In **monitor** mode it keeps a snapshot per site in a key-value store and emits one record per change: **new-url**, **removed-url**, **lastmod-changed**, **content-changed** and **status-changed**.

Content changes are detected without a browser: each HTML page (max 1 MB, 20 s timeout) is scoped to a CSS selector (`main, article, body` by default), stripped of scripts, styles, comments, SVG and whitespace noise, optionally stripped of numbers and dates, and hashed with SHA-256. Every record also carries the page `title`, `metaDescription`, `canonical`, `h1`, `wordCount`, a 300-character `textPreview` and, when content changed, a compact word-level `changedPreview` (added/removed word counts plus the first words of each).

It is built for SEO teams watching their own or competitors' sites, product and pricing-page monitoring, documentation and compliance teams that need an audit trail of content changes, and AI agents that need a cheap, structured "what changed on this site" feed. Results are available as JSON, CSV, Excel, through the API, webhooks and the Apify MCP server.

### How to use

1. Enter one or more **Sites or sitemap URLs** (`https://example.com` or `https://example.com/sitemap.xml`).
2. Pick a **Mode**: `discover` to list the sitemap, or `monitor` to track changes. Optionally narrow the pages with **Include/Exclude URL patterns**, set **Max pages per site**, a **Content CSS selector**, and a **Webhook URL**.
3. Click **Start**. The first monitor run writes `baseline` records; schedule the Actor (hourly, daily, weekly) and every later run emits only the changes.

### Input

| Field | Type | Required | Description |
|---|---|---|---|
| `sites` | string\[] | yes | Site or sitemap URLs, one per line |
| `mode` | `discover` | `monitor` | no | `discover` lists sitemap URLs; `monitor` diffs against the previous run (default `monitor`) |
| `checkContent` | boolean | no | Monitor mode: fetch and hash each HTML page (default `true`). Off = sitemap-level changes only |
| `contentSelector` | string | no | Comma-separated CSS selectors; first match is hashed (default `main, article, body`) |
| `ignoreNumbers` | boolean | no | Strip digits, dates and times before hashing (default `false`) |
| `maxPages` | integer | no | Max page URLs per site after filtering, in sitemap order (default 500) |
| `includePatterns` | string\[] | no | Keep only matching URLs: globs (`*/blog/*`), regexes (`/\.html$/`) or substrings |
| `excludePatterns` | string\[] | no | Drop matching URLs, same syntax |
| `maxConcurrency` | integer | no | Parallel page fetches, 1–20 (default 5); same-host requests are spaced ≥ 500 ms |
| `stateStoreName` | string | no | Named key-value store holding one snapshot per site (default `website-change-monitor-state`) |
| `resetBaseline` | boolean | no | Ignore the stored snapshot and write a fresh baseline |
| `webhookUrl` | string | no | POST target for the run summary and change list |
| `notifyOnlyOnChanges` | boolean | no | Call the webhook only when changes were found (default `true`) |

### Output example

A `discover` record:

```json
{
  "changeType": "discovered",
  "site": "https://www.sitemaps.org/sitemap.xml",
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "detectedAt": "2026-10-16T12:49:14.231Z"
}
```

`changefreq` and `priority` are included whenever the sitemap provides them:

```json
{ "changeType": "discovered", "url": "https://nodejs.org/en/download/current", "lastmod": "2026-10-15T14:35:17.575Z", "changefreq": "always", "priority": 0.7, "...": "..." }
```

A `content-changed` record from a monitor run (`before`/`after` hold the full previous and current page snapshots):

```json
{
  "changeType": "content-changed",
  "site": "https://example.com",
  "url": "https://example.com/pricing",
  "changedFields": ["contentHash", "lastmod"],
  "title": "Pricing — Example",
  "lastmod": "2026-10-16",
  "status": 200,
  "contentHash": "3f9a1c…",
  "changedPreview": {
    "addedWords": 3,
    "removedWords": 1,
    "addedPreview": "$59 SSO included",
    "removedPreview": "$49",
    "similarity": 0.94
  },
  "before": { "url": "https://example.com/pricing", "status": 200, "lastmod": "2026-09-01", "contentHash": "a81b…", "title": "Pricing — Example", "h1": "Pricing that scales", "wordCount": 412, "textPreview": "Pricing that scales Team plan costs $49 per month…", "checkedAt": "2026-10-09T12:00:04.101Z" },
  "after":  { "url": "https://example.com/pricing", "status": 200, "lastmod": "2026-10-16", "contentHash": "3f9a1c…", "title": "Pricing — Example", "h1": "Pricing that scales", "wordCount": 414, "textPreview": "Pricing that scales Team plan costs $59 per month…", "checkedAt": "2026-10-16T12:00:03.877Z" },
  "detectedAt": "2026-10-16T12:00:09.000Z"
}
```

`changeType` values: `discovered`, `baseline`, `new-url`, `removed-url`, `lastmod-changed`, `content-changed`, `status-changed`, `error`. When a page changes in several ways at once you get one record whose `changeType` is the most significant change (status > content > lastmod) and whose `changedFields` lists all of them. Error items look like `{ "changeType": "error", "site": "...", "error": "NO_SITEMAP_URLS", "errorMessage": "No page URLs found. Tried: ..." }` and are charged at the page-checked rate, because the work to discover the problem was done.

### Pricing

Pay per event. You only pay for what the Actor produces; set a **maximum cost per run** in the run options to cap spend.

| Event | Price | Notes |
|---|---|---|
| Page checked | $0.0005 | Per URL listed in discover mode, per baseline record on the first monitor run, and per page checked on later monitor runs. Error items included |
| Change detected | $0.002 | Per change record (new-url, removed-url, lastmod-changed, content-changed, status-changed) |
| Actor start | $0.00005 | Apify platform event |

**Example:** checking 1,000 pages ≈ **$0.50**; each detected change adds **$0.002**. A daily monitor of a 300-page site with 5 changes per day ≈ $0.16/day. The first monitor run writes one baseline record per page and is charged at the page-checked price only.

### Limits and fair use

- Only sitemap-listed pages are tracked; pages missing from the sitemap are not discovered. `maxPages` cuts the list in sitemap order, so in monitor mode set it above the site's page count or the tail of the sitemap will look "removed".
- Pages are fetched with plain HTTP from Apify datacenter IPs, HTML only, up to 1 MB each, 20 s timeout, at most ~2 requests per second per host, and `robots.txt` Disallow rules are honoured (such pages get `status: "robots-disallowed"`). JavaScript-rendered content is not executed; use a selector that targets server-rendered markup.
- The word-level `changedPreview` is computed on the first 5,000 characters of normalized text kept in the snapshot; the hash covers the whole selected element.
- A snapshot is saved only after a run completes within the spending limit, so a run that hits its maximum cost does not corrupt the baseline; raise the limit and run again.
- Only public pages are accessed. No login, no proxies, no personal data beyond what the site publishes.

### Related Actors

Other pay-per-event Actors from the same account, all limited-permission and agent-friendly:

- [Shopify Store Products Scraper & Price Change Monitor](https://apify.com/owlabee/shopify-catalog-monitor) — price, stock and new/removed product change events for Shopify stores.
- [Podcast RSS Feed Scraper — Episodes & Transcripts](https://apify.com/owlabee/podcast-rss-scraper) — new-episode monitoring for podcast RSS feeds, with transcripts.
- [Domain WHOIS, DNS, Email Security & TLS Checker](https://apify.com/owlabee/domain-intelligence) — WHOIS, DNS, email security, TLS and tech stack for the domains you watch.

### FAQ

**Does this need proxies or an API key?** No. It reads public sitemaps and pages over plain HTTP from Apify's standard infrastructure. If a site blocks datacenter traffic, its pages come back as `status-changed`/`fetch-error` records rather than failing the run.

**How is cost controlled?** Cost = pages checked × $0.0005 + changes × $0.002. Limit pages with `maxPages` and `includePatterns`/`excludePatterns`, turn off `checkContent` for a sitemap-only check, and set the run's maximum total charge; the Actor stops cleanly when it is reached.

**What happens on the first monitor run?** There is nothing to compare against, so every page becomes a `baseline` record (charged as a page check). Changes appear from the second run on. Use `resetBaseline` to start over, or a different `stateStoreName` to keep separate baselines.

**How do I avoid false positives from counters, dates or ads?** Set `contentSelector` to the element that holds the real content (for example `article` or `#main-content`) and turn on `ignoreNumbers`. Scripts, styles, comments, SVG and whitespace differences are already ignored.

**Can I run it on a schedule or call it from an AI agent?** Yes — use Apify Schedules, the API, or the Apify MCP server. The Actor is pay-per-event and uses limited permissions (its own dataset and the state store you name), so agents can discover and call it. Example prompts: "Which pages on example.com changed this week?", "Tell me when the competitor's pricing page changes", "List every URL in acme.com's sitemap with its last-modified date".

**Does it work with sitemap indexes and gzipped sitemaps?** Yes: sitemap indexes are followed recursively, `.xml.gz` files are decompressed, plain-text sitemaps are accepted, and `robots.txt` `Sitemap:` lines are used when you give a site URL.

**What happens on errors?** A site without a usable sitemap or an invalid URL produces an `error` item with a clear code instead of failing the whole run; a page that cannot be fetched is recorded with `status: "fetch-error"` and compared normally next time.

### Legal

Use the data in compliance with the source's terms and applicable law. The Actor honours `robots.txt`, keeps request rates polite and only reads publicly available pages. It is not a consumer reporting agency and its output must not be used for employment, credit, insurance, or tenant screening decisions.

# Actor input Schema

## `sites` (type: `array`):

One entry per line. A site URL (https://example.com) is resolved through robots.txt `Sitemap:` lines, then /sitemap.xml and /sitemap_index.xml. A direct sitemap URL (.xml, .xml.gz, .txt, sitemap indexes) is used as-is. The prefilled value is the cheap, always-available example used by Apify's daily automated test.

## `mode` (type: `string`):

`discover` lists every page URL found in the sitemaps with lastmod/changefreq/priority (no page fetches, no state). `monitor` compares this run with the previous snapshot and emits change records; the first monitor run writes baseline records.

## `checkContent` (type: `boolean`):

Monitor mode only. Fetch each HTML page (max 1 MB, 20 s timeout), normalize its text and hash it, so `content-changed` and `status-changed` records are produced in addition to sitemap-level changes. Turn off to compare sitemap data only (much faster, no page requests).

## `contentSelector` (type: `string`):

Comma-separated CSS selectors; the first one that matches a non-empty element is hashed. Default `main, article, body`. Use it to ignore headers, footers and sidebars, e.g. `#content, .post-body`.

## `ignoreNumbers` (type: `boolean`):

Strip digits, dates and times before hashing so view counters, timestamps and prices do not count as content changes.

## `maxPages` (type: `integer`):

Upper bound on page URLs taken from the sitemaps of each site (after include/exclude filtering, in sitemap order). In monitor mode set it above the site's real page count so removals are detected correctly.

## `includePatterns` (type: `array`):

Only keep page URLs matching at least one pattern. Globs (`https://example.com/blog/*`), regular expressions (`/\/docs\/v\d+\//`) or plain substrings (`/pricing`). Empty = keep all.

## `excludePatterns` (type: `array`):

Drop page URLs matching any pattern (same syntax as include patterns), e.g. `*/tag/*`, `/\.pdf$/`.

## `maxConcurrency` (type: `integer`):

Parallel page fetches. Requests to the same host are additionally spaced at least 500 ms apart to stay polite (about 2 requests per second per host).

## `stateStoreName` (type: `string`):

Named key-value store that keeps one snapshot per site between monitor runs. Use a different name to keep independent baselines (e.g. per client or per schedule).

## `resetBaseline` (type: `boolean`):

Ignore the stored snapshot and write a fresh baseline for every site in this run.

## `webhookUrl` (type: `string`):

If set, the Actor POSTs a JSON summary of the run and the list of change records to this URL (10 s timeout, 2 retries). Failures are logged, never fatal.

## `notifyOnlyOnChanges` (type: `boolean`):

Monitor mode: call the webhook only when at least one change record was produced. Turn off to receive a webhook after every run.

## Actor input object example

```json
{
  "sites": [
    "https://www.sitemaps.org/sitemap.xml"
  ],
  "mode": "discover",
  "checkContent": true,
  "ignoreNumbers": false,
  "maxPages": 15,
  "maxConcurrency": 5,
  "stateStoreName": "website-change-monitor-state",
  "resetBaseline": false,
  "notifyOnlyOnChanges": true
}
```

# Actor output Schema

## `results` (type: `string`):

One item per page checked or change detected in the default dataset (changeType, url, title, lastmod, status, changefreq, detectedAt; error items carry error and errorMessage). Fields are described in dataset_schema.json.

## `summary` (type: `string`):

SUMMARY record in the run's default key-value store: sitemaps and pages checked, changes detected, charges and webhook delivery status, written when the run finishes.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "https://www.sitemaps.org/sitemap.xml"
    ],
    "mode": "discover",
    "maxPages": 15
};

// Run the Actor and wait for it to finish
const run = await client.actor("owlabee/website-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": ["https://www.sitemaps.org/sitemap.xml"],
    "mode": "discover",
    "maxPages": 15,
}

# Run the Actor and wait for it to finish
run = client.actor("owlabee/website-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "https://www.sitemaps.org/sitemap.xml"
  ],
  "mode": "discover",
  "maxPages": 15
}' |
apify call owlabee/website-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,owlabee/website-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lWwYQcG3MvjjNFYfq/builds/wuKt3rZErkek3qq1U/openapi.json