Website Change Monitor — Sitemap & Page Content Diff

What it does

Website Change Monitor discovers every page of a site through its sitemaps and tells you what changed since the last run. Give it site URLs or sitemap URLs; it follows robots.txt Sitemap: lines, sitemap indexes, plain and gzipped sitemaps (.xml, .xml.gz, .txt), and falls back to /sitemap.xml and /sitemap_index.xml. In discover mode it lists every page URL with lastmod, changefreq and priority. In monitor mode it keeps a snapshot per site in a key-value store and emits one record per change: new-url, removed-url, lastmod-changed, content-changed and status-changed.

Content changes are detected without a browser: each HTML page (max 1 MB, 20 s timeout) is scoped to a CSS selector (main, article, body by default), stripped of scripts, styles, comments, SVG and whitespace noise, optionally stripped of numbers and dates, and hashed with SHA-256. Every record also carries the page title, metaDescription, canonical, h1, wordCount, a 300-character textPreview and, when content changed, a compact word-level changedPreview (added/removed word counts plus the first words of each).

It is built for SEO teams watching their own or competitors' sites, product and pricing-page monitoring, documentation and compliance teams that need an audit trail of content changes, and AI agents that need a cheap, structured "what changed on this site" feed. Results are available as JSON, CSV, Excel, through the API, webhooks and the Apify MCP server.

How to use

1. Enter one or more Sites or sitemap URLs (https://example.com or https://example.com/sitemap.xml).
2. Pick a Mode: discover to list the sitemap, or monitor to track changes. Optionally narrow the pages with Include/Exclude URL patterns, set Max pages per site, a Content CSS selector, and a Webhook URL.
3. Click Start. The first monitor run writes baseline records; schedule the Actor (hourly, daily, weekly) and every later run emits only the changes.

Input

Field	Type	Required	Description
sites	string[]	yes	Site or sitemap URLs, one per line
mode	discover | monitor	no	discover lists sitemap URLs; monitor diffs against the previous run (default monitor)
checkContent	boolean	no	Monitor mode: fetch and hash each HTML page (default true). Off = sitemap-level changes only
contentSelector	string	no	Comma-separated CSS selectors; first match is hashed (default main, article, body)
ignoreNumbers	boolean	no	Strip digits, dates and times before hashing (default false)
maxPages	integer	no	Max page URLs per site after filtering, in sitemap order (default 500)
includePatterns	string[]	no	Keep only matching URLs: globs (*/blog/*), regexes (/\.html$/) or substrings
excludePatterns	string[]	no	Drop matching URLs, same syntax
maxConcurrency	integer	no	Parallel page fetches, 1–20 (default 5); same-host requests are spaced ≥ 500 ms
stateStoreName	string	no	Named key-value store holding one snapshot per site (default website-change-monitor-state)
resetBaseline	boolean	no	Ignore the stored snapshot and write a fresh baseline
webhookUrl	string	no	POST target for the run summary and change list
notifyOnlyOnChanges	boolean	no	Call the webhook only when changes were found (default true)

Output example

A discover record:

{
  "changeType": "discovered",
  "site": "https://www.sitemaps.org/sitemap.xml",
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "detectedAt": "2026-10-16T12:49:14.231Z"
}

changefreq and priority are included whenever the sitemap provides them:

{ "changeType": "discovered", "url": "https://nodejs.org/en/download/current", "lastmod": "2026-10-15T14:35:17.575Z", "changefreq": "always", "priority": 0.7, "...": "..." }

A content-changed record from a monitor run (before/after hold the full previous and current page snapshots):

{
  "changeType": "content-changed",
  "site": "https://example.com",
  "url": "https://example.com/pricing",
  "changedFields": ["contentHash", "lastmod"],
  "title": "Pricing — Example",
  "lastmod": "2026-10-16",
  "status": 200,
  "contentHash": "3f9a1c…",
  "changedPreview": {
    "addedWords": 3,
    "removedWords": 1,
    "addedPreview": "$59 SSO included",
    "removedPreview": "$49",
    "similarity": 0.94
  },
  "before": { "url": "https://example.com/pricing", "status": 200, "lastmod": "2026-09-01", "contentHash": "a81b…", "title": "Pricing — Example", "h1": "Pricing that scales", "wordCount": 412, "textPreview": "Pricing that scales Team plan costs $49 per month…", "checkedAt": "2026-10-09T12:00:04.101Z" },
  "after":  { "url": "https://example.com/pricing", "status": 200, "lastmod": "2026-10-16", "contentHash": "3f9a1c…", "title": "Pricing — Example", "h1": "Pricing that scales", "wordCount": 414, "textPreview": "Pricing that scales Team plan costs $59 per month…", "checkedAt": "2026-10-16T12:00:03.877Z" },
  "detectedAt": "2026-10-16T12:00:09.000Z"
}

changeType values: discovered, baseline, new-url, removed-url, lastmod-changed, content-changed, status-changed, error. When a page changes in several ways at once you get one record whose changeType is the most significant change (status > content > lastmod) and whose changedFields lists all of them. Error items look like { "changeType": "error", "site": "...", "error": "NO_SITEMAP_URLS", "errorMessage": "No page URLs found. Tried: ..." } and are charged at the page-checked rate, because the work to discover the problem was done.

Pricing

Pay per event. You only pay for what the Actor produces; set a maximum cost per run in the run options to cap spend.

Event	Price	Notes
Page checked	$0.0005	Per URL listed in discover mode, per baseline record on the first monitor run, and per page checked on later monitor runs. Error items included
Change detected	$0.002	Per change record (new-url, removed-url, lastmod-changed, content-changed, status-changed)
Actor start	$0.00005	Apify platform event

Example: checking 1,000 pages ≈ $0.50; each detected change adds $0.002. A daily monitor of a 300-page site with 5 changes per day ≈ $0.16/day. The first monitor run writes one baseline record per page and is charged at the page-checked price only.

Limits and fair use

• Only sitemap-listed pages are tracked; pages missing from the sitemap are not discovered. maxPages cuts the list in sitemap order, so in monitor mode set it above the site's page count or the tail of the sitemap will look "removed".
• Pages are fetched with plain HTTP from Apify datacenter IPs, HTML only, up to 1 MB each, 20 s timeout, at most ~2 requests per second per host, and robots.txt Disallow rules are honoured (such pages get status: "robots-disallowed"). JavaScript-rendered content is not executed; use a selector that targets server-rendered markup.
• The word-level changedPreview is computed on the first 5,000 characters of normalized text kept in the snapshot; the hash covers the whole selected element.
• A snapshot is saved only after a run completes within the spending limit, so a run that hits its maximum cost does not corrupt the baseline; raise the limit and run again.
• Only public pages are accessed. No login, no proxies, no personal data beyond what the site publishes.

Related Actors

Other pay-per-event Actors from the same account, all limited-permission and agent-friendly:

• Shopify Store Products Scraper & Price Change Monitor — price, stock and new/removed product change events for Shopify stores.
• Podcast RSS Feed Scraper — Episodes & Transcripts — new-episode monitoring for podcast RSS feeds, with transcripts.
• Domain WHOIS, DNS, Email Security & TLS Checker — WHOIS, DNS, email security, TLS and tech stack for the domains you watch.

FAQ

Does this need proxies or an API key? No. It reads public sitemaps and pages over plain HTTP from Apify's standard infrastructure. If a site blocks datacenter traffic, its pages come back as status-changed/fetch-error records rather than failing the run.

How is cost controlled? Cost = pages checked × $0.0005 + changes × $0.002. Limit pages with maxPages and includePatterns/excludePatterns, turn off checkContent for a sitemap-only check, and set the run's maximum total charge; the Actor stops cleanly when it is reached.

What happens on the first monitor run? There is nothing to compare against, so every page becomes a baseline record (charged as a page check). Changes appear from the second run on. Use resetBaseline to start over, or a different stateStoreName to keep separate baselines.

How do I avoid false positives from counters, dates or ads? Set contentSelector to the element that holds the real content (for example article or #main-content) and turn on ignoreNumbers. Scripts, styles, comments, SVG and whitespace differences are already ignored.

Can I run it on a schedule or call it from an AI agent? Yes — use Apify Schedules, the API, or the Apify MCP server. The Actor is pay-per-event and uses limited permissions (its own dataset and the state store you name), so agents can discover and call it. Example prompts: "Which pages on example.com changed this week?", "Tell me when the competitor's pricing page changes", "List every URL in acme.com's sitemap with its last-modified date".

Does it work with sitemap indexes and gzipped sitemaps? Yes: sitemap indexes are followed recursively, .xml.gz files are decompressed, plain-text sitemaps are accepted, and robots.txt Sitemap: lines are used when you give a site URL.

What happens on errors? A site without a usable sitemap or an invalid URL produces an error item with a clear code instead of failing the whole run; a page that cannot be fetched is recorded with status: "fetch-error" and compared normally next time.

Legal

Use the data in compliance with the source's terms and applicable law. The Actor honours robots.txt, keeps request rates polite and only reads publicly available pages. It is not a consumer reporting agency and its output must not be used for employment, credit, insurance, or tenant screening decisions.