Site Diff Engine

Most "change monitors" watch one page. This one watches a whole site and tells you what actually moved between two runs — pages that appeared, pages that vanished, and pages whose content or SEO surface really changed. Timestamps, tokens and view counters never trigger a false alert, and neither does a trailing slash.

How it works

1. Discover every page via sitemap.xml / sitemap_index.xml, same-site link-following, or both (crawlMode: "auto"), capped by maxPages.
2. Canonicalize every URL so /page, /page/, /page/index.html and /page?utm_source=x&ref=y all collapse to the exact same page identity — a page's own <link rel="canonical"> wins when present. This is the single most important correctness detail: without it, every run reports phantom adds and removes.
3. Fingerprint each page's cleaned main content (nav/ads/footers stripped, then timestamps, CSRF tokens, session ids and view counters stripped) separately from its title / H1 / meta description — so a copy edit and an SEO-relevant change never look the same.
4. Diff this run's site snapshot against the last one: ADDED, REMOVED, CHANGED (changeCategory: STRUCTURAL, BODY or BOTH), UNCHANGED.
5. Flag near-duplicates within the same run — thin category pages, boilerplate legal pages, auto-generated tag pages — via word-shingle Jaccard similarity.

The first run on a site only records a baseline (status: "BASELINE") — it never reports the whole site as "changed" just because nothing existed before.

Input

{
  "startUrl": "https://example.com/",
  "maxPages": 200,
  "crawlMode": "auto",
  "urlExcludePatterns": ["/(cart|checkout|login)(/|$)"],
  "onlyChanges": true
}

Output

One dataset row per page:

field	meaning
status	BASELINE · ADDED · REMOVED · CHANGED · UNCHANGED
changeCategory	STRUCTURAL (title/H1/meta), BODY, BOTH, or null
title / h1 / metaDescription	current values (null for REMOVED)
previousTitle / previousH1 / previousMetaDescription	prior run's values
nearDuplicateOf	canonical URLs of pages this one is near-identical to

SUMMARY in the key-value store gives run-level counts (added/removed/changed/unchanged, structural vs body change counts, near-duplicate page count).

Honest limits

• No JavaScript rendering. Pages that build their body client-side (SPA shells) will fingerprint as near-empty. Server-rendered and static sites work well.
• State is deliberately lean. Only fingerprints and the small structural fields (title/H1/ meta) are persisted between runs, not full page bodies — so a CHANGED body row tells you a page changed, not a word-level diff of what changed.
• Near-duplicate detection is O(n²) over the pages in one run. Fine at the maxPages scale this Actor targets; keep maxPages sane on very large sites.
• robots.txt is not consulted. Use urlExcludePatterns to keep the crawl off pages you don't want touched.
• Canonicalization is URL-rule + <link rel="canonical"> based. Sites with exotic session-based URL schemes (path-embedded session ids, etc.) may need urlExcludePatterns to avoid crawl explosion.

Schedule it

Run daily or weekly depending on how often the site changes. Point it at the same stateStoreName every run so the diff has something to compare against.