Sitemap Change Monitor – New, Updated & Removed Pages

Sitemap Change Monitor watches the XML sitemaps of any websites and tells you which pages were added, updated (their <lastmod> changed) or removed since the last run. Schedule it daily or weekly to get a clean feed of what competitors, publishers, docs sites or your own site actually changed – without crawling every page.

• 🔎 Auto-discovers sitemaps from robots.txt, then /sitemap.xml and /sitemap_index.xml
• 🗂️ Follows sitemap indexes, gzipped .xml.gz sitemaps, news/image sitemaps and plain-text sitemaps
• 🆕 Only changes mode – remembers every site's URLs in a named key-value store and outputs only the difference
• 🎯 Include / exclude URL patterns (e.g. only /blog/)
• 🛡️ No false "removed" alerts: removal detection is skipped when a sitemap fails to load or the URL limit is hit
• No login, no API keys, no browser – just HTTP requests to public sitemap files

Input

Field	Description
sites	Homepages (https://example.com) or direct sitemap URLs. Empty = demo (https://crawlee.dev).
onlyChanges	Output only new / updated / removed URLs (default on). Off = every URL with a changeType.
outputBaselineUrls	On the first run of a site, output all URLs as new (default on). Turn off to record the baseline silently.
reportRemoved	Report URLs that disappeared from the sitemaps (default on).
trackLastmod	Report updated when <lastmod> changes (default on).
includeUrlRegex / excludeUrlRegex	Optional regex filters on page URLs.
maxUrlsPerSite	Default 10,000 (max 100,000).
maxSitemapsPerSite	Default 200 sitemap files.
stateStoreName	Named key-value store with each site's previous URLs – use one name per monitor.

{
  "sites": ["https://competitor.com", "https://example.com/sitemap_index.xml"],
  "onlyChanges": true,
  "includeUrlRegex": "/blog/|/pricing",
  "maxUrlsPerSite": 20000
}

Output

One item per changed URL:

{
  "site": "https://www.gov.uk",
  "url": "https://www.gov.uk/hmrc-internal-manuals/advance-tax-certainty-service/atcs02100",
  "changeType": "updated",
  "lastmod": "2026-09-30T10:48:14+01:00",
  "previousLastmod": "2026-08-12T09:00:00+01:00",
  "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
  "isBaseline": false,
  "detectedAt": "2026-11-02T17:40:01.000Z"
}

changeType is new, updated, removed (or unchanged when Only output changes is off). A RUN_SUMMARY record in the default key-value store has per-site counts and sitemap errors.

Use cases

• Competitor monitoring – new product, landing, pricing or blog pages the day they go live
• SEO – check your own sitemaps after each release: pages accidentally dropped or added
• Content & news tracking – feed new articles from publishers' news sitemaps into Slack, email, Google Sheets or an LLM summariser (via Apify integrations / webhooks)
• Docs & changelog watching – know when a vendor publishes new documentation
• Lead generation – new location, job or listing pages on directory sites

Tips

• Run on a schedule (Apify Schedules) with the same stateStoreName; connect a Slack / email integration to the run's dataset.
• Very large sites: raise maxUrlsPerSite and memory, or point the Actor at one specific child sitemap.
• Sites that rewrite every <lastmod> on each deploy: turn off trackLastmod.

Limitations

• Only sees pages that the site lists in its sitemaps; it does not crawl links or compare page content.
• updated relies on the site's own <lastmod> values, which some sites omit or set inaccurately.
• Sites that block bots or require JavaScript to serve sitemaps are not supported.

Legal note

The Actor downloads only publicly published sitemap files (which sites publish specifically for automated discovery) and respects their URLs as given; it does not log in, bypass protection or collect personal data. You are responsible for using the results in line with the target sites' terms and applicable law.