Sitemap Change Monitor: New Competitor Pages

What does Sitemap Change Monitor do?

Sitemap Change Monitor watches public XML sitemaps (including sitemap indexes and .xml.gz files) and robots.txt Sitemap: directives, then outputs only the URLs that changed since the previous run — added, removed, or <lastmod> changed. It is a lightweight, browser-free sitemap change detection API alternative for SEO and content ops teams.

It does not crawl page HTML, bypass logins, or use proxies. Public sitemap data only.

---

Why use Sitemap Change Monitor?

Use case	How it helps
Competitor new-page tracking	Spot new URLs in a competitor's sitemap as they publish
Content / launch detection	Catch product, blog, or docs launches the moment they appear in the sitemap
SEO monitoring	Track lastmod churn and removals without scraping every page
Scheduled automation	Pair with Apify Schedules + webhooks / Zapier / Make / Slack

Benefits

• Cheap compute — plain HTTP + fast XML parse (no browser)
• Stateful diffs — named key-value store baseline across scheduled runs
• Index-aware — recursively expands sitemap indexes with configurable caps
• Gzip + robots.txt — handles .xml.gz and discovers sitemaps from robots.txt
• Optional filters — include/exclude URL regexes
• Optional webhook — JSON POST when changes are found

---

What data can Sitemap Change Monitor extract?

Field	Type	Description
changeType	string	added | removed | lastmodChanged
url	string	Page URL from <loc>
lastmod	string | null	Current <lastmod> (null if omitted or removed)
previousLastmod	string | null	Prior baseline lastmod
sourceSitemap	string	Concrete sitemap that listed the URL
detectedAt	string	ISO-8601 detection time

You can download the dataset as JSON, CSV, Excel, or HTML from the Apify Console, or via the Apify API.

---

How to monitor sitemap changes

1. Open Sitemap Change Monitor on Apify Store and click Try for free
2. Paste one or more public sitemap.xml, .xml.gz, or robots.txt URLs
3. Keep Emit all URLs on first run unchecked to record a quiet baseline (recommended)
4. (Optional) Set include/exclude regexes and a webhook URL
5. Click Start, or create a Schedule (e.g. hourly or daily)
6. On later runs, read the Dataset for changes and the OUTPUT key for a summary

Scheduling tips

• Use the same stateStoreName and stateKey on every scheduled run so diffs stay continuous
• Start with hourly or daily cadence; aggressive polling is rarely needed for sitemaps
• Set maxSitemaps / maxUrls to bound cost on large sites

---

How much will it cost?

Designed for pay-per-result style pricing (each dataset row = one meaningful change). Compute stays low because the Actor only does HTTP fetches and XML parsing — no headless browser. Exact Store pricing is configured by the publisher; free-tier compute applies while you try it.

---

Input

See the Input tab for the full form. Main fields:

Field	Required	Default	Description
sitemapUrls	✅	—	Public sitemap / index / .xml.gz / robots.txt URLs
maxSitemaps	❌	50	Cap on sitemap documents fetched while expanding indexes
maxUrls	❌	10000	Cap on page URLs tracked per run
stateStoreName	❌	sitemap-change-monitor-state	Named KVS for baselines across runs
stateKey	❌	sitemap-state	Record key inside that store
emitAllOnFirstRun	❌	false	If false, first run baselines only (no dataset rows)
includeUrlRegex	❌	—	Keep only matching page URLs
excludeUrlRegex	❌	—	Drop matching page URLs
webhookUrl	❌	—	POST JSON summary when changes exist
requestTimeoutSecs	❌	30	Per-request timeout
maxRetries	❌	3	Retries with exponential backoff

Example input

{
  "sitemapUrls": [
    "https://www.apify.com/sitemap.xml",
    "https://example.com/robots.txt"
  ],
  "maxSitemaps": 50,
  "maxUrls": 10000,
  "stateStoreName": "sitemap-change-monitor-state",
  "stateKey": "sitemap-state",
  "emitAllOnFirstRun": false,
  "includeUrlRegex": "",
  "excludeUrlRegex": ".*\\.(pdf|jpg|png)$"
}

---

Output

Dataset example

[
  {
    "changeType": "added",
    "url": "https://example.com/blog/new-post",
    "lastmod": "2026-10-08",
    "previousLastmod": null,
    "sourceSitemap": "https://example.com/sitemap-posts.xml",
    "detectedAt": "2026-10-08T15:30:00.000Z"
  },
  {
    "changeType": "lastmodChanged",
    "url": "https://example.com/pricing",
    "lastmod": "2026-10-07",
    "previousLastmod": "2026-09-01",
    "sourceSitemap": "https://example.com/sitemap.xml",
    "detectedAt": "2026-10-08T15:30:00.000Z"
  },
  {
    "changeType": "removed",
    "url": "https://example.com/old-landing",
    "lastmod": null,
    "previousLastmod": "2026-01-01",
    "sourceSitemap": "https://example.com/sitemap.xml",
    "detectedAt": "2026-10-08T15:30:00.000Z"
  }
]

Key-value store

Location	Key	Contents
Default store	OUTPUT	Run summary (totals + per-root stats)
Named store (stateStoreName)	stateKey	Baseline map of URL → lastmod per input root

Webhook payload (when configured and changes exist)

Includes totalChanges, added / removed / lastmodChanged, per-root summaries, and a sample of up to 50 change rows. Full rows are always in the dataset.

---

FAQ

Does the first run flood my dataset?

No — by default emitAllOnFirstRun is false, so the first run only writes a baseline and an OUTPUT summary. Set it to true if you want every URL emitted as added once.

Can I pass robots.txt instead of a sitemap?

Yes. If a URL path ends with robots.txt, the Actor reads Sitemap: lines and expands those sitemaps (still public HTTP only).

Are gzipped sitemaps supported?

Yes. .xml.gz URLs and gzip-encoded responses are decompressed automatically.

Does this respect robots.txt beyond sitemap discovery?

The Actor uses robots.txt only to discover listed sitemaps. It sends an honest User-Agent, uses timeouts and retries with backoff, and does not scrape page bodies. You are responsible for complying with each site's terms and for polite scheduling.

How do scheduled runs keep state?

Use a named key-value store (stateStoreName) and a stable stateKey. The default run store is not shared across runs; the named store is.

Local development

cd sitemap-change-monitor
# Edit storage/key_value_stores/default/INPUT.json
apify run
# Inspect storage/datasets/default/ and storage/key_value_stores/

---

Legal & acceptable use

• Public sitemaps and robots.txt only. Do not use this Actor on URLs that require login, cookies, API keys, or other authentication.
• Respect publisher terms of service and rate limits. Prefer reasonable schedules.
• You are responsible for how you use the output (attribution, redistribution, downstream automation).
• This Actor does not bypass paywalls or crawl HTML page content.

Our Actors are ethical and do not extract private user data. They only read what site owners publish in public sitemaps / robots.txt. Results may still include URLs that identify people or businesses; comply with GDPR and other applicable laws, and scrape only when you have a legitimate purpose.

---

Limitations

• Very large indexes are truncated by maxSitemaps / maxUrls
• Sites that omit <lastmod> can only surface added / removed (not lastmod changes)
• Fetch failures are logged per root; other roots still process
• Duplicate <loc> values: first occurrence wins within a run

---

Changelog

0.1

• Initial release: sitemap + index + gzip + robots.txt discovery, named-store baselines, dataset diffs, OUTPUT summary, optional webhook and URL filters

---

Support

Open an issue on the Actor’s Apify page or contact the author through Apify Console messaging. For programmatic access, see the API tab.