Sitemap New Pages Watch: Added & Removed URLs

Give this Actor a public HTTPS site root. It reads the site's robots.txt, fetches only the sitemap files that robots.txt permits, stores a per-domain baseline of listed URLs, and on every later run reports which URLs were added, removed, or had their sitemap lastmod changed. Results are available as a dataset (JSON) and as a CSV file.

It does not crawl pages. It only reads sitemap XML, so runs are small and fast.

Typical uses

• SEO and content teams tracking when a competitor or partner publishes new pages.
• Agencies watching client sites for pages that disappear after a redeploy.
• Directory, catalog and documentation owners who want a scheduled "what changed" list.

Quick start (copy this input)

{
    "startUrl": "https://www.python.org/",
    "maxUrls": 5000,
    "reportUpdated": true
}

Replace startUrl with a site you are allowed to monitor. Whether a given site publishes a usable sitemap is up to that site; the Actor reports NO_SITEMAP when none is found.

1. First run creates the baseline and outputs one BASELINE_CREATED row. No changes are reported.
2. Later runs compare against the baseline and output one row per change.

Input

Field	Default	Meaning
startUrl (required)		Public HTTPS site root, no path or query.
includePathPrefixes	none	Track only URLs under these paths (e.g. /blog/). Each distinct set has its own baseline.
reportUpdated	true	Also report URLs whose lastmod changed.
maxUrls	5000	Hard cap (max 50000). Exceeding it ends the run as INCOMPLETE_SCAN.
maxSitemapFiles	25	Cap on sitemap files read, including those named in sitemap indexes (max 100).
maxChangesReported	2000	Cap on dataset rows; totals are still in SUMMARY.
resetBaseline	false	Replace the stored baseline with this run's snapshot.
baselineStoreName	pal-sitemap-watch-baselines	Named key-value store holding baselines.

Output

Dataset rows (recordType: "CHANGE") contain changeType (ADDED, REMOVED, UPDATED), url, lastmod, previousLastmod, sourceSitemap, startUrl, domain, detectedAt, previousCheckedAt and baselineCreatedAt. Runs without changes produce one RUN_STATUS row with a status and message.

Key-value store records of the run: SUMMARY (counts, skip counters, status) and CHANGES_CSV (the same rows as CSV).

Run statuses

Status	Meaning	Run result	Billable event
BASELINE_CREATED	First run for this site/filter	Succeeded	No
NO_CHANGES	Nothing changed	Succeeded	No
CHANGES_DETECTED	Added, removed or updated URLs found	Succeeded	Yes (one sitemap-change-report event)
ROBOTS_BLOCKED	robots.txt forbids access to the sitemaps	Succeeded	No
NO_SITEMAP	No sitemap found, or it contains no usable URLs	Succeeded	No
INCOMPLETE_SCAN	A limit was hit or a listed file failed; nothing saved	Failed	No
SITEMAP_ERROR / FETCH_ERROR	Broken XML, HTTP errors, unsafe redirect	Failed	No
INVALID_INPUT	Bad or unsafe input	Failed	No

A failed or incomplete run never overwrites the baseline, so the next good run still compares against the last complete snapshot.

The code requests one sitemap-change-report charge event only when changes are reported. No price is configured in this project; pricing is set separately by the publisher.

Scheduling

Create a Task with your input and add an Apify Schedule (for example daily). Keep the default baselineStoreName: baselines live in a named key-value store, which persists across runs, unlike the per-run default store. Use a different name per customer or project if you want separate baselines. Because baselines are keyed by hostname, two tasks watching the same host with the same filters and the same store share one baseline.

Safety behavior

• HTTPS only, default port only, no credentials in URLs, no IP-address hosts, no localhost or internal-style hostnames.
• The hostname must resolve only to public addresses (checked before every request).
• Redirects are followed at most 3 times and only to the same HTTPS host; anything else is refused.
• robots.txt is read first. Sitemap files and listed URLs that it disallows for this Actor are skipped. A 401/403 on robots.txt is treated as a block; an unreachable robots.txt is an error.
• Response sizes are bounded (robots 512 KB, sitemap download 10 MB, decompressed 20 MB). Sitemap DTD/entity declarations are rejected.
• URLs that look like they contain an email address are dropped. The Actor never visits listed pages, logs in, uses proxies, or sends notifications.

Limitations and caveats

• A sitemap is not the whole site. Pages missing from the sitemap are invisible, and a URL leaving the sitemap does not necessarily mean the page was deleted.
• lastmod quality varies. Many sites omit it or set it to the build time, which can create noisy UPDATED rows. Turn off reportUpdated for such sites.
• Caps are strict. Sites with more than maxUrls URLs (max 50,000) cannot be monitored in a single baseline; use includePathPrefixes to split them.
• Only URLs on exactly the monitored hostname are tracked; sitemaps on other hosts or subdomains are ignored.
• Robots handling uses Python's standard parser, which follows first-match rules and may differ from other crawlers on complex wildcard rules.
• Address checks happen before each request but the HTTP library resolves the name again when connecting, so DNS rebinding cannot be ruled out entirely.
• Sitemap index nesting is limited to 3 levels, and gzip sitemaps are supported.
• Time-of-run is when the Actor observed the change, not when the site published it.
• The offline test suite uses mocked HTTP only. Behavior on real sites has not been benchmarked here, so no performance, accuracy or demand figures are claimed.

Development

python -m unittest discover -s tests

Tests need no network and no Apify runtime.