Sitemap Extractor & Website URL Change Monitor

Track a website's sitemap inventory for SEO research, competitor publishing and content pipelines. Find newly listed URLs, changes to the site's declared lastmod, and URLs removed from a completely scanned sitemap tree. Feed the resulting URL deltas into your crawler or knowledge-base refresh workflow.

Who this is for

Sitemap monitoring for SEO and content pipelines. Use the sitemap extractor to find URLs entering or leaving a website's sitemap inventory and to track changes in declared lastmod dates. Pass URL deltas into an SEO review or downstream crawler. A sitemap removal is not proof of an HTTP 404, and lastmod is not proof of a content edit.

Quick start

Export sitemap URLs immediately:

{"sitemapUrls":["https://docs.apify.com/sitemap.xml"],"maxUrlsPerRoot":1000,"maxSitemapsPerRoot":10,"outputMode":"all"}

Monitor changes on a schedule:

{"sitemapUrls":["https://docs.apify.com/sitemap.xml"],"maxUrlsPerRoot":10000,"maxSitemapsPerRoot":20,"urlContains":["/platform/"],"watchId":"docs-growth","outputMode":"changes","trackRemovals":true}

Paste the exact sitemap URL, not a website home page or article. XML sitemaps, gzip XML and nested sitemap indexes are supported. Indexes are followed on the same HTTPS origin, up to four nested levels. The Actor does not guess sitemap locations, discover them from robots.txt, crawl pages or verify page status codes.

Change tracking

The first successful changes-mode run establishes a silent baseline and returns a check summary. Later runs with the same Watch ID return newly observed URLs and changed lastmod values. The optional URL-contains filter is a case-sensitive OR filter applied after comparison; all scanned URLs are remembered, so changing filters does not replay existing URLs.

Removal requires two consecutive complete scans. A capped scan preserves unseen entries and disables removals. After an incomplete scan, the next complete scan re-establishes the removal baseline. A source or parsing failure preserves the previous snapshot and does not generate a check charge. This avoids treating a missing child sitemap, robots restriction or coverage limit as deleted pages.

removed means no longer listed in the scanned sitemap tree. It does not establish that a page was deleted or returns HTTP 404. lastmod is exactly what the website declares; it is not proof the content changed. Sites that update every lastmod on each build may produce many alerts. URLs without dates can still be new or removed, but cannot reveal body edits.

Output

Each recordType=url contains pageUrl, lastmod, changeFrequency, numeric priority when valid, source sitemapUrl, rootSitemapUrl, changeType, changedFields, previousLastmod, firstSeenAt and observedAt.

changeType is baseline, new, changed, unchanged or removed. Only lastmod changes trigger changed; priority, change frequency and source-file moves are informational. new means new to your Watch ID, not necessarily newly published.

A check_summary is always returned per successful root with URL/file counts, the sitemap-file inventory, duplicates, coverageComplete, coverageLimits, removalsCompared and change counts. Inspect coverage before treating an inventory as complete. source_error describes failed roots. The key-value store's SUMMARY contains the overall run and spending-limit status.

Pricing

$0.20 per successful root-sitemap check, including up to 10,000 unique URLs and 20 sitemap files, plus $0.00005 per run start at fixed 512 MB memory. Platform usage is included. No per-URL or per-summary charge.

Each input root is a separate check, even if two roots overlap. Empty, unchanged and successfully capped checks count; their coverage is reported explicitly. Failed checks do not generate the root event; the standard start fee still applies. Ten roots cost $2.00005 per run, or $60.0015 if checked daily for 30 days. These are buyer costs, not developer earnings. Maximum cost per run is checked before starting each root.

Coverage and source protections

• Up to ten roots per run; 1–10,000 unique URLs per root (default 1,000); 1–20 sitemap files including the root (default ten). Four nested index levels, a two-minute source-check budget, four MB compressed/wire data and 16 MB decoded XML per file. Limits are lower than the sitemap protocol's maximums; large sites may need separate roots.
• Strict valid XML with no DOCTYPE or entity declarations. RSS/Atom, HTML, plain-text maps, image/video detail records and cross-origin child sitemaps are unsupported. Only the main page loc is exported; media-extension locs are not treated as pages.
• Reads robots.txt before sitemap files, using the PublicWatch product token. Disallowed maps fail the root. Missing robots.txt (404/410) is allowed; other failures and non-text responses reject the check. At least half a second between sitemap requests; longer declared crawl delays are honored up to five seconds. Longer delays are unsupported within this monitor's time budget.
• Only public HTTPS hostnames can be fetched. Private/reserved IPs, credential URLs, cross-origin sitemap redirects and protection pages are rejected. TLS validation stays enabled; there is no login, proxy or blocking bypass. Page URLs are listed without fetching their contents.
• Conflicting lastmod values for duplicate URLs reject the check instead of inventing a change. Sitemap documents can change during pagination; the result is not an atomic site snapshot.
• History retains up to 30,000 URL entries per root, prioritizing recently observed entries; partial scans can accumulate entries up to that cap. Repeated truncation may cause evicted URLs to be observed as new again. Full page bodies are never stored. A new Watch ID resets history. Do not overlap runs using the same Watch ID.

Source format: sitemaps.org protocol.

Related tools: WordPress Content Watch, Google News Keyword Watch, Steam Review Change Watch.

Try one source before scheduling

1. Open the Input tab and use one supported source with outputMode: "all" for an immediate export.
2. Check the returned records and the check summary. Confirm the source, coverage and price suit your workflow.
3. To monitor, set a stable Watch ID and outputMode: "changes". The first successful run establishes a silent baseline.
4. Reuse that Watch ID on later runs. A summary without change records means the check completed without matching changes.
5. Add an Apify schedule after the manual checks work. Avoid overlapping runs with the same Watch ID. Use Maximum cost per run to cap event charges.

Public-data monitors by the same developer: browse the catalog.