XML Sitemap Change Monitor & Integrity Auditor

What does Sitemap Monitor & Integrity Auditor do?

Sitemap Monitor & Integrity Auditor is an HTTP-only sitemap audit and change-monitoring API. It discovers sitemap declarations through robots.txt, walks sitemap indexes, validates XML, gzip, and text sitemaps, and reports trustworthy URL inventory changes without crawling every listed page by default.

It distinguishes publisher-declared <lastmod> changes from actual page-content changes and never turns an incomplete sitemap tree into a wave of false URL removals.

Why use Sitemap Monitor & Integrity Auditor?

• Detect missing, broken, malformed, oversized, or circular sitemap infrastructure.
• Track new URLs, declared lastmod changes, and conservatively confirmed removals.
• Reuse unchanged sitemap state with ETag, Last-Modified, and HTTP 304 responses.
• Feed crawler queues, RAG ingestion, migration checks, Make, n8n, Zapier, or webhooks.
• Schedule monitors and consume results through the Apify API.
• Process large inventories with streaming XML parsing, SQLite working state, and sharded persistent storage.

The Actor does not use a browser, proxy, CAPTCHA service, or AI. It only fetches pages listed in a sitemap when page-status checking is explicitly enabled.

What data can it return?

Record type	What it contains
site_summary	Overall status, URL counts, issues, requests, bytes, and duration
url_observation	Current URL inventory or changes such as URL_ADDED
sitemap_observation	HTTP validators, hashes, sizes, reuse, and parse status
audit_finding	Deterministic issue code, severity, resource, and explanation

Every input site ends as SUCCESS, PARTIAL, or FAILED. Only a complete SUCCESS run can advance monitor state.

Completeness describes whether required sitemap resources were parsed and traversed. Audit severity describes findings about those resources; a warning or informational protocol finding does not by itself make inventory incomplete.

How much does sitemap monitoring cost?

The pay-per-event model charges for site checks, delivered logical URL records, and optional page checks at the rates shown in the Actor's Pricing tab. A URL copied to both the default dataset and the changes dataset is charged once. An unchanged scheduled monitor still performs a site check; a complete zero-event batch adds no URL-result charges.

Input

See the Input tab for every safety limit and timeout. A snapshot:

{
  "sites": [{"url": "https://example.com", "externalId": "main-site"}],
  "mode": "snapshot",
  "outputMode": "all"
}

A scheduled monitor:

{
  "sites": [{"url": "https://example.com", "monitorKey": "production"}],
  "mode": "monitor",
  "outputMode": "changesOnly"
}

Changing include or exclude patterns creates a new baseline. monitorKey must consistently identify one site, and the same monitor must not run concurrently. The lock is best-effort: its read/write operations are not atomic and do not provide distributed locking guarantees.

Output

You can download the default dataset in JSON, HTML, CSV, or Excel. The default key-value store also contains an OUTPUT summary and CHANGE_BATCHES descriptors. Monitor runs also create one run-specific changes dataset shared by that run's sites.

{
  "recordType": "url_observation",
  "schemaVersion": 2,
  "changeType": "URL_BASELINE",
  "url": "https://example.com/new-page",
  "currentLastmod": "2026-09-09",
  "eventId": "stable-sha256-event-id",
  "batchId": "complete-site-batch-id",
  "generation": 1
}

The first complete successful monitor run emits one URL_BASELINE event per current URL, including in changesOnly mode. Later complete runs emit additions, publisher-declared lastmod changes, pending and confirmed removal review events, and removal cancellations. An empty inventory still produces a complete zero-event batch. Partial and failed observations publish no complete synchronization batch and do not advance state.

Only consume change rows whose batchId appears in CHANGE_BATCHES; filter the referenced dataset by that batch ID. A row alone does not prove the full site batch was published. Delivery can repeat, so deduplicate with eventId. The descriptor is persisted before the state manifest advances; if state commit fails, a later run may repeat the same logical transition. This is replayable dataset delivery, not an acknowledged queue or exactly-once processing.

The default dataset retains the current inventory, audit, sitemap, and summary records. The dedicated changes feed is authoritative for removals, which are review flags only; the Actor never deletes downstream documents. outputMode=all still returns the current inventory; changesOnly includes first-run baseline rows.

Runnable SQLite consumer

Set APIFY_RUN_ID and APIFY_TOKEN, then run:

$env:APIFY_RUN_ID = "your-run-id"
$env:APIFY_TOKEN = "your-api-token"
python examples/consume_changes.py

The consumer reads completed descriptors and paginates their datasets. It upserts baseline/add/change events into a local document table, writes pending and confirmed removals to a review table, clears review flags on cancellation, and applies each event with its deduplication record in one SQLite transaction. Replaying the same run is safe. CHANGE_BATCHES supports recovery when final OUTPUT is missing. Keep the producing run's default key-value store and each referenced dataset for as long as consumers need replay access.

Sitemap synchronization tracks publisher declarations, not page contents. A page may change while its sitemap URL and lastmod stay unchanged; use a separate content-crawling pipeline when content freshness matters. The five-million URL setting is a configurable safety ceiling, not a demonstrated capacity guarantee. Memory remains proportional to the current diff for this release.

Tips

• Use changesOnly for scheduled monitors to minimize result volume and cost.
• Leave page checks disabled unless HTTP status validation is required.
• Start with one site, inspect audit findings, and then expand a batch.
• A PARTIAL result intentionally preserves the last successful baseline.

FAQ and support

Does a lastmod change mean the page content changed?

No. URL_LASTMOD_CHANGED means only that the sitemap publisher changed its declared value.

Why is a missing URL initially pending?

The default requires two consecutive complete runs before confirming removal. This protects automations from temporary sitemap failures.

Is it safe to audit internal URLs?

No. The Actor rejects localhost, private, link-local, reserved, metadata-service, credential-bearing, non-HTTP(S), and nonstandard-port URLs, including redirect destinations.

Use only sites you are authorized to access and comply with applicable terms and laws. This Actor reads publicly accessible sitemap infrastructure and does not intentionally collect private user data. If output may contain personal data, ensure you have a legitimate basis to process it. For help, use the Actor's Issues tab; use the API tab for programmatic integration.