Sitemap Monitor & Integrity Auditor

What does Sitemap Monitor & Integrity Auditor do?

Sitemap Monitor & Integrity Auditor is an HTTP-only sitemap audit and change-monitoring API. It discovers sitemap declarations through robots.txt, walks sitemap indexes, validates XML, gzip, and text sitemaps, and reports trustworthy URL inventory changes without crawling every listed page by default.

It distinguishes publisher-declared <lastmod> changes from actual page-content changes and never turns an incomplete sitemap tree into a wave of false URL removals.

Why use Sitemap Monitor & Integrity Auditor?

• Detect missing, broken, malformed, oversized, or circular sitemap infrastructure.
• Track new URLs, declared lastmod changes, and conservatively confirmed removals.
• Reuse unchanged sitemap state with ETag, Last-Modified, and HTTP 304 responses.
• Feed crawler queues, RAG ingestion, migration checks, Make, n8n, Zapier, or webhooks.
• Schedule monitors and consume results through the Apify API.
• Process large inventories with streaming XML parsing, SQLite working state, and sharded persistent storage.

The Actor does not use a browser, proxy, CAPTCHA service, or AI. It only fetches pages listed in a sitemap when page-status checking is explicitly enabled.

What data can it return?

Record type	What it contains
site_summary	Overall status, URL counts, issues, requests, bytes, and duration
url_observation	Current URL inventory or changes such as URL_ADDED
sitemap_observation	HTTP validators, hashes, sizes, reuse, and parse status
audit_finding	Deterministic issue code, severity, resource, and explanation

Every input site ends as SUCCESS, PARTIAL, or FAILED. Only a complete SUCCESS run can advance monitor state.

How much does sitemap monitoring cost?

The pay-per-event model charges $0.02 per site check, $0.0001 per delivered URL record, and $0.0001 per optional page check. An unchanged scheduled monitor still performs and charges for a site check; a changesOnly run does not pay for URLs it does not deliver. Actual platform pricing appears in the Actor's Pricing tab.

Input

See the Input tab for every safety limit and timeout. A snapshot:

{
  "sites": [{"url": "https://example.com", "externalId": "main-site"}],
  "mode": "snapshot",
  "outputMode": "all"
}

A scheduled monitor:

{
  "sites": [{"url": "https://example.com", "monitorKey": "production"}],
  "mode": "monitor",
  "outputMode": "changesOnly"
}

Changing include or exclude patterns creates a new baseline. Do not run the same monitorKey concurrently.

Output

You can download the dataset in JSON, HTML, CSV, or Excel. The default key-value store also contains an OUTPUT summary for automation conditions.

{
  "recordType": "url_observation",
  "changeType": "URL_ADDED",
  "url": "https://example.com/new-page",
  "currentLastmod": "2026-09-09"
}

Tips

• Use changesOnly for scheduled monitors to minimize result volume and cost.
• Leave page checks disabled unless HTTP status validation is required.
• Start with one site, inspect audit findings, and then expand a batch.
• A PARTIAL result intentionally preserves the last successful baseline.

FAQ and support

Does a lastmod change mean the page content changed?

No. URL_LASTMOD_CHANGED means only that the sitemap publisher changed its declared value.

Why is a missing URL initially pending?

The default requires two consecutive complete runs before confirming removal. This protects automations from temporary sitemap failures.

Is it safe to audit internal URLs?

No. The Actor rejects localhost, private, link-local, reserved, metadata-service, credential-bearing, non-HTTP(S), and nonstandard-port URLs, including redirect destinations.

Use only sites you are authorized to access and comply with applicable terms and laws. This Actor reads publicly accessible sitemap infrastructure and does not intentionally collect private user data. If output may contain personal data, ensure you have a legitimate basis to process it. For help, use the Actor's Issues tab; use the API tab for programmatic integration.