Sitemap SEO Regression Monitor

Discover public XML sitemaps, audit sitemap-listed pages for core technical SEO signals, and persist state so scheduled runs can emit only meaningful regressions and recoveries.

The Actor is designed for low-maintenance SEO regression monitoring without a browser, proxy, API key, or LLM. It uses bounded HTTP requests and sitemap scope rather than crawling an entire site graph.

What it checks

For every audited sitemap URL the Actor can return:

• HTTP status and final URL
• redirect chain and response time
• content type / HTML detection
• page title
• meta description
• canonical URL
• meta robots and X-Robots-Tag
• noindex state
• first H1 and H1 count
• sitemap lastmod, changefreq, and priority
• optional normalized visible-content hash
• current SEO issue codes

Audit-mode issue codes include NON_200_STATUS, REDIRECTED, NOINDEX, TITLE_MISSING, META_DESCRIPTION_MISSING, H1_MISSING, MULTIPLE_H1, CANONICAL_MISSING, CANONICAL_DIFFERENT, NON_HTML_CONTENT, and PAGE_FETCH_ERROR.

Regression monitoring

Enable monitorMode to compare the current sitemap/page state with a previous run stored in a named Apify key-value store.

Meaningful events include:

• URL_ADDED
• URL_REMOVED_FROM_SITEMAP
• SEO_REGRESSION
• SEO_RECOVERY
• SEO_CHANGED
• PAGE_ERROR
• SITEMAP_ERROR
• SITEMAP_RECOVERED
• UNCHANGED

Regression codes include high-value signals such as:

• BECAME_NON_200
• BECAME_NOINDEX
• TITLE_REMOVED
• META_DESCRIPTION_REMOVED
• H1_REMOVED
• CANONICAL_REMOVED
• CANONICAL_DRIFT
• MULTIPLE_H1_INTRODUCED
• BECAME_NON_HTML
• REDIRECT_INTRODUCED
• PAGE_FETCH_ERROR

Recoveries are classified separately, for example RESTORED_200, NOINDEX_REMOVED, TITLE_RESTORED, REDIRECT_REMOVED, CANONICAL_SELF_REFERENCE_RESTORED, and PAGE_RECOVERED.

Quiet first baseline

For scheduled monitoring, use:

{
  "startUrls": ["https://fastapi.tiangolo.com/sitemap.xml"],
  "monitorMode": true,
  "emitChangesOnly": true,
  "baselineOnly": true,
  "monitorKey": "production-seo"
}

On the first successful baseline, existing pages are saved to state without flooding the dataset. Later runs emit changes. Keeping baselineOnly: true is safe: it only suppresses the first successful baseline for that state scope.

If the very first sitemap attempt fails, the later first successful page snapshot is still treated as a quiet baseline; a SITEMAP_RECOVERED status event may still be emitted.

Removal safety

The Actor is deliberately conservative about removals.

URL_REMOVED_FROM_SITEMAP is emitted only when the current sitemap traversal is complete. If a sitemap traversal is truncated by configured limits or a nested sitemap partially fails, missing URLs are not treated as removals and their previous state is retained. This avoids false removal alerts caused by incomplete sitemap visibility.

Configuration that changes the observed URL/page semantics (URL filters, discovery window, content-hash mode, page-size/redirect bounds, and User-Agent) receives a separate monitoring state scope. This prevents configuration edits from looking like site regressions.

Sitemap discovery

You can supply either a website root or a direct sitemap URL.

For website roots the Actor:

1. checks robots.txt for Sitemap: declarations;
2. otherwise tries common paths such as /sitemap.xml, /sitemap_index.xml, and /wp-sitemap.xml;
3. recursively follows sitemap indexes within configured depth/file limits.

Supported sitemap inputs include:

• XML urlset
• XML sitemapindex
• XML namespaces
• gzip sitemap payloads
• UTF-8 BOM and UTF-16 XML
• plain-text URL sitemaps
• common malformed bare ampersands / illegal XML 1.0 control characters when safely repairable

robots.txt is used for sitemap discovery. The Actor does not interpret robots allow/disallow rules as an authorization mechanism.

Basic audit example

{
  "startUrls": ["https://fastapi.tiangolo.com/sitemap.xml"],
  "maxUrlsPerSite": 100,
  "includeSiteSummary": true,
  "detectContentChanges": false,
  "concurrency": 10
}

Audit mode writes PAGE_AUDIT records plus an optional SITE_SUMMARY.

Advanced site definitions

Use sites when you want a stable site ID or label:

{
  "sites": [
    {
      "url": "https://example.com/sitemap.xml",
      "siteId": "example-production",
      "label": "Example production"
    }
  ],
  "monitorMode": true,
  "emitChangesOnly": true,
  "monitorKey": "daily"
}

A stable siteId lets a monitored input URL change without automatically changing the site's state identity.

URL filtering

Large sitemaps can be narrowed using prefix and regex filters:

{
  "startUrls": ["https://example.com/sitemap.xml"],
  "includeUrlPrefixes": ["https://example.com/products/"],
  "excludeUrlPrefixes": ["https://example.com/products/archive/"],
  "excludeUrlRegex": ["[?&]preview="]
}

Regex length and execution time are bounded to reduce ReDoS risk.

Output records

The default dataset can contain four record types:

• PAGE_AUDIT — one audited page in normal audit mode
• URL_CHANGE — one page event in monitoring mode
• SITE_STATUS — sitemap failure/recovery event
• SITE_SUMMARY — optional site-level counts and discovery diagnostics

Run-level diagnostics are also stored in the default key-value store as RUN_SUMMARY.

Typical monitoring record:

{
  "recordType": "URL_CHANGE",
  "status": "SUCCESS",
  "siteId": "example-production",
  "url": "https://example.com/pricing",
  "httpStatus": 200,
  "title": null,
  "canonical": "https://example.com/pricing",
  "noindex": false,
  "changeType": "SEO_REGRESSION",
  "changedFields": ["title"],
  "regressionCodes": ["TITLE_REMOVED"],
  "recoveryCodes": [],
  "severity": "WARNING"
}

Content-change detection

Set detectContentChanges: true to hash normalized visible body text. Script, style, noscript, template, and SVG content is excluded from the visible-text hash.

Content hashing is optional because page-copy changes can be much noisier than technical SEO regressions.

Safety and reliability

The Actor includes defensive controls for public web monitoring:

• localhost/private/link-local/reserved/metadata-address blocking
• redirect-target revalidation
• embedded URL credential rejection
• CR/LF User-Agent injection rejection
• bounded redirects, retries, response bytes, concurrency, sitemap depth, sitemap files, and URLs
• bounded streaming gzip decompression
• external DTD / XML entity declaration rejection
• full bounded-payload unsafe XML scan
• regex execution timeouts
• shared HTTP connection pool
• retry of transient HTTP 408/425/429/5xx responses, including page audits
• state is advanced only after dataset output persistence succeeds
• sitemap failure preserves previous page state
• page failure preserves the last healthy page snapshot

The public input schema rejects unknown fields, and the SSRF/public-network guard is not user-disableable.

Important limitations

• Public HTTP(S) resources only; no private-network or authenticated sitemap/page access.
• No JavaScript rendering. SEO signals must be present in the HTTP response HTML/headers.
• The Actor audits URLs listed by the sitemap; it is not a general link crawler.
• Network/CDN behavior can differ by geography, User-Agent, or request timing.
• A sitemap that is intentionally incomplete cannot prove removal of URLs outside its visible scope.
• Real WAN throughput depends on target response times and rate limiting; local mock stress numbers are not Cloud throughput claims.

Pricing

The Actor is designed for Apify pay-per-event pricing.

• page-audited: $0.00050 per audited sitemap URL ($0.50 / 1,000 URLs)
• apify-actor-start: configure the Apify synthetic start event at $0.00005 per run
• platform usage: recommended Included in the Actor price

page-audited is charged for every sitemap URL that is actually fetched and evaluated for HTTP and SEO signals, including non-200 responses and page-fetch-error observations. This is intentional for monitoring mode: an unchanged check still consumes the page audit and is the value being delivered. Sitemap-level failure/status records and site summaries do not add a separate custom event charge.

Do not also enable apify-default-dataset-item, because that would double-charge dataset output on top of the explicit page-audit event.

The Actor respects the run spending limit. Before page work it checks how many page-audit events remain chargeable, stops later work when the limit is exhausted, and preserves monitoring state for URLs that were not paid/audited. An interrupted quiet baseline is marked incomplete and resumes from unseen URLs on the next funded run instead of repeatedly charging the first sitemap entries.

Cloud benchmark sanity

Apify Cloud checks on 2026-08-24 against one public FastAPI sitemap, with concurrency: 10, produced:

Audited URLs	Dataset results	Runtime	Displayed platform cost
10	11	~4 s	~$0.001
50	51	~4 s	~$0.001
100	101	~7 s	~$0.001

The cost shown in the Apify UI is rounded and these tests use one responsive host, so they are only a sanity benchmark, not a promise for every website.

Validation status for v0.1.1

Before Cloud deployment, the packaged source was validated with:

• 208/208 unittest tests passing
• 208/208 pytest tests passing
• 96% selected source statement coverage (src/main.py 99%, src/core.py 96%)
• 5,000-case XML mutation fuzz: 0 unexpected exceptions
• 5,000-case HTML mutation fuzz: 0 unexpected exceptions
• 5,000-URL sitemap parse stress
• 2,000-page mock audit stress
• PPE audit billing tests, unchanged/baseline billing, zero-budget stop, partial-baseline continuation, and unpaid-regression state-preservation tests
• input/dataset/output project-schema checks
• pay-per-event configuration invariants
• recursive nested input title + description checks for Apify input-schema compatibility
• Python compile validation

See VALIDATION_REPORT.md for details and the distinction between local mock stress and real Apify Cloud performance.

Responsible use

Audit only public websites you are authorized to access and use reasonable concurrency. Respect site terms, applicable law, and target infrastructure capacity.