# Sitemap SEO Regression Monitor (`produkdigitalali/sitemap-seo-regression-monitor`) Actor

Discover XML sitemaps, audit page SEO signals, and monitor status, redirects, canonical, noindex, metadata, H1, and sitemap regressions across runs.

- **URL**: https://apify.com/produkdigitalali/sitemap-seo-regression-monitor.md
- **Developed by:** [ProdukDigitalAli](https://apify.com/produkdigitalali) (community)
- **Categories:** Developer tools, Automation, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 seo page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap SEO Regression Monitor

Discover public XML sitemaps, audit sitemap-listed pages for core technical SEO signals, and persist state so scheduled runs can emit only meaningful regressions and recoveries.

The Actor is designed for low-maintenance SEO regression monitoring without a browser, proxy, API key, or LLM. It uses bounded HTTP requests and sitemap scope rather than crawling an entire site graph.

### What it checks

For every audited sitemap URL the Actor can return:

- HTTP status and final URL
- redirect chain and response time
- content type / HTML detection
- page title
- meta description
- canonical URL
- meta robots and `X-Robots-Tag`
- `noindex` state
- first H1 and H1 count
- sitemap `lastmod`, `changefreq`, and `priority`
- optional normalized visible-content hash
- current SEO issue codes

Audit-mode issue codes include `NON_200_STATUS`, `REDIRECTED`, `NOINDEX`, `TITLE_MISSING`, `META_DESCRIPTION_MISSING`, `H1_MISSING`, `MULTIPLE_H1`, `CANONICAL_MISSING`, `CANONICAL_DIFFERENT`, `NON_HTML_CONTENT`, and `PAGE_FETCH_ERROR`.

### Regression monitoring

Enable `monitorMode` to compare the current sitemap/page state with a previous run stored in a named Apify key-value store.

Meaningful events include:

- `URL_ADDED`
- `URL_REMOVED_FROM_SITEMAP`
- `SEO_REGRESSION`
- `SEO_RECOVERY`
- `SEO_CHANGED`
- `PAGE_ERROR`
- `SITEMAP_ERROR`
- `SITEMAP_RECOVERED`
- `UNCHANGED`

Regression codes include high-value signals such as:

- `BECAME_NON_200`
- `BECAME_NOINDEX`
- `TITLE_REMOVED`
- `META_DESCRIPTION_REMOVED`
- `H1_REMOVED`
- `CANONICAL_REMOVED`
- `CANONICAL_DRIFT`
- `MULTIPLE_H1_INTRODUCED`
- `BECAME_NON_HTML`
- `REDIRECT_INTRODUCED`
- `PAGE_FETCH_ERROR`

Recoveries are classified separately, for example `RESTORED_200`, `NOINDEX_REMOVED`, `TITLE_RESTORED`, `REDIRECT_REMOVED`, `CANONICAL_SELF_REFERENCE_RESTORED`, and `PAGE_RECOVERED`.

#### Quiet first baseline

For scheduled monitoring, use:

```json
{
  "startUrls": ["https://fastapi.tiangolo.com/sitemap.xml"],
  "monitorMode": true,
  "emitChangesOnly": true,
  "baselineOnly": true,
  "monitorKey": "production-seo"
}
```

On the first successful baseline, existing pages are saved to state without flooding the dataset. Later runs emit changes. Keeping `baselineOnly: true` is safe: it only suppresses the first successful baseline for that state scope.

If the very first sitemap attempt fails, the later first successful page snapshot is still treated as a quiet baseline; a `SITEMAP_RECOVERED` status event may still be emitted.

### Removal safety

The Actor is deliberately conservative about removals.

`URL_REMOVED_FROM_SITEMAP` is emitted only when the current sitemap traversal is complete. If a sitemap traversal is truncated by configured limits or a nested sitemap partially fails, missing URLs are **not** treated as removals and their previous state is retained. This avoids false removal alerts caused by incomplete sitemap visibility.

Configuration that changes the observed URL/page semantics (URL filters, discovery window, content-hash mode, page-size/redirect bounds, and User-Agent) receives a separate monitoring state scope. This prevents configuration edits from looking like site regressions.

### Sitemap discovery

You can supply either a website root or a direct sitemap URL.

For website roots the Actor:

1. checks `robots.txt` for `Sitemap:` declarations;
2. otherwise tries common paths such as `/sitemap.xml`, `/sitemap_index.xml`, and `/wp-sitemap.xml`;
3. recursively follows sitemap indexes within configured depth/file limits.

Supported sitemap inputs include:

- XML `urlset`
- XML `sitemapindex`
- XML namespaces
- gzip sitemap payloads
- UTF-8 BOM and UTF-16 XML
- plain-text URL sitemaps
- common malformed bare ampersands / illegal XML 1.0 control characters when safely repairable

`robots.txt` is used for sitemap discovery. The Actor does not interpret robots allow/disallow rules as an authorization mechanism.

### Basic audit example

```json
{
  "startUrls": ["https://fastapi.tiangolo.com/sitemap.xml"],
  "maxUrlsPerSite": 100,
  "includeSiteSummary": true,
  "detectContentChanges": false,
  "concurrency": 10
}
```

Audit mode writes `PAGE_AUDIT` records plus an optional `SITE_SUMMARY`.

### Advanced site definitions

Use `sites` when you want a stable site ID or label:

```json
{
  "sites": [
    {
      "url": "https://example.com/sitemap.xml",
      "siteId": "example-production",
      "label": "Example production"
    }
  ],
  "monitorMode": true,
  "emitChangesOnly": true,
  "monitorKey": "daily"
}
```

A stable `siteId` lets a monitored input URL change without automatically changing the site's state identity.

### URL filtering

Large sitemaps can be narrowed using prefix and regex filters:

```json
{
  "startUrls": ["https://example.com/sitemap.xml"],
  "includeUrlPrefixes": ["https://example.com/products/"],
  "excludeUrlPrefixes": ["https://example.com/products/archive/"],
  "excludeUrlRegex": ["[?&]preview="]
}
```

Regex length and execution time are bounded to reduce ReDoS risk.

### Output records

The default dataset can contain four record types:

- `PAGE_AUDIT` — one audited page in normal audit mode
- `URL_CHANGE` — one page event in monitoring mode
- `SITE_STATUS` — sitemap failure/recovery event
- `SITE_SUMMARY` — optional site-level counts and discovery diagnostics

Run-level diagnostics are also stored in the default key-value store as `RUN_SUMMARY`.

Typical monitoring record:

```json
{
  "recordType": "URL_CHANGE",
  "status": "SUCCESS",
  "siteId": "example-production",
  "url": "https://example.com/pricing",
  "httpStatus": 200,
  "title": null,
  "canonical": "https://example.com/pricing",
  "noindex": false,
  "changeType": "SEO_REGRESSION",
  "changedFields": ["title"],
  "regressionCodes": ["TITLE_REMOVED"],
  "recoveryCodes": [],
  "severity": "WARNING"
}
```

### Content-change detection

Set `detectContentChanges: true` to hash normalized visible body text. Script, style, noscript, template, and SVG content is excluded from the visible-text hash.

Content hashing is optional because page-copy changes can be much noisier than technical SEO regressions.

### Safety and reliability

The Actor includes defensive controls for public web monitoring:

- localhost/private/link-local/reserved/metadata-address blocking
- redirect-target revalidation
- embedded URL credential rejection
- CR/LF User-Agent injection rejection
- bounded redirects, retries, response bytes, concurrency, sitemap depth, sitemap files, and URLs
- bounded streaming gzip decompression
- external DTD / XML entity declaration rejection
- full bounded-payload unsafe XML scan
- regex execution timeouts
- shared HTTP connection pool
- retry of transient HTTP 408/425/429/5xx responses, including page audits
- state is advanced only after dataset output persistence succeeds
- sitemap failure preserves previous page state
- page failure preserves the last healthy page snapshot

The public input schema rejects unknown fields, and the SSRF/public-network guard is not user-disableable.

### Important limitations

- Public HTTP(S) resources only; no private-network or authenticated sitemap/page access.
- No JavaScript rendering. SEO signals must be present in the HTTP response HTML/headers.
- The Actor audits URLs listed by the sitemap; it is not a general link crawler.
- Network/CDN behavior can differ by geography, User-Agent, or request timing.
- A sitemap that is intentionally incomplete cannot prove removal of URLs outside its visible scope.
- Real WAN throughput depends on target response times and rate limiting; local mock stress numbers are not Cloud throughput claims.

### Pricing

The Actor is designed for Apify pay-per-event pricing.

- `page-audited`: **$0.00050 per audited sitemap URL** ($0.50 / 1,000 URLs)
- `apify-actor-start`: configure the Apify synthetic start event at **$0.00005 per run**
- platform usage: recommended **Included** in the Actor price

`page-audited` is charged for every sitemap URL that is actually fetched and evaluated for HTTP and SEO signals, including non-200 responses and page-fetch-error observations. This is intentional for monitoring mode: an unchanged check still consumes the page audit and is the value being delivered. Sitemap-level failure/status records and site summaries do not add a separate custom event charge.

Do not also enable `apify-default-dataset-item`, because that would double-charge dataset output on top of the explicit page-audit event.

The Actor respects the run spending limit. Before page work it checks how many page-audit events remain chargeable, stops later work when the limit is exhausted, and preserves monitoring state for URLs that were not paid/audited. An interrupted quiet baseline is marked incomplete and resumes from unseen URLs on the next funded run instead of repeatedly charging the first sitemap entries.

#### Cloud benchmark sanity

Apify Cloud checks on 2026-08-24 against one public FastAPI sitemap, with `concurrency: 10`, produced:

| Audited URLs | Dataset results | Runtime | Displayed platform cost |
| ---: | ---: | ---: | ---: |
| 10 | 11 | ~4 s | ~$0.001 |
| 50 | 51 | ~4 s | ~$0.001 |
| 100 | 101 | ~7 s | ~$0.001 |

The cost shown in the Apify UI is rounded and these tests use one responsive host, so they are only a sanity benchmark, not a promise for every website.

### Validation status for v0.1.1

Before Cloud deployment, the packaged source was validated with:

- 208/208 `unittest` tests passing
- 208/208 `pytest` tests passing
- 96% selected source statement coverage (`src/main.py` 99%, `src/core.py` 96%)
- 5,000-case XML mutation fuzz: 0 unexpected exceptions
- 5,000-case HTML mutation fuzz: 0 unexpected exceptions
- 5,000-URL sitemap parse stress
- 2,000-page mock audit stress
- PPE audit billing tests, unchanged/baseline billing, zero-budget stop, partial-baseline continuation, and unpaid-regression state-preservation tests
- input/dataset/output project-schema checks
- pay-per-event configuration invariants
- recursive nested input `title` + `description` checks for Apify input-schema compatibility
- Python compile validation

See `VALIDATION_REPORT.md` for details and the distinction between local mock stress and real Apify Cloud performance.

### Responsible use

Audit only public websites you are authorized to access and use reasonable concurrency. Respect site terms, applicable law, and target infrastructure capacity.

# Actor input Schema

## `startUrls` (type: `array`):

Public website roots or direct sitemap URLs. Website roots use robots.txt declarations first, then common sitemap paths.

## `sites` (type: `array`):

Optional objects with url, stable siteId, and label. Use siteId when a monitored site's input URL may change while its state identity should remain stable.

## `maxSites` (type: `integer`):

Maximum unique site definitions processed in one run.

## `maxUrlsPerSite` (type: `integer`):

Hard cap on sitemap URLs audited per site. Large sitemaps are truncated at this limit.

## `maxSitemapsPerSite` (type: `integer`):

Maximum sitemap/index files followed recursively for one site.

## `maxSitemapDepth` (type: `integer`):

Maximum recursive sitemap-index depth.

## `maxSitemapBytes` (type: `integer`):

Maximum downloaded/uncompressed bytes accepted for one sitemap file.

## `includeUrlPrefixes` (type: `array`):

If provided, only sitemap URLs beginning with at least one prefix are audited.

## `excludeUrlPrefixes` (type: `array`):

Sitemap URLs beginning with any of these prefixes are skipped.

## `includeUrlRegex` (type: `array`):

Optional case-insensitive regular expressions. A URL must match at least one pattern. Patterns are length-limited and time-bounded.

## `excludeUrlRegex` (type: `array`):

Optional case-insensitive regular expressions used to skip matching sitemap URLs.

## `detectContentChanges` (type: `boolean`):

Hash normalized visible HTML text so content changes can be detected across monitoring runs. Disabled by default to focus on technical SEO signals.

## `includeSiteSummary` (type: `boolean`):

Emit a SITE\_SUMMARY record with discovery, issue, and change counts. Suppressed automatically in changes-only monitoring mode.

## `monitorMode` (type: `boolean`):

Persist bounded per-site snapshots and classify URL additions/removals, SEO regressions, recoveries, ordinary changes, and repeated page errors across runs.

## `emitChangesOnly` (type: `boolean`):

When monitoring, suppress UNCHANGED page records so scheduled runs return only meaningful changes and status events.

## `baselineOnly` (type: `boolean`):

On the first monitoring run for a site, save the current sitemap/page state without emitting every existing URL as URL\_ADDED.

## `monitorKey` (type: `string`):

Namespace for persistent monitoring state. Reuse the same value across related scheduled runs.

## `concurrency` (type: `integer`):

Maximum concurrent page audits across the run.

## `siteConcurrency` (type: `integer`):

Maximum site discovery pipelines processed concurrently.

## `requestTimeoutSeconds` (type: `integer`):

Per-request timeout in seconds.

## `maxRetries` (type: `integer`):

Retries transient timeout/network/429/5xx failures with bounded backoff.

## `maxRedirects` (type: `integer`):

Maximum redirect hops per sitemap/page request. Every target is revalidated before following it.

## `maxPageBytes` (type: `integer`):

Maximum response body bytes downloaded for one audited page.

## `userAgent` (type: `string`):

Optional HTTP User-Agent for public sitemap and page requests. CR/LF header injection is rejected.

## Actor input object example

```json
{
  "startUrls": [
    "https://fastapi.tiangolo.com/sitemap.xml"
  ],
  "sites": [],
  "maxSites": 20,
  "maxUrlsPerSite": 500,
  "maxSitemapsPerSite": 50,
  "maxSitemapDepth": 5,
  "maxSitemapBytes": 10485760,
  "includeUrlPrefixes": [],
  "excludeUrlPrefixes": [],
  "includeUrlRegex": [],
  "excludeUrlRegex": [],
  "detectContentChanges": false,
  "includeSiteSummary": true,
  "monitorMode": false,
  "emitChangesOnly": false,
  "baselineOnly": false,
  "monitorKey": "default",
  "concurrency": 10,
  "siteConcurrency": 3,
  "requestTimeoutSeconds": 20,
  "maxRetries": 2,
  "maxRedirects": 5,
  "maxPageBytes": 2097152,
  "userAgent": "Mozilla/5.0 (compatible; SitemapSEORegressionMonitor/0.1; +https://apify.com/)"
}
```

# Actor output Schema

## `dataset` (type: `string`):

Structured PAGE\_AUDIT, URL\_CHANGE, SITE\_STATUS, and SITE\_SUMMARY records.

## `runSummary` (type: `string`):

Site, sitemap, page, baseline, state, change, and processing error counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://fastapi.tiangolo.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("produkdigitalali/sitemap-seo-regression-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://fastapi.tiangolo.com/sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("produkdigitalali/sitemap-seo-regression-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://fastapi.tiangolo.com/sitemap.xml"
  ]
}' |
apify call produkdigitalali/sitemap-seo-regression-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,produkdigitalali/sitemap-seo-regression-monitor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/axM493Fs9E6aUv3Pa/builds/YXmzylfFOTBjH5pmd/openapi.json