# Sitemap Health Monitor: Broken Links, 404 & Noindex Checker (`pankovs/sitemap-health-monitor`) Actor Scheduled SEO audit of every URL in your sitemap: broken links, 404s, 5xx errors, redirect chains, noindex, canonical, robots.txt and hreflang conflicts. Each run diffs against the last one and reports new, still-broken and fixed issues, with a webhook alert on new ones. - **URL**: https://apify.com/pankovs/sitemap-health-monitor.md - **Developed by:** [Pankovs](https://apify.com/pankovs) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $0.50 / 1,000 url checkeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Sitemap Health Monitor: Broken Link, 404 & Noindex Checker with Scheduled Diff Alerts **A sitemap-focused technical SEO audit and link checker: find broken links, 404 pages, 5xx server errors and redirect chains on every URL in your sitemap.** Point it at your site (or your sitemap), and it checks each listed page: HTTP status, redirect hops, timeouts. It also catches the SEO problems a plain broken link checker misses: pages that are `noindex`, whose `rel=canonical` points somewhere else, that your own `robots.txt` blocks, or whose `hreflang` language versions don't link back to each other. What makes it different is the **scheduled diff**. The monitor remembers each run, so the next run tells you **what broke since last time, what is still broken, and what got fixed**. Schedule it daily or weekly and get a webhook ping (Slack, Zapier, Make) only when something new goes wrong. You pay $0.50 per 1,000 URLs checked, with no start fee. By **Pankovs**. Contact: ### Use cases - **Broken link checker:** find broken links, 4xx client errors and 5xx server errors across a site's sitemap. - **Indexability check:** catch pages listed in the sitemap that are `noindex`, canonicalised elsewhere or blocked by `robots.txt`. - **Post-deploy / migration QA:** run it right after a release or a site migration and see exactly what broke. - **Scheduled SEO monitoring:** replace manual Screaming Frog "Compare" runs or ContentKing-style continuous monitoring with a cheap, scheduled, pay-per-URL SEO audit. No subscription, no seat limits. - **International SEO:** catch broken `hreflang` pairs on multi-language sites: a language version that doesn't link back, returns 404, redirects, or is missing from the sitemap. - **Agencies:** one saved task per client site, each with its own webhook alert. Ready-made example tasks (Examples tab): quick one-off scan, scheduled monitor with a Slack/Zapier alert, pre-migration / post-deploy QA, and a gentle multi-client check for small shared-hosting sites. ### What it checks For every URL listed in your sitemap(s): | Check | Reported as issue | |---|---| | HTTP status | `client-error` (4xx), `server-error` (5xx) | | Redirects | `redirect-chain`: more than 1 hop before the final page (or a redirect loop) | | Indexability | `noindex`: page in the sitemap but blocked by `` or `X-Robots-Tag` | | Canonical | `canonical-mismatch`: the page's `rel=canonical` points to a different URL | | Crawlability | `robots-disallow`: page in the sitemap but blocked by your `robots.txt` (`User-agent: *` rules) | | International | `hreflang-mismatch`: a `` pair is broken (see below) | | Availability | `timeout` (no response within the timeout), `fetch-error` (DNS, TLS, connection refused) | It also records final URL after redirects, the number of redirect hops, and response time. `robots.txt` is read once per host and applied the way Google does it: rules of the `User-agent: *` group(s), the longest matching `Allow`/`Disallow` wins (`Allow` wins a tie), `*` wildcards and `$` end anchors are supported. The issue's `detail` names the rule, e.g. `robots.txt Disallow: /private/ (User-agent: *)`. A missing `robots.txt` (4xx) means everything is allowed. If `robots.txt` answers 5xx, 429 or times out, the run adds a `robots-unavailable` notice and leaves earlier `robots-disallow` issues unchanged (not resolved). Disallowed URLs are still checked for status, redirects and the rest, and cost the same as any other URL. `hreflang` is read from each page's `` tags. For every language version a page points to, the monitor checks that the version answers HTTP 200 without a redirect, links back to the page (reciprocal pair), and, if it is on the same host, is listed in your sitemap. One issue per page lists every broken pair as ` -> (): `, e.g. `en -> de (https://www.example.com/de/): no return link`. Problems: `no return link`, `HTTP 404` (any non-2xx status), `redirects to `, `not in the sitemap`, `invalid URL`. Versions already checked in this run are reused; others are fetched once (at most one extra request per checked URL, 500 per run) and are **not charged**. If a version can't be verified (timeout, rate limit, bot protection, or the per-run limit, which adds a `hreflang-partial` notice), the page keeps its previous `hreflang-mismatch` state instead of a false new or resolved issue. Not read: hreflang in HTTP `Link:` headers or in sitemap `` entries. Canonical comparison ignores tracking parameters (`utm_*`, `fbclid`, `gclid`, `msclkid`, `mc_cid`, `mc_eid`, `redirect_source`), host letter case, default ports and `#fragments`. Path case is significant (`/Blog` and `/blog` are different URLs). **Not reported as issues:** HTTP 429 (rate limiting) and bot-protection / challenge pages. A server that throttles the monitor is not a broken page; see [Rate limiting](#rate-limiting) and [Cloudflare / WAF allowlist](#cloudflare--waf-allowlist). Sitemap support: `robots.txt` discovery (`Sitemap:` lines, absolute or relative), sitemap index files (nested), gzipped `.xml.gz`, plain-text sitemaps, and namespace-prefixed XML (`` / ``). Discovery always starts at the host root: for `https://example.com/news/` it reads `https://example.com/robots.txt` and says so in a `discovery-root` notice. To check one section only, set `sitemapUrls` to that section's sitemap. If `robots.txt` lists no sitemap, the actor probes `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml` and `/sitemap.xml.gz`. If none exists, the run ends with a `no-sitemap` notice: set `sitemapUrls` to your sitemap's address. Discovery is capped at 120 s in total. If `robots.txt` / the probe paths time out or drop the connection, the run ends with a `sitemap-timeout` notice listing those URLs instead (the site may still have a sitemap). Sitemap downloads get a 60-second timeout and one retry. A sitemap that can't be read (HTTP 404/5xx, blocked, not XML, too large) or that contains no URLs is never silent: the run adds a `sitemap-error` notice naming each failing sitemap with its HTTP status or reason, lists them in `sitemapErrors`, and the run's status message says how many sitemaps failed. Issues on URLs listed only in a failing sitemap are kept as `unchecked`, not marked resolved. ### Input | Field | Type | Default | Description | |---|---|---|---| | `siteUrl` | string | none | Your homepage, e.g. `example.com` or `https://www.example.com` (`https://` is added when missing). Sitemaps are discovered from the host's `/robots.txt`. | | `sitemapUrls` | string\[] | `[]` | Explicit sitemap / sitemap index URLs. Overrides discovery. | | `maxUrls` | integer | 500 | Max URLs checked per run (1–10,000). | | `notifyWebhookUrl` | string | none | Send an alert here when there are **new** issues or a sitemap **starts failing**. | | `webhookFormat` | string | `auto` | `auto`: a readable message for Slack URLs (`hooks.slack.com`), the summary JSON for any other URL. `slack`: always a chat message `{"text": ...}` (Mattermost, Rocket.Chat, Discord with `/slack` at the end of its webhook URL). `json`: always the summary JSON. | | `concurrency` | integer | 5 | Parallel checks (1–20). | | `hostDelayMs` | integer | 250 | Minimum gap between requests to the same host, so your server isn't strained. Raised automatically when the server answers 429. | | `timeoutSecs` | integer | 15 | Per-request timeout (1–60). | ```json { "siteUrl": "https://www.example.com", "maxUrls": 1000, "notifyWebhookUrl": "https://hooks.slack.com/services/XXX/YYY/ZZZ" } ``` ### Output The dataset holds **only issues** (not every healthy URL) plus **one summary record** per run. Issue record: ```json { "recordType": "issue", "change": "new", "issue": "client-error", "url": "https://www.example.com/old-pricing", "status": 404, "finalUrl": "https://www.example.com/old-pricing", "redirects": 0, "responseMs": 182, "detail": "HTTP 404", "firstSeenAt": "2026-10-09T06:00:04.112Z", "checkedAt": "2026-10-09T06:00:04.112Z", "resolution": null } ``` - `change`: `new` (first seen this run), `persisting` (also present last run), `resolved` (was present last run, gone now). - `resolution` (resolved rows only): `fixed` (URL re-checked and healthy) or `removed-from-sitemap`. - `firstSeenAt`: when the issue was first detected, so you can see how long it has been open. Summary record (also saved as `OUTPUT` in the run's key-value store): ```json { "recordType": "summary", "site": "www.example.com", "checkedAt": "2026-10-09T06:00:04.112Z", "previousCheckedAt": "2026-10-08T06:00:03.954Z", "sitemaps": ["https://www.example.com/sitemap.xml"], "sitemapErrors": [], "newSitemapErrors": [], "discoveryComplete": true, "urlsDiscovered": 812, "urlsChecked": 810, "urlsRateLimited": 2, "urlsBlocked": 0, "siteBlock": null, "chargeLimitReached": false, "issues": { "new": 1, "persisting": 3, "resolved": 1, "open": 4, "unchecked": 0 }, "openByType": { "redirect-chain": 1, "noindex": 1, "canonical-mismatch": 1, "server-error": 1 }, "newIssues": [{ "url": "https://www.example.com/blog", "issue": "server-error", "detail": "HTTP 500" }], "notices": [{ "code": "rate-limited", "message": "2 URLs kept returning HTTP 429 after 3 retries ..." }], "webhook": { "status": 200, "error": null } } ``` - `sitemapErrors`: every sitemap that failed this run, as `{ "sitemap", "error", "block" }`, e.g. `{ "sitemap": "https://www.example.com/news-sitemap.xml", "error": "Sitemap returned HTTP 404", "block": null }`. `newSitemapErrors`: the ones that were not failing in the previous run (these trigger the webhook). - `urlsChecked`: URLs actually checked (and charged) this run. - `urlsRateLimited`: URLs that still got HTTP 429 after retries. Not checked, not charged. - `urlsBlocked`: URLs refused by bot protection (challenge pages) or, when the whole site blocks the monitor, by 401/403. Not checked, not charged. - `siteBlock`: `null`, or why the whole site refused the monitor (`auth-required`, `forbidden`, `bot-protection`, `rate-limited`). - `issues.open`: open issues on URLs checked **this run** (never more than the URLs checked here). `issues.unchecked`: earlier issues on URLs not checked this run. They are kept for the next run and are neither counted as open nor marked resolved. - `notices`: run-level messages. `site-blocked` (site protection refused the monitor), `rate-limited`, `no-sitemap` (set `sitemapUrls`), `sitemap-timeout` (discovery URLs did not respond within 120 s), `robots-unavailable` (a host's `robots.txt` returned 5xx/429 or timed out, so its rules were not applied), `hreflang-partial` (some hreflang language versions outside this run were not fetched; see above), `sitemap-error` (a sitemap failed or has no URLs; names each one with its HTTP status or reason), `discovery-root` (`siteUrl` has a path, but discovery uses the host root `robots.txt`), `charge-limit` (the run's maximum charge was reached: "checked N of M URLs"; raise the maximum cost per run to check them all). Issue types in `issue` / `openByType`: `client-error`, `server-error`, `redirect-chain`, `noindex`, `canonical-mismatch`, `robots-disallow`, `hreflang-mismatch`, `timeout`, `fetch-error`. A `robots-disallow` row looks like: ```json { "recordType": "issue", "change": "new", "issue": "robots-disallow", "url": "https://www.example.com/private/report", "status": 200, "detail": "robots.txt Disallow: /private/ (User-agent: *)" } ``` A Slack incoming webhook (`hooks.slack.com`) gets a readable message: the site, the new, open and resolved counts, up to 10 new issues with their URL and reason, and each sitemap that started failing. Every other webhook URL (Zapier, Make, n8n, your own endpoint) receives the same summary object as its JSON body; set `webhookFormat` to choose explicitly. The alert is sent when a run finds new issues (`issues.new > 0`) or when a sitemap starts failing (`newSitemapErrors` is not empty). A sitemap that keeps failing alerts once, not on every run; if it recovers and breaks again, you get a new alert. The sitemap health is stored in the same snapshot as the issues. ### Recurring monitoring (Apify Schedules) The actor keeps a snapshot of open issues per site and sitemap scope (named key-value store `sitemap-health-monitor`), so every run diffs against the previous one automatically. Runs that find the sitemap on their own use key `snapshot-`. Runs with an explicit `sitemapUrls` list use `snapshot-!`, where the hash is stable for the same set of sitemap URLs (order, duplicates and surrounding spaces don't matter). So a task that checks only `news-sitemap.xml` keeps its own baseline and never marks the full-site task's issues as resolved, and the full-site task never sees them as new again. The first run of a new `sitemapUrls` selection starts from the site's earlier results for the pages in those sitemaps, so known issues stay `persisting` instead of coming back as `new`. To monitor: 1. Open the actor and click **Save as a new task** with your input (site URL and webhook). 2. Go to **Schedules**, click **Create new**, pick a cadence (e.g. daily at 06:00), and add the task. 3. Done. Each run produces only the changes; the webhook fires only when there are new issues or a sitemap stops working. Use one task per site. The first run reports all current issues as `new` (it has no baseline yet). The diff only compares URLs checked in both runs. If a run checks fewer URLs (lower `maxUrls`, charge limit, rate limiting, site protection, or a sitemap couldn't be read), issues on the URLs it skipped are counted in `issues.unchecked`. They are never marked resolved by mistake, and when those URLs are checked again they come back as `persisting`, not `new`. URL selection follows sitemap order, so repeated runs with the same `maxUrls` check the same URLs. ### How this differs from a one-off crawl A one-off site audit tells you everything that is wrong today, and next week you get the same list again and have to work out what changed yourself. This actor is built for SEO monitoring: it keeps the previous run's results, so each report is a diff (new, still broken, fixed), and the alert fires only on new problems. It checks the URLs you actually want indexed (your sitemap), not every link a crawler can reach, which keeps runs fast and cheap enough to schedule daily. ### Rate limiting Some servers (Shopify stores in particular) answer HTTP 429 "Too Many Requests" when crawled. The monitor then: 1. Waits as long as the `Retry-After` header asks (up to 30 s; exponential backoff if the header is missing) and retries up to 3 times. 2. Slows down for that host for the rest of the run: the gap between requests doubles on every 429, up to 3 seconds. 3. If a URL still gets 429, it is reported in `urlsRateLimited` and a `rate-limited` notice. It is **not** reported as a broken page, **not** charged, and keeps its previous state. If many URLs are rate-limited, lower `concurrency` (e.g. 2), raise `hostDelayMs` (e.g. 1000), or allowlist the monitor (below). ### Cloudflare / WAF allowlist The monitor never tries to bypass bot protection, CAPTCHAs or challenge pages. If most of the first checks (at least 80% of the first 10) are refused with 401/403/429 or a challenge page, or the sitemap itself is behind a challenge, the run stops early with one `site-blocked` notice instead of hundreds of false errors. Nothing is charged for the refused URLs and earlier results are left as they were. To monitor a site behind a firewall, allowlist the monitor by User-Agent (`SitemapHealthMonitor`). Requests come from Apify's cloud IPs, which change, so allowlisting by IP won't work. - **Cloudflare:** open Security, then WAF, Custom rules, Create rule. Field `User Agent`, operator `contains`, value `SitemapHealthMonitor`. Action **Skip**: tick all remaining custom rules, rate limiting rules, managed rules and Super Bot Fight Mode. If you use Bot Fight Mode (free plan), it can't be skipped by a rule, so turn it off or put the sitemap URLs on an unprotected path. - **Other WAFs / CDNs** (Akamai, Fastly, Sucuri, Imperva, AWS WAF): add an allow rule matching the User-Agent substring `SitemapHealthMonitor`, placed before bot-management and rate-limit rules. - **Login-protected sites (401):** the monitor can't log in. Monitor the public part of the site, or put a public sitemap in `sitemapUrls`. Treat a User-Agent allow rule as a convenience, not as security, because anyone can send that User-Agent. Keep the rule limited to skipping bot checks and rate limits, not authentication. ### Pricing (pay per event) | Event | Price | |---|---| | `url-checked`: one sitemap URL checked | **$0.0005** ($0.50 per 1,000 URLs) | No start fee. A daily check of a 500-URL site costs about **$0.25/day**. URLs that were rate-limited or refused by site protection are not charged. The actor respects your run's maximum charge: it stops checking new URLs once the limit is reached and still saves the summary and snapshot. The hreflang check adds no event: language versions fetched to verify a pair are not charged. ### Safety The actor only requests `http(s)` URLs on public addresses. Private, loopback, link-local and cloud metadata IPs are blocked, including hostnames that resolve to them, redirects that point to them and IPv6 spellings of them (IPv4-mapped such as `[::ffff:127.0.0.1]`, NAT64, 6to4); only public IPv6 (global unicast) is allowed. This applies to the webhook URL too. It sends a clear User-Agent (`SitemapHealthMonitor/0.1; +https://pankovs.com`) and spaces requests per host. ### Local development ```bash npm install npm run typecheck && npm run lint && npm test npm run smoke # build + run dist/main.js twice against a local fixture site, asserts the diff ``` Run the built actor locally against any public site: ```bash npm run build mkdir -p storage/key_value_stores/default echo '{"siteUrl":"https://www.sitemaps.org","maxUrls":30}' > storage/key_value_stores/default/INPUT.json CRAWLEE_STORAGE_DIR=./storage node dist/main.js python scripts/show-run.py storage ``` `SHM_ALLOW_PRIVATE_HOSTS=1` lifts the private-IP block for local fixture testing only; it is ignored when running on the Apify platform. ### More Real output, prices and changelog: [Sitemap Health Monitor, broken links and 404 alerts](https://pankovs.com/sitemap-health-monitor). # Changelog This Actor's version history is a separate document: https://apify.com/pankovs/sitemap-health-monitor/changelog.md # Actor input Schema ## `siteUrl` (type: `string`): Your site's homepage, e.g. example.com or https://www.example.com (https:// is added when missing). Sitemaps are discovered from the host's /robots.txt (Sitemap: lines), falling back to /sitemap.xml. Ignored when Sitemap URLs are set. ## `sitemapUrls` (type: `array`): Explicit sitemap or sitemap-index URLs (.xml, .xml.gz or plain-text). Overrides discovery from Website URL. ## `maxUrls` (type: `integer`): Stop after this many sitemap URLs (each checked URL is one billable event). ## `notifyWebhookUrl` (type: `string`): Optional. When a run finds NEW issues or a sitemap starts failing, an alert is POSTed here. A Slack incoming webhook (hooks.slack.com) gets a readable message; Zapier, Make, n8n or your own endpoint get the full summary record as JSON. See Webhook format. ## `webhookFormat` (type: `string`): Auto: a readable chat message for Slack URLs (hooks.slack.com), the full summary JSON for every other URL. Chat message: always send {"text": ...}, for Slack-compatible webhooks on other hosts (Mattermost, Rocket.Chat, Discord with /slack at the end of the URL). Summary JSON: always send the full summary record. ## `concurrency` (type: `integer`): Parallel URL checks. ## `hostDelayMs` (type: `integer`): Minimum spacing between requests to one host, so monitoring never strains your server. ## `timeoutSecs` (type: `integer`): A URL that does not respond in time is reported as a timeout issue. ## Actor input object example ```json { "siteUrl": "https://www.sitemaps.org", "sitemapUrls": [], "maxUrls": 500, "webhookFormat": "auto", "concurrency": 5, "hostDelayMs": 250, "timeoutSecs": 15 } ``` # Actor output Schema ## `issues` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "siteUrl": "https://www.sitemaps.org" }; // Run the Actor and wait for it to finish const run = await client.actor("pankovs/sitemap-health-monitor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "siteUrl": "https://www.sitemaps.org" } # Run the Actor and wait for it to finish run = client.actor("pankovs/sitemap-health-monitor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "siteUrl": "https://www.sitemaps.org" }' | apify call pankovs/sitemap-health-monitor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,pankovs/sitemap-health-monitor" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/Bfb63qdGtUSuRkQd7/builds/BQ9bSN2TQJfN6cPAw/openapi.json