Sitemap Health Monitor: Broken Link, 404 & Noindex Checker with Scheduled Diff Alerts

A sitemap-focused technical SEO audit and link checker: find broken links, 404 pages, 5xx server errors and redirect chains on every URL in your sitemap. Point it at your site (or your sitemap), and it checks each listed page: HTTP status, redirect hops, timeouts. It also catches the SEO problems a plain broken link checker misses: pages that are noindex, whose rel=canonical points somewhere else, that your own robots.txt blocks, or whose hreflang language versions don't link back to each other.

What makes it different is the scheduled diff. The monitor remembers each run, so the next run tells you what broke since last time, what is still broken, and what got fixed. Schedule it daily or weekly and get a webhook ping (Slack, Zapier, Make) only when something new goes wrong. You pay $0.50 per 1,000 URLs checked, with no start fee.

By Pankovs. Contact: info@pankovs.com

Use cases

• Broken link checker: find broken links, 4xx client errors and 5xx server errors across a site's sitemap.
• Indexability check: catch pages listed in the sitemap that are noindex, canonicalised elsewhere or blocked by robots.txt.
• Post-deploy / migration QA: run it right after a release or a site migration and see exactly what broke.
• Scheduled SEO monitoring: replace manual Screaming Frog "Compare" runs or ContentKing-style continuous monitoring with a cheap, scheduled, pay-per-URL SEO audit. No subscription, no seat limits.
• International SEO: catch broken hreflang pairs on multi-language sites: a language version that doesn't link back, returns 404, redirects, or is missing from the sitemap.
• Agencies: one saved task per client site, each with its own webhook alert.

Ready-made example tasks (Examples tab): quick one-off scan, scheduled monitor with a Slack/Zapier alert, pre-migration / post-deploy QA, and a gentle multi-client check for small shared-hosting sites.

What it checks

For every URL listed in your sitemap(s):

Check	Reported as issue
HTTP status	client-error (4xx), server-error (5xx)
Redirects	redirect-chain: more than 1 hop before the final page (or a redirect loop)
Indexability	noindex: page in the sitemap but blocked by <meta name="robots"> or X-Robots-Tag
Canonical	canonical-mismatch: the page's rel=canonical points to a different URL
Crawlability	robots-disallow: page in the sitemap but blocked by your robots.txt (User-agent: * rules)
International	hreflang-mismatch: a <link rel="alternate" hreflang> pair is broken (see below)
Availability	timeout (no response within the timeout), fetch-error (DNS, TLS, connection refused)

It also records final URL after redirects, the number of redirect hops, and response time.

robots.txt is read once per host and applied the way Google does it: rules of the User-agent: * group(s), the longest matching Allow/Disallow wins (Allow wins a tie), * wildcards and $ end anchors are supported. The issue's detail names the rule, e.g. robots.txt Disallow: /private/ (User-agent: *). A missing robots.txt (4xx) means everything is allowed. If robots.txt answers 5xx, 429 or times out, the run adds a robots-unavailable notice and leaves earlier robots-disallow issues unchanged (not resolved). Disallowed URLs are still checked for status, redirects and the rest, and cost the same as any other URL.

hreflang is read from each page's <link rel="alternate" hreflang="..."> tags. For every language version a page points to, the monitor checks that the version answers HTTP 200 without a redirect, links back to the page (reciprocal pair), and, if it is on the same host, is listed in your sitemap. One issue per page lists every broken pair as <page locale> -> <version locale> (<URL>): <problem>, e.g. en -> de (https://www.example.com/de/): no return link. Problems: no return link, HTTP 404 (any non-2xx status), redirects to <URL>, not in the sitemap, invalid URL. Versions already checked in this run are reused; others are fetched once (at most one extra request per checked URL, 500 per run) and are not charged. If a version can't be verified (timeout, rate limit, bot protection, or the per-run limit, which adds a hreflang-partial notice), the page keeps its previous hreflang-mismatch state instead of a false new or resolved issue. Not read: hreflang in HTTP Link: headers or in sitemap <xhtml:link> entries.

Canonical comparison ignores tracking parameters (utm_*, fbclid, gclid, msclkid, mc_cid, mc_eid, redirect_source), host letter case, default ports and #fragments. Path case is significant (/Blog and /blog are different URLs).

Not reported as issues: HTTP 429 (rate limiting) and bot-protection / challenge pages. A server that throttles the monitor is not a broken page; see Rate limiting and Cloudflare / WAF allowlist.

Sitemap support: robots.txt discovery (Sitemap: lines, absolute or relative), sitemap index files (nested), gzipped .xml.gz, plain-text sitemaps, and namespace-prefixed XML (<sm:urlset> / <sm:loc>). Discovery always starts at the host root: for https://example.com/news/ it reads https://example.com/robots.txt and says so in a discovery-root notice. To check one section only, set sitemapUrls to that section's sitemap. If robots.txt lists no sitemap, the actor probes /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.xml.gz. If none exists, the run ends with a no-sitemap notice: set sitemapUrls to your sitemap's address. Discovery is capped at 120 s in total. If robots.txt / the probe paths time out or drop the connection, the run ends with a sitemap-timeout notice listing those URLs instead (the site may still have a sitemap). Sitemap downloads get a 60-second timeout and one retry.

A sitemap that can't be read (HTTP 404/5xx, blocked, not XML, too large) or that contains no URLs is never silent: the run adds a sitemap-error notice naming each failing sitemap with its HTTP status or reason, lists them in sitemapErrors, and the run's status message says how many sitemaps failed. Issues on URLs listed only in a failing sitemap are kept as unchecked, not marked resolved.

Input

Field	Type	Default	Description
siteUrl	string	none	Your homepage, e.g. example.com or https://www.example.com (https:// is added when missing). Sitemaps are discovered from the host's /robots.txt.
sitemapUrls	string[]	[]	Explicit sitemap / sitemap index URLs. Overrides discovery.
maxUrls	integer	500	Max URLs checked per run (1–10,000).
notifyWebhookUrl	string	none	Send an alert here when there are new issues or a sitemap starts failing.
webhookFormat	string	auto	auto: a readable message for Slack URLs (hooks.slack.com), the summary JSON for any other URL. slack: always a chat message {"text": ...} (Mattermost, Rocket.Chat, Discord with /slack at the end of its webhook URL). json: always the summary JSON.
concurrency	integer	5	Parallel checks (1–20).
hostDelayMs	integer	250	Minimum gap between requests to the same host, so your server isn't strained. Raised automatically when the server answers 429.
timeoutSecs	integer	15	Per-request timeout (1–60).

{
  "siteUrl": "https://www.example.com",
  "maxUrls": 1000,
  "notifyWebhookUrl": "https://hooks.slack.com/services/XXX/YYY/ZZZ"
}

Output

The dataset holds only issues (not every healthy URL) plus one summary record per run.

Issue record:

{
  "recordType": "issue",
  "change": "new",
  "issue": "client-error",
  "url": "https://www.example.com/old-pricing",
  "status": 404,
  "finalUrl": "https://www.example.com/old-pricing",
  "redirects": 0,
  "responseMs": 182,
  "detail": "HTTP 404",
  "firstSeenAt": "2026-10-09T06:00:04.112Z",
  "checkedAt": "2026-10-09T06:00:04.112Z",
  "resolution": null
}

• change: new (first seen this run), persisting (also present last run), resolved (was present last run, gone now).
• resolution (resolved rows only): fixed (URL re-checked and healthy) or removed-from-sitemap.
• firstSeenAt: when the issue was first detected, so you can see how long it has been open.

Summary record (also saved as OUTPUT in the run's key-value store):

{
  "recordType": "summary",
  "site": "www.example.com",
  "checkedAt": "2026-10-09T06:00:04.112Z",
  "previousCheckedAt": "2026-10-08T06:00:03.954Z",
  "sitemaps": ["https://www.example.com/sitemap.xml"],
  "sitemapErrors": [],
  "newSitemapErrors": [],
  "discoveryComplete": true,
  "urlsDiscovered": 812,
  "urlsChecked": 810,
  "urlsRateLimited": 2,
  "urlsBlocked": 0,
  "siteBlock": null,
  "chargeLimitReached": false,
  "issues": { "new": 1, "persisting": 3, "resolved": 1, "open": 4, "unchecked": 0 },
  "openByType": { "redirect-chain": 1, "noindex": 1, "canonical-mismatch": 1, "server-error": 1 },
  "newIssues": [{ "url": "https://www.example.com/blog", "issue": "server-error", "detail": "HTTP 500" }],
  "notices": [{ "code": "rate-limited", "message": "2 URLs kept returning HTTP 429 after 3 retries ..." }],
  "webhook": { "status": 200, "error": null }
}

• sitemapErrors: every sitemap that failed this run, as { "sitemap", "error", "block" }, e.g. { "sitemap": "https://www.example.com/news-sitemap.xml", "error": "Sitemap returned HTTP 404", "block": null }. newSitemapErrors: the ones that were not failing in the previous run (these trigger the webhook).
• urlsChecked: URLs actually checked (and charged) this run.
• urlsRateLimited: URLs that still got HTTP 429 after retries. Not checked, not charged.
• urlsBlocked: URLs refused by bot protection (challenge pages) or, when the whole site blocks the monitor, by 401/403. Not checked, not charged.
• siteBlock: null, or why the whole site refused the monitor (auth-required, forbidden, bot-protection, rate-limited).
• issues.open: open issues on URLs checked this run (never more than the URLs checked here). issues.unchecked: earlier issues on URLs not checked this run. They are kept for the next run and are neither counted as open nor marked resolved.
• notices: run-level messages. site-blocked (site protection refused the monitor), rate-limited, no-sitemap (set sitemapUrls), sitemap-timeout (discovery URLs did not respond within 120 s), robots-unavailable (a host's robots.txt returned 5xx/429 or timed out, so its rules were not applied), hreflang-partial (some hreflang language versions outside this run were not fetched; see above), sitemap-error (a sitemap failed or has no URLs; names each one with its HTTP status or reason), discovery-root (siteUrl has a path, but discovery uses the host root robots.txt), charge-limit (the run's maximum charge was reached: "checked N of M URLs"; raise the maximum cost per run to check them all).

Issue types in issue / openByType: client-error, server-error, redirect-chain, noindex, canonical-mismatch, robots-disallow, hreflang-mismatch, timeout, fetch-error. A robots-disallow row looks like:

{
  "recordType": "issue",
  "change": "new",
  "issue": "robots-disallow",
  "url": "https://www.example.com/private/report",
  "status": 200,
  "detail": "robots.txt Disallow: /private/ (User-agent: *)"
}

A Slack incoming webhook (hooks.slack.com) gets a readable message: the site, the new, open and resolved counts, up to 10 new issues with their URL and reason, and each sitemap that started failing. Every other webhook URL (Zapier, Make, n8n, your own endpoint) receives the same summary object as its JSON body; set webhookFormat to choose explicitly. The alert is sent when a run finds new issues (issues.new > 0) or when a sitemap starts failing (newSitemapErrors is not empty). A sitemap that keeps failing alerts once, not on every run; if it recovers and breaks again, you get a new alert. The sitemap health is stored in the same snapshot as the issues.

Recurring monitoring (Apify Schedules)

The actor keeps a snapshot of open issues per site and sitemap scope (named key-value store sitemap-health-monitor), so every run diffs against the previous one automatically. Runs that find the sitemap on their own use key snapshot-<host>. Runs with an explicit sitemapUrls list use snapshot-<host>!<hash>, where the hash is stable for the same set of sitemap URLs (order, duplicates and surrounding spaces don't matter). So a task that checks only news-sitemap.xml keeps its own baseline and never marks the full-site task's issues as resolved, and the full-site task never sees them as new again. The first run of a new sitemapUrls selection starts from the site's earlier results for the pages in those sitemaps, so known issues stay persisting instead of coming back as new. To monitor:

1. Open the actor and click Save as a new task with your input (site URL and webhook).
2. Go to Schedules, click Create new, pick a cadence (e.g. daily at 06:00), and add the task.
3. Done. Each run produces only the changes; the webhook fires only when there are new issues or a sitemap stops working.

Use one task per site. The first run reports all current issues as new (it has no baseline yet). The diff only compares URLs checked in both runs. If a run checks fewer URLs (lower maxUrls, charge limit, rate limiting, site protection, or a sitemap couldn't be read), issues on the URLs it skipped are counted in issues.unchecked. They are never marked resolved by mistake, and when those URLs are checked again they come back as persisting, not new. URL selection follows sitemap order, so repeated runs with the same maxUrls check the same URLs.

How this differs from a one-off crawl

A one-off site audit tells you everything that is wrong today, and next week you get the same list again and have to work out what changed yourself. This actor is built for SEO monitoring: it keeps the previous run's results, so each report is a diff (new, still broken, fixed), and the alert fires only on new problems. It checks the URLs you actually want indexed (your sitemap), not every link a crawler can reach, which keeps runs fast and cheap enough to schedule daily.

Rate limiting

Some servers (Shopify stores in particular) answer HTTP 429 "Too Many Requests" when crawled. The monitor then:

1. Waits as long as the Retry-After header asks (up to 30 s; exponential backoff if the header is missing) and retries up to 3 times.
2. Slows down for that host for the rest of the run: the gap between requests doubles on every 429, up to 3 seconds.
3. If a URL still gets 429, it is reported in urlsRateLimited and a rate-limited notice. It is not reported as a broken page, not charged, and keeps its previous state.

If many URLs are rate-limited, lower concurrency (e.g. 2), raise hostDelayMs (e.g. 1000), or allowlist the monitor (below).

Cloudflare / WAF allowlist

The monitor never tries to bypass bot protection, CAPTCHAs or challenge pages. If most of the first checks (at least 80% of the first 10) are refused with 401/403/429 or a challenge page, or the sitemap itself is behind a challenge, the run stops early with one site-blocked notice instead of hundreds of false errors. Nothing is charged for the refused URLs and earlier results are left as they were.

To monitor a site behind a firewall, allowlist the monitor by User-Agent (SitemapHealthMonitor). Requests come from Apify's cloud IPs, which change, so allowlisting by IP won't work.

• Cloudflare: open Security, then WAF, Custom rules, Create rule. Field User Agent, operator contains, value SitemapHealthMonitor. Action Skip: tick all remaining custom rules, rate limiting rules, managed rules and Super Bot Fight Mode. If you use Bot Fight Mode (free plan), it can't be skipped by a rule, so turn it off or put the sitemap URLs on an unprotected path.
• Other WAFs / CDNs (Akamai, Fastly, Sucuri, Imperva, AWS WAF): add an allow rule matching the User-Agent substring SitemapHealthMonitor, placed before bot-management and rate-limit rules.
• Login-protected sites (401): the monitor can't log in. Monitor the public part of the site, or put a public sitemap in sitemapUrls.

Treat a User-Agent allow rule as a convenience, not as security, because anyone can send that User-Agent. Keep the rule limited to skipping bot checks and rate limits, not authentication.

Pricing (pay per event)

Event	Price
url-checked: one sitemap URL checked	$0.0005 ($0.50 per 1,000 URLs)

No start fee. A daily check of a 500-URL site costs about $0.25/day. URLs that were rate-limited or refused by site protection are not charged. The actor respects your run's maximum charge: it stops checking new URLs once the limit is reached and still saves the summary and snapshot.

The hreflang check adds no event: language versions fetched to verify a pair are not charged.

Safety

The actor only requests http(s) URLs on public addresses. Private, loopback, link-local and cloud metadata IPs are blocked, including hostnames that resolve to them, redirects that point to them and IPv6 spellings of them (IPv4-mapped such as [::ffff:127.0.0.1], NAT64, 6to4); only public IPv6 (global unicast) is allowed. This applies to the webhook URL too. It sends a clear User-Agent (SitemapHealthMonitor/0.1; +https://pankovs.com) and spaces requests per host.

Local development

npm install
npm run typecheck && npm run lint && npm test
npm run smoke        # build + run dist/main.js twice against a local fixture site, asserts the diff

Run the built actor locally against any public site:

npm run build
mkdir -p storage/key_value_stores/default
echo '{"siteUrl":"https://www.sitemaps.org","maxUrls":30}' > storage/key_value_stores/default/INPUT.json
CRAWLEE_STORAGE_DIR=./storage node dist/main.js
python scripts/show-run.py storage

SHM_ALLOW_PRIVATE_HOSTS=1 lifts the private-IP block for local fixture testing only; it is ignored when running on the Apify platform.

More

Real output, prices and changelog: Sitemap Health Monitor, broken links and 404 alerts.