Broken Url Monitor

Stateful sitemap-based URL monitoring Actor.

This Actor continuously checks all URLs listed in a sitemap and detects regressions and changes over time, not just one-off failures.

---

What it does

• Parses sitemap files and sitemap indexes (nested sitemaps supported)
• Validates all URLs using HTTP HEAD (automatic fallback to GET)
• Stores a baseline snapshot on first run
• Compares subsequent runs against the previous snapshot
• Reports only meaningful changes, classified by severity

---

Detected changes

• Broken URLs (4xx / 5xx)
• Status code changes
• Newly introduced redirects
• URL recoveries (error → OK)
• Large response time regressions

---

Why this Actor

Most link checkers:

• Run once
• Dump raw status codes
• Require manual comparison

This Actor:

• Is stateful
• Safe to run repeatedly or on a schedule
• Produces diff-based output
• Designed for monitoring, not crawling

---

Input

Field	Type	Required	Default	Description
sitemapUrl	string	✅	–	URL of a sitemap or sitemap index
requestMethod	string	❌	head	HTTP method (head or get)
followRedirects	boolean	❌	true	Follow HTTP redirects
timeoutMs	number	❌	10000	Request timeout in milliseconds
maxConcurrency	number	❌	10	Number of parallel URL checks

---

First run (baseline)

On the first execution:

• All URLs are checked
• Results are stored as a baseline
• No diffs are reported

Example output:

{
  "baseline": true,
  "summary": {
    "total": 84,
    "ok": 84,
    "redirect": 0,
    "clientError": 0,
    "serverError": 0
  },
  "message": "Baseline stored. Monitoring is now active."
}

This allows you to validate a sitemap once for free before committing to monitoring.

---

Subsequent runs (monitoring)

On later executions:

• URLs are rechecked
• Results are compared against the previous snapshot
• Only changes are reported

Example output:

{
  "baseline": false,
  "summary": {
    "total": 84,
    "ok": 82,
    "errors": 2
  },
  "changes": {
    "critical": [
      {
        "url": "https://example.com/page",
        "previous": { "status": 200 },
        "current": { "status": 404 }
      }
    ],
    "warning": [],
    "info": []
  },
  "unchangedCount": 82
}

---

Severity levels

• critical
  • Status code regressions (e.g. 200 → 404)
• warning
  • Newly introduced redirects
  • Response time regressions (>4× increase)
• info
  • URL recoveries
  • Newly added URLs

---

Request Strategy (HEAD → GET Fallback)

To reduce false positives and unnecessary bandwidth usage:

1. URLs are checked using HEAD requests by default
2. If a server responds with 405 or fails:

• The Actor automatically retries using GET

This handles:

• Servers that block HEAD
• Misconfigured CDNs
• Legacy infrastructure

---

Recommended usage

• Run on a schedule (daily or hourly)
• Use Apify webhooks or integrations for alerts
• One Actor run equals one monitoring snapshot

Ideal for:

• SEO monitoring
• Content audits
• Website migrations
• Regression detection after deployments

---

Notes

• Each sitemap is monitored independently
• State is stored automatically per sitemap
• Safe to rerun indefinitely
• No proxies required for most sites

---

Webhook & Alerts

This Actor is designed to be used with Apify webhooks.

Example: Alert on broken URLs

Trigger condition:

• Event: ACTOR.RUN.SUCCEEDED
• Payload filter (pseudo):

payload.OUTPUT.baseline === false &&
payload.OUTPUT.changes.critical.length > 0

Example: Slack alert message (JSON)

{
  "text": ":rotating_light: Broken URLs detected",
  "attachments": [
    {
      "fields": [
        { "title": "Sitemap", "value": "{{sitemapUrl}}" },
        { "title": "Critical changes", "value": "{{criticalChanges}}" },
        { "title": "Warnings", "value": "{{warningChanges}}" }
      ]
    }
  ]
}

Example: Email summary

Send when:

• baseline === false
• criticalChanges > 0 Include:
• sitemap URL
• number of broken URLs
• link to Actor run