Sitemap & Broken Link Checker - Fast XML Crawler & 404 Auditor

A resilient, ultra-fast sitemap crawler and link health auditor built on Crawlee for Python. Automatically discovers, decompresses, and crawls XML sitemaps, verifies HTTP response status codes (200, 301, 404, 500), maps redirect hops, and exports a cleaned sitemap.xml with dead links excluded.

---

🚀 Why Choose This Actor?

Many legacy sitemap tools frequently crash on gzip-compressed files (.xml.gz), fail on nested sitemap indexes, or lack the ability to check HTTP status codes.

Sitemap & Broken Link Checker solves this with an asynchronous, high-concurrency engine:

• Universal Sitemap Support: Handles standard XML, .xml.gz gzip archives, nested <sitemapindex> trees, and automated robots.txt sitemap discovery.
• Deep HTTP Status Verification: Tests URLs with asynchronous HTTP HEAD requests (falling back to GET), verifying 200 OK, 301/302 redirects, and 404/500 errors.
• Redirect Chain Mapping: Accurately traces multi-hop redirects (e.g., HTTP -> HTTPS -> Non-www) and flags circular loops.
• Clean Sitemap Export: Automatically writes a fresh, sanitized sitemap.xml to Key-Value Store with broken URLs removed—ready to submit to Google Search Console or Bing Webmaster Tools.
• Ultra-Low Resource Footprint: Runs in a lightweight 256 MB – 512 MB RAM footprint at 25–50 URLs per second, saving up to 90% in platform compute costs compared to heavy browser crawlers.

---

⚡ How to Use in 3 Simple Steps

1. Enter Target Domain or Sitemap URL: Paste your website root (e.g. https://example.com) or a direct XML URL (e.g. https://example.com/sitemap.xml) into Start URLs.
2. Configure Scope: Set maxUrls (e.g., 100 for a health check or 10000 for a large domain) and keep autoDiscoverSitemap and checkStatus checked.
3. Run & Review: Click Start. Review your overall site health percentage in the console, or open the Key-Value Store tab to download your BROKEN_LINKS list and CLEAN_SITEMAP.

---

📊 Key-Value Store Outputs

• OUTPUT: Machine-readable JSON summary containing total URLs, valid count, broken count, redirect count, and overall health score.
• BROKEN_LINKS: Filtered JSON inventory of all dead 404s and 500s, including response latency and the originating sitemap file.
• CLEAN_SITEMAP: Valid, standards-compliant XML sitemap containing only healthy, verified URLs, ready for direct search engine submission.

---

💡 Real-World Use Cases

• SEO & Technical Site Migrations: Audit sitemaps before and after domain migrations to catch unintended 404s and permanent redirect chains.
• Google Search Console Cleanup: Remove dead and redirected URLs from your sitemap to protect crawl budget and improve indexing velocity.
• Continuous CI/CD Monitoring: Run on an automated Apify schedule (e.g. weekly or monthly) to catch broken links before they harm organic rankings.

---

❓ FAQ & Support

Can this Actor discover sitemaps automatically?

Yes. If you provide a standard domain URL (e.g. https://example.com), the Actor will automatically query robots.txt for Sitemap: directives and check standard paths (/sitemap.xml, /sitemap_index.xml).

Does it support gzipped .xml.gz sitemaps?

Yes. It detects gzip compression automatically and decompresses the XML stream in memory without writing temporary files to disk.

Need custom features or have feedback?

Submit an inquiry via the Issues tab on Apify Console. We respond promptly and continually maintain this Actor.