Broken Link Checker & Sitemap Extractor (SEO, robots.txt)

Find broken links (404, 410, 5xx, DNS errors, timeouts), redirect chains and broken images across any website, or extract every URL from its XML sitemaps (sitemap indexes and .gz included). Each run also returns a robots.txt summary: declared sitemaps, disallow rules, and which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot...) are blocked.

It uses plain HTTP, so it's fast and cheap. There's no browser and no proxy. You pay per URL checked, with no subscription.

Why use it

• Two tools in one. A full-site broken link checker plus a sitemap extractor/validator.
• Shows where each broken link lives. Every result lists up to 5 pages it was found on, with the anchor text, so you can fix it in minutes.
• Low false-alarm rate. Each URL gets a HEAD request with GET fallback, and 429 responses are retried after Retry-After. Bot walls (401/403/429/999 from LinkedIn, Stack Overflow, npm...) are reported as RESTRICTED, not as broken.
• Redirect audit. You get finalUrl, redirectCount and the full redirectChain, which helps find redirect chains and http→https hops.
• Sitemap health. It lists every <loc> with lastmod, changefreq and priority. Optionally it checks the HTTP status of every sitemap URL, so you can catch 404s and redirects that waste crawl budget.
• Polite by design. It respects robots.txt and Crawl-delay and caps per-host concurrency. URLs blocked by robots.txt are skipped and not charged.
• Export anywhere: JSON, CSV, Excel, Google Sheets, API, Make, Zapier or n8n. Schedule it weekly to monitor your site.

How to use it

1. Enter one or more websites under Websites, e.g. https://example.com.
2. Choose a Mode:
   • Broken link checker crawls up to Max pages internal pages and checks every unique link and image on them, both internal and external.
   • Sitemap extractor reads robots.txt Sitemap: lines, or /sitemap.xml and /sitemap_index.xml, follows nested indexes and lists all URLs.
3. Optional: turn on Only output broken links, or Check HTTP status of sitemap URLs.
4. Click Start. Use the Broken links only view, or filter linkStatus = BROKEN.

Input example

{
  "startUrls": [{ "url": "https://crawlee.dev" }],
  "mode": "brokenLinks",
  "maxPagesPerSite": 50,
  "maxUrlsPerSite": 3000,
  "checkExternalLinks": true,
  "checkImages": true
}

Output example (broken link)

{
  "type": "LINK",
  "site": "https://crawlee.dev",
  "url": "https://github.com//github.com/apify/crawlee/pull/1890/issues",
  "statusCode": 404,
  "ok": false,
  "linkStatus": "BROKEN",
  "finalUrl": "https://github.com//github.com/apify/crawlee/pull/1890/issues",
  "redirectCount": 0,
  "contentType": "text/html",
  "responseMs": 212,
  "linkType": "external",
  "isImage": false,
  "foundOn": [{ "page": "https://crawlee.dev/js/api/changelog", "text": "#1890", "kind": "link" }],
  "linkText": "#1890"
}

Output example (sitemap URL)

{ "type": "SITEMAP_URL", "site": "https://crawlee.dev", "url": "https://crawlee.dev/blog", "lastmod": null, "changefreq": "weekly", "priority": "0.5", "sitemap": "https://crawlee.dev/sitemap.xml" }

Every site also gets one SITE_SUMMARY item. It covers robots.txt (sitemapsDeclared, disallowRulesForAll, aiBotsBlocked, blocksAllBotsCompletely) and totals (pagesCrawled, urlsChecked, brokenCount, brokenInternal, brokenExternal, restrictedCount, redirectedCount, sitemapUrlsFound).

linkStatus values: OK, BROKEN (4xx/5xx, DNS error, timeout), RESTRICTED (401/403/429/999: the server blocks bots, though the link usually works in a browser), and SKIPPED_ROBOTS_TXT.

Pricing (pay per event)

Event	Price
Link checked (any URL requested and status-checked, including crawled pages)	$0.0003 ($0.30 per 1,000)
Sitemap URL extracted (sitemap mode without status check)	$0.0001 ($0.10 per 1,000)

Example: a 100-page site with 2,500 unique links costs $0.75 (2,500 × $0.0003). Extracting a 10,000-URL sitemap costs $1.00. URLs skipped because of robots.txt are free. Use Max URLs per site to cap your spend.

FAQ

Does it render JavaScript? No. It reads the HTML your server returns, which is what search engines see first. Links that are injected only by client-side JavaScript won't be found.

Why is a working link marked RESTRICTED? Some sites (LinkedIn, Stack Overflow, npm, Cloudflare-protected sites) refuse automated requests. We don't bypass bot protection, so those links get their own status instead of being flagged as broken.

Does it follow nofollow links? It checks them, but it doesn't crawl further from pages marked <meta name="robots" content="nofollow">.

Can I check only my sitemap for 404s? Yes. Use Sitemap mode with Check HTTP status of sitemap URLs on, and Only output broken links on.

Can I run it on a schedule? Yes. Create an Apify Schedule (for example weekly) and send results to Slack, email or Google Sheets through an integration.