Sitemap Link Checker: broken links, redirect chains and noindex pages

Give it a sitemap (or just your homepage) and get one row per URL with the HTTP status code, the full redirect chain, response time, noindex (meta robots and X-Robots-Tag), canonical and the sitemap's lastmod. Turn on the link crawl to also check every link on those pages and find broken links inside your content.

• Reads any sitemap: sitemap indexes (nested), .xml.gz, plain-text sitemaps, and Sitemap: lines in robots.txt. Give a homepage and it finds the sitemap for you.
• Every redirect hop recorded: see 301 → 302 → 200 chains, redirect loops, and redirects that end in a 404.
• SEO checks in the same pass: noindex pages that are still in your sitemap, and pages whose canonical points somewhere else.
• Broken link crawl (one level): checks the links on each page, internal only or outbound too, and tells you which page each broken link was found on.
• Fast and light: plain HTTP requests, no browser, so it runs in 256–512 MB.
• Polite by default: respects robots.txt (including Crawl-delay), sends at most 2 requests per second per site by default, backs off automatically when a site answers 429 or 503, and identifies itself with a clear user agent.

What can you use it for?

Use case	How
Find broken pages listed in your sitemap	Add the sitemap, filter isBroken = true
Clean up redirect chains after a migration	Look at redirectCount > 1 in the Redirect chains and errors view
Find noindex pages that shouldn't be in the sitemap	Filter noindex = true
Check canonicals	Filter canonicalIsSelf = false
Find broken links in your content	Turn on Also check links found on pages
Monitor a site	Schedule a daily or weekly run and get the SUMMARY record by webhook
Feed other tools	Export the 200 OK URLs to the Lighthouse Bulk Auditor or your own pipeline

How to use it

1. Paste one or more sitemap URLs, for example https://www.example.com/sitemap.xml, or add your homepage under Start URLs.
2. Set Max URLs to check. The default is 50 so a first run is quick; raise it for a full site.
3. Optional: turn on Also check links found on pages to find broken links.
4. Click Start. Download the results as JSON, CSV, Excel or HTML, or read them with the API.

Input

Field	Description	Default
sitemapUrls	Sitemaps, sitemap indexes, .gz sitemaps or robots.txt URLs	["https://apify.com/sitemap.xml"]
startUrls	Pages or homepages to check directly	–
discoverSitemaps	Read the sitemaps listed in each start URL's robots.txt (falls back to /sitemap.xml)	true
maxUrls	Cap on URLs from start URLs and sitemaps (shared fairly between several sitemaps)	50
crawlLinks	Also check <a href> links found on checked pages, one level deep	false
sameDomainOnly	Only check links on the same host (www. ignored)	true
maxLinkUrls	With the link crawl on: cap on extra on-page link URLs, on top of maxUrls	500
maxConcurrency	Requests in flight at once, across all sites	10
maxRequestsPerSecondPerHost	Per-site rate limit; slows down automatically on 429/503	2
requestTimeoutSecs	Timeout per request (each redirect hop)	15
maxRedirects	Longer chains are reported as an error	10
maxRetries	Retries for timeouts, 429 and 503 (honours Retry-After)	2

Example: check a whole site and every internal link on it.

{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "maxUrls": 5000,
  "crawlLinks": true,
  "maxLinkUrls": 5000,
  "sameDomainOnly": true
}

Output

One dataset row per URL. Real example from a test run against apify.com:

{
  "url": "https://apify.com/alternatives/gmapsextractor-alternative",
  "source": "sitemap",
  "foundOn": null,
  "sitemapUrl": "https://apify.com/sitemap/pages.xml",
  "lastmod": null,
  "robotsBlocked": false,
  "statusCode": 308,
  "finalStatusCode": 200,
  "statusClass": "2xx",
  "isBroken": false,
  "finalUrl": "https://apify.com/alternatives/outscraper-alternative",
  "redirectCount": 1,
  "redirectChain": [
    { "url": "https://apify.com/alternatives/gmapsextractor-alternative", "statusCode": 308, "location": "https://apify.com/alternatives/outscraper-alternative" }
  ],
  "contentType": "text/html; charset=utf-8",
  "noindex": false,
  "noindexSource": null,
  "metaRobots": "robots: index,follow",
  "xRobotsTag": null,
  "canonical": "https://apify.com/alternatives/outscraper-alternative",
  "canonicalIsSelf": true,
  "linksFound": null,
  "error": null
}

Field	Meaning
source	sitemap, startUrl, or page (a link found on a checked page)
foundOn	For links: the page that links to this URL
statusCode / finalStatusCode	First response (e.g. 301) and the status after all redirects
isBroken	Final status is 4xx/5xx, or the request failed (DNS, timeout, TLS, redirect loop)
redirectChain	Every hop: URL, status, and where it pointed
responseTimeMs	Network time until the final response headers, summed over all hops
noindex, noindexSource	meta, header (X-Robots-Tag), or both
canonical, canonicalIsSelf	From <link rel="canonical"> or the Link header
robotsBlocked	robots.txt disallows this URL, so it was not requested
error	Why the check failed, or a note such as "Rate limited by the site (429)". A final 429 is not counted as broken

There is also a SUMMARY record in the key-value store with counts of 2xx/3xx/4xx/5xx/errors (by first and by final status), the number of broken and rate-limited URLs and redirect chains with more than one hop, noindex and non-self canonical counts, average response time, the sitemaps that were read (with any errors), and lists of the broken URLs and long redirect chains (up to 200 each).

How much does it cost?

$0.50 per 1,000 URLs checked ($0.0005 per URL), plus a start fee of $0.00005 per run at the default memory (Apify's standard Actor-start event). Platform usage is included, so that is the whole bill: checking a 10,000-URL sitemap costs about $5.

You only pay for URLs that returned an HTTP response, whatever the status (a 404 is a useful result). URLs skipped because of robots.txt and requests that got no HTTP response at all (DNS failure, connection refused, timeout) are free. Set Maximum cost per run in the run options to cap your spend: the Actor checks as many URLs as that allows and stops cleanly. See the Pricing tab for the current price.

Good to know

• robots.txt is always respected. If your own site blocks bots, allow the SitemapLinkChecker user agent for the paths you want checked. A robots.txt that returns a 5xx error means "don't crawl" (RFC 9309), so that site's URLs are skipped and marked robotsBlocked.
• Pages are fetched with GET, because some servers answer HEAD wrongly. Only HTML pages are read (up to 3 MB) to look for meta robots, canonical and links; other files are closed after the headers.
• No JavaScript rendering. Links added by JavaScript after the page loads aren't seen. Use a browser-based crawler for those.
• The link crawl is one level deep. Links found on pages are checked but not crawled further. Pages from your sitemaps and start URLs are checked first, then the links found on them, up to maxLinkUrls.
• No personal data is collected. The Actor stores URLs, status codes and SEO headers only, never page text. mailto: and tel: links are ignored.
• Only check sites you own or have permission to test, and keep the rate limit gentle on sites that aren't yours.

Further reading: How to find broken links from your sitemap.xml (and check every URL in it), a step-by-step walkthrough on dev.to.

FAQ

Can it check a sitemap with 50,000 URLs? Yes. Set maxUrls high enough. At the default 2 requests per second per site, 50,000 URLs on one site take about 7 hours; raise maxRequestsPerSecondPerHost on your own site to go faster.

Why is a URL marked broken with no status code? The request failed before any response: DNS lookup, connection refused, TLS certificate problem, timeout, or a redirect loop. The error field says which.

Does it follow redirects to other domains? Yes, every hop is followed and recorded, up to maxRedirects.

Can I run it on a schedule? Yes. Create a task with your input, add a schedule, and use a webhook or integration to get the SUMMARY record after each run.

Questions or a site that doesn't work? Open an issue on the Issues tab.