Sitemap Health Check: Crawl, Broken Links and On-Page SEO

Crawl your website and compare the pages found with your sitemap. Get page statuses, redirect chains, search visibility settings and on-page SEO checks in a dataset you can sort and filter.

Start with Your website or sitemap, leave Crawl and compare sitemap selected, and set Maximum pages or URLs to check to the size of the sample you want. The example checks up to 50 pages; increase that for your site. The crawl follows links on the same site, including its www and non-www forms, with a page cap and a link-depth limit.

It reports pages found outside the sitemap and sitemap pages with no incoming link observed on the checked pages. These are observations from a bounded crawl: they do not prove that a page is an orphan or that your entire site was checked. It reads server HTML, without JavaScript or login.

Pick a mode

You want to…	Set mode to	Give it	You get
Crawl a site and compare its sitemap	crawl (prefilled choice)	your website or sitemap URL	one page row per checked URL, with incoming links observed and sitemap comparison
Check only the URLs in your sitemap	sitemap	a sitemap or site URL	one row per URL listed in the sitemaps
Check a list of URLs, such as a migration redirect map	urlList	one URL per line	one row per URL
Check links and images on supplied pages	pageLinks	page URLs or a sitemap URL	one row per unique link or image, and one row per supplied page with broken and redirected links

In crawl mode, a website input starts at that page; a sitemap input also starts at the site's home page. Sitemap-listed pages are additional starting points. All starting points have depth 0, and the default follows up to 3 link steps from each. Sitemap pages and pages found through links share the page limit. Links to other sites and image addresses are not followed in crawl mode. pageLinks checks links on supplied pages or sitemap-listed pages without discovering further pages.

Sitemaps are found from robots.txt Sitemap: lines or common sitemap paths. Sitemap indexes and gzip sitemaps are supported.

Read the sitemap comparison

Crawl page rows carry inSitemap, linkedFrom (up to 20 source pages, with anchor text) and sitemapComparison. inSitemap is true when the URL was found in a sitemap, false when a complete sitemap read for that page's site establishes its absence, and null when no sitemap or an incomplete read leaves membership unknown:

Value	What it means
listed-linked	listed in a sitemap, with an incoming link observed on a checked page
listed-no-link-observed	listed in a sitemap, with no incoming link observed on the checked pages; investigate before calling it an orphan
not-listed	checked page absent from the sitemap files read; those files were read without reported errors or truncation
no-sitemap	no sitemap file was read; no sitemap comparison is available
unknown-incomplete-sitemap	page not among the URLs read, but a sitemap error or limit prevents a reliable comparison

The summary's pageCapReached, depthLimitReached, linksTruncated and timeLimitReached explain the coverage. siteSitemaps records discovery and completeness separately for each input site; sitemapComplete means every input site's sitemap read was complete. A complete sitemap read does not mean a complete site crawl. Sitemap differences are separate from problems: a page can be healthy and still warrant a sitemap review.

What it flags

URLs and pages (problems on sitemap, urlList, crawl and pageLinks page rows):

Code	Meaning
broken	the URL ends in a 4xx or 5xx status; crawl separately identifies the access signals below
redirected	the URL redirects; every hop is listed in redirectChain with its status
redirect-loop, too-many-redirects	the redirects go round in a circle, or run past 10 hops
noindex	noindex or none in <meta name="robots"> (or googlebot / bingbot), or in an X-Robots-Tag header
canonical-elsewhere	the page's <link rel="canonical">, or a Link: rel="canonical" header (used on PDFs and feeds), points to a different URL
blocked-by-robots, redirect-target-blocked-by-robots	the site's own robots.txt disallows the URL, or a redirect leads into a disallowed path; the URL is reported but not requested
fetch-error	no usable answer: the name doesn't resolve, the connection fails or times out, the site's security certificate fails (expired, self-signed, incomplete chain), or the address has an invalid port (a URL that cannot be read at all is invalid-url)
sitemap-requires-login	crawl: a sitemap-listed page appears to require login; check whether it belongs in the sitemap
host-request-limit	the host stopped after consecutive 429/503 responses or connection failures, so this check was not performed; the row is returned without a charge
host-bot-blocked	crawl: a page on this host answered with a bot challenge, so the rest of that host was left alone; the queued pages are listed without a request and without a charge
has-broken-links	pageLinks page rows: at least one link on the page is broken (see brokenLinks)

Also possible: robots-unreachable (the site's robots.txt answered with a server error, so its URLs were skipped), other-host-not-checked, invalid-url, unexpected-status.

Review broken, redirected, noindex and canonical-elsewhere sitemap URLs before keeping them in the sitemap. A blocked or refused request tells you what this checker could access, not what a search engine has indexed.

In crawl mode, access: "requires-login" is informational for a page reached only through public links. It is not followed further or reported as a broken link. For a sitemap-listed page it also raises sitemap-requires-login. Crawl also reports bot-blocked for a detected challenge and rate-limited for a 429 response. It does not bypass challenges: after a page answers with a bot challenge, the crawl asks that host for nothing else, lists the pages still queued for it as host-bot-blocked (a free, not-checked row), and says so in the summary's stoppedHosts and notes. Login detection uses HTTP authentication signals, familiar login paths and password forms; a plain 403 is access-forbidden, not proof of a login wall. Sitemap files that require login or refuse access are recorded with an error in the summary's sitemaps list.

Links (problems on pageLinks link rows):

Code	Meaning
broken	4xx (404, 410 and so on)
server-error	5xx
access-denied	401, 403 or 429: the linked site refused our checker. Many large sites do this to all automated clients, so check these by hand before removing them
fetch-error	no usable answer, as above (a site that no longer exists usually shows up here)
redirected, redirect-loop, too-many-redirects	as above; finalUrl is where the link ends up
blocked-by-robots	the linked site's robots.txt disallows our checker, so the link was not requested

mailto:, tel:, javascript: and same-page # links are skipped, and so is a link whose address cannot be read as a URL at all. On a page row, links.broken counts links that are broken, server-error, fetch-error, redirect-loop or too-many-redirects; access-denied links are counted separately in links.accessDenied.

On-page rules

On every HTML page it reads (in all modes), seoIssues lists the rules the page fails, each with a severity, a detail (such as the length found) and a fix hint, and seo holds what was found: title, meta description, H1s, lang, JSON-LD / microdata types, image and missing-alt counts, hreflang codes.

Rule	Severity	Fails when
title-missing	error	no <title>, or it is empty (a <title> inside an inline SVG doesn't count)
title-multiple	warning	more than one <title>, with a nonempty first title
title-too-long / title-too-short	notice	over 60 / under 30 characters
meta-description-missing	warning	no <meta name="description">, or it is empty
meta-description-multiple	warning	more than one meta description, with a nonempty first description
meta-description-too-long / meta-description-too-short	notice	over 155 / under 70 characters
h1-missing / h1-multiple	warning / notice	no <h1> / more than one
canonical-missing / canonical-multiple	notice / warning	no <link rel="canonical"> / several different ones
html-lang-missing	notice	no lang on <html>
viewport-missing	notice	no <meta name="viewport">
img-alt-missing	notice	an <img> has no alt attribute (alt="" is fine)
structured-data-missing	notice	no JSON-LD, microdata or RDFa
json-ld-invalid	error	a JSON-LD block is not valid JSON
hreflang-invalid	warning	an hreflang value is not a language code or x-default

The length limits are the ones common audit tools use; search engines publish no fixed limits. Notices are worth a look, not necessarily a fix: several <h1> elements, for example, are valid HTML. Turn the rules off with checkOnPage: false.

Input

Field	What it does	Default
url	Your website or sitemap; for supplied-page and URL-list modes, paste one URL per line	required; example: https://books.toscrape.com/
mode	crawl, sitemap, urlList or pageLinks	sitemap for omitted API inputs; form prefill: crawl
maxUrls	Maximum checked URLs/pages across sitemap seeds and discovered pages	5,000; form prefill: 50
maxDepth	Crawl: maximum link steps from each starting page, including sitemap-listed pages; 0 checks starting pages only	3
onlyProblems	Hide healthy rows; retain crawl sitemap differences and login information; summary counts all rows	false
checkPageTags	Read meta robots and canonical in sitemap and URL-list mode; crawl and page-links mode always read HTML	true
checkOnPage	Run the on-page rules on HTML pages read	true
maxSitemaps	Maximum sitemap files, including indexes	100
delaySecs	Minimum request spacing per host; a larger robots.txt delay wins in crawl (other modes cap that delay at 10 seconds)	1
timeoutSecs	Request timeout in seconds	20
maxCrawlSecs	Crawl time budget in seconds, including discovery and request waits; saves accumulated results when reached	240
maxLinks	Crawl: maximum distinct source/target link observations; pageLinks: maximum unique links checked	2,000
checkExternalLinks	Only pageLinks: also check links to other sites, respecting their robots.txt	true
checkImages	Only pageLinks: also check image addresses	true

The readable example is a public scraping-practice bookstore. Replace it with your site. The form groups optional settings under Advanced settings. A crawl with no usable HTTP or HTTPS input fails with an input error. With mixed valid and invalid inputs, valid sites are checked and invalid addresses get free type: "input" diagnostic rows plus invalidInputs in the summary.

Existing saved tasks and API inputs retain their mode and limits: omitting mode still selects sitemap. For a crawl through the API, set mode: "crawl" explicitly and supply maxUrls so your limit is clear. The schema and code default are both 5,000 in every mode; the form's example prefill is 50.

Compatibility fields remain accepted through JSON/API: urls, startUrls, sitemapUrls, sitemapUrl, sites, domains, site, domain, website and checkLinks. respectRobotsTxt and sameHostOnly remain accepted for old modes, with true as their defaults. Crawl always obeys robots.txt and stays on the input site even if these old options are false. maxConcurrencyPerHost is accepted but has no effect and is no longer a form field: requests to one host go out one at a time, since the outbound guard holds that host's turn for the whole request, so no setting can put two of them in flight.

Examples:

{ "mode": "crawl", "url": "https://books.toscrape.com/", "maxUrls": 50, "maxDepth": 3 }

{ "mode": "sitemap", "url": "https://example.com/sitemap_index.xml", "onlyProblems": true }

{ "mode": "urlList", "urls": ["http://example.com/old-page", "https://example.com/pricing"] }

{ "mode": "pageLinks", "url": "https://example.com/sitemap.xml", "maxLinks": 5000, "onlyProblems": true }

Output

Crawl page rows contain url, finalUrl, finalStatus, redirectChain, problems, seoIssues, crawlDepth, inSitemap, linkedFrom and sitemapComparison. Access-limited rows also carry access and an explanation in error. Use the comparison field to find sitemap differences, and problems for status and search-visibility issues. Check seoIssues separately: ok: true does not mean that every on-page rule passed.

A broken link found in pageLinks mode (a real row from one of our test runs on a public blog post):

{
  "type": "link",
  "url": "http://stackoverflow.com/jobs",
  "ok": false,
  "problems": ["redirected", "broken"],
  "finalStatus": 404,
  "finalUrl": "https://stackoverflow.com/jobs",
  "redirectCount": 1,
  "redirectChain": [{"url": "http://stackoverflow.com/jobs", "status": 301, "location": "https://stackoverflow.com/jobs"}],
  "kind": "link",
  "internal": false,
  "foundOn": [{"page": "https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/", "anchorText": "Stack Overflow Jobs", "nofollow": false}],
  "foundOnCount": 1,
  "error": null
}

The page row for that post lists links (found, checked, broken, redirected, accessDenied, notChecked), brokenLinks and redirectedLinks (each with URL, anchor text and final status), plus its own status, canonical, noindex and seoIssues.

A sitemap or urlList row:

{
  "type": "url",
  "url": "http://jekyllrb.com/docs",
  "ok": false,
  "problems": ["redirected", "canonical-elsewhere"],
  "finalStatus": 200,
  "finalUrl": "http://jekyllrb.com/docs/",
  "redirectCount": 1,
  "redirectChain": [{"url": "http://jekyllrb.com/docs", "status": 301, "location": "http://jekyllrb.com/docs/"}],
  "noindex": false,
  "canonicalUrl": "https://jekyllrb.com/docs/",
  "canonicalElsewhere": true,
  "blockedByRobots": false,
  "seoIssues": [
    {"rule": "meta-description-too-long", "severity": "notice", "detail": "222 characters", "fix": "Descriptions over 155 characters are usually cut off."},
    {"rule": "h1-multiple", "severity": "notice", "detail": "2 <h1> elements", "fix": "Several <h1> elements: fine in HTML5, but one clear main heading is the usual advice."}
  ]
}

Rows also carry sitemap and lastmod (sitemap mode), contentType, method, error and checkedAt. The last row ("type": "summary", also saved as the OUTPUT record) gives the totals: sitemaps read (status, URL count, gzip, parse errors), URLs and links checked, duplicates, caps reached, and counts per problem, per on-page rule and per final status.

How it checks, and how we test it

• Crawl uses GET to read pages and their links. Other modes start with HEAD, or a GET if the server refuses HEAD. Redirects are followed one hop at a time (up to 10) so every hop is recorded. HTML pages that answer 200 are read (first 1 MB) for meta robots, canonical, the on-page rules and, in crawl and pageLinks, their links.
• In pageLinks, each unique link is checked once, however many pages it appears on; foundOn lists up to 20 of those pages with the anchor text used on each.
• Broken sitemap XML is reported, and the URLs read before the error are still checked.
• robots.txt rules are cached and obeyed, also for links to other sites. Up to five robots redirects are followed; loops or unreadable rules stop page checks. A detected human challenge or 429 in robots.txt stops further probes and page requests to that host for the run. Requests are serialized per host by the outbound guard, with at least one second between completed requests, or a longer delaySecs or site Crawl-delay (uncapped in crawl mode; capped at 10 seconds in other modes). The guard backs off on 429, 503 and connection failures, and stops after three consecutive such failures or a Retry-After longer than 120 seconds. User agent MadrascoSitemapHealth, so site owners can allow or block it by name.
• Crawl has a default 240-second time budget covering discovery, sitemap reads, pacing and page requests. A wait longer than the remaining budget stops the crawl without shortening the site's delay. Accumulated page observations and the summary are saved with timeLimitReached: true; interrupted sitemap comparisons remain unknown. A platform timeout may shorten this budget to leave time for saving results.
• A 3 October 2026 local crawl of the public Books to Scrape practice site stopped at the configured 50-page cap and reported no-sitemap on each row. The crawl also reached its link-observation cap; this demonstrates a bounded sample, not whole-site coverage.
• A capped local crawl of the public Jekyll site read its sitemap and reported both listed pages with incoming links observed and listed pages without them. An independent AI agent checked membership against the saved sitemap XML and incoming links against the saved HTML. Membership left unknown by the capped sitemap read stays null.
• AI agents tested the existing sitemap, URL-list and supplied-page link checks on public sites we did not build, using a second tool: curl for statuses and redirects, and a separate implementation of each on-page rule on a different HTML parser; the agents investigated the differences. Recorded check (27 Sep 2026, 7 sites on WordPress, Shopify, Ghost, Jekyll, Hugo and Drupal): the on-page rules agreed on 85 of 86 pages (on the other, the site sent a different page to each tool); link statuses and redirect chains agreed with curl on 91 of 97 links (the other 6: two where curl skipped a redirect hop that we recorded, three on a site that rate-limits automated clients and answered each tool differently, one we did not request because robots.txt disallowed it).

Limits

• It checks what the server returns to an automated HTTP client. It does not run JavaScript, so titles, canonicals, noindex or links added by scripts are not seen.
• Some sites answer automated clients differently from browsers: they refuse them (access-denied) or send them somewhere else. Check access-denied links by hand before removing them.
• A site whose security certificate is incomplete may still open in a browser but is reported as fetch-error; the error field says which certificate check failed.
• It reports the signals a search engine reads; it can't tell you how a search engine will treat a page, and it gives no ranking advice or SEO score.
• pageLinks checks the pages you give it (or the pages your sitemap lists); it does not crawl your site to find pages.
• Crawl reports detected login walls, bot challenges, forbidden access and rate limits separately. It does not log in, solve challenges or bypass restrictions. Detection is based on response signals and can miss unfamiliar access walls.

Use with AI agents

For a crawl, call with {"mode": "crawl", "url": "<website or sitemap>", "maxUrls": 50}. Existing sitemap, urlList and pageLinks inputs remain supported. Read problems, seoIssues, access and sitemapComparison separately; ok reflects problems only. Always read the summary's limits before making claims about site coverage.

Pricing

No charge from us for now: you pay only Apify's platform usage of your run, at the default 256 MB memory. We may add a per-URL price in a later release; any price is shown on this page and by Apify before you start a run.

When a price exists, only rows the checker really asked a page for are chargeable. A row reported without a request is free: robots.txt disallowed it, the crawl recognised it as a login-only path before asking, a malformed URL never reached a page, the host's request or failure budget was already spent, or the host was left alone after a page-level bot challenge.

Support

Use the Issues tab on this actor's page. Built and maintained by Madrasco using AI agents. Platform support replies may be AI-assisted; a human owner can be reached on request. You can also email support@madrasco.dev.