Sitemap Link Auditor: Broken Links, Redirects & Slow Pages

Give it a site you own. It finds the site's sitemap, lists the URLs the site published there, and checks each one: HTTP status, the full redirect chain, how long the response took, content type, and the lastmod the sitemap declares. You get one flat JSON row per URL with a plain list of issues, ready for a dashboard, a CI check, or an AI agent.

This Actor only audits URLs the site itself published in its own sitemap. It does not crawl, it does not follow links found in HTML, and it does not check external links.

What you get per URL

Field	Meaning
site	The site from your input
url	The URL as published in the sitemap
finalUrl	Where the request ended up
statusCode	Final HTTP status (number), null when the URL was not fetched
redirectChain	[{url, status}] for every hop, including the final response
redirectChainTruncated	true when more than 5 redirects were attempted; the chain stops at the cap
redirectCount	Number of redirects followed (max 5)
responseTimeMs	Time to the final response headers
contentType	Content-Type of the final response, or null
lastmod	<lastmod> from the sitemap, or null
status	OK, ISSUE, TIMEOUT, ROBOTS_DISALLOWED, EXTERNAL_HOST_SKIPPED, REFUSED or ERROR
issues	List of {severity, code, message} — see below
checkedAt	ISO timestamp

Issue codes: BROKEN_4XX, SERVER_ERROR_5XX, REDIRECT_CHAIN_LONG (3+ hops, or more than 5 when redirectChainTruncated is set), REDIRECT_TO_DIFFERENT_HOST, SLOW_RESPONSE (>3000 ms), TIMEOUT, ROBOTS_DISALLOWED.

Error rows (free, no issues list): INVALID_SITE, NO_SITEMAP, SITEMAP_UNUSABLE, SITEMAP_FETCH_FAILED, CHECK_FAILED, REFUSED_PRIVATE_ADDRESS, FETCH_FAILED.

Example input

{ "sites": ["www.sitemaps.org"], "maxUrlsPerSite": 3 }

Example output (real run, one row shown in full)

{
  "site": "www.sitemaps.org",
  "url": "https://www.sitemaps.org/",
  "finalUrl": "https://www.sitemaps.org/",
  "statusCode": 200,
  "redirectChain": [{ "url": "https://www.sitemaps.org/", "status": 200 }],
  "redirectCount": 0,
  "responseTimeMs": 142,
  "contentType": "text/html",
  "lastmod": "2016-11-21",
  "issues": [],
  "checkedAt": "2026-10-28T05:31:12.744Z",
  "status": "OK"
}

A site without a sitemap returns one free row instead:

{ "site": "example.com", "error": "NO_SITEMAP", "message": "No sitemap found via robots.txt, /sitemap.xml or /sitemap_index.xml" }

How discovery works

1. https://{host}/robots.txt — every Sitemap: line is read.
2. No usable sitemap there? Then /sitemap.xml, then /sitemap_index.xml.
3. Sitemap indexes are followed up to 3 levels deep; gzip documents are inflated; <loc>, <lastmod>, CDATA and entities are parsed, including sloppy real-world XML. Parsing is a single linear pass (each tag is located by forward scanning, never by a regex retried at every position), so unclosed or truncated markup cannot stall the run. XML parsing is capped at 4 MB per document (above this the document is reported as SITEMAP_UNUSABLE, since this Actor checks at most 5000 URLs anyway), and only the first maxUrlsPerSite entries are collected.
4. Nothing found → one free NO_SITEMAP row for that site. A sitemap that was found but rejected (too large, bad gzip, not a sitemap, disallowed by robots.txt) produces one free SITEMAP_UNUSABLE row naming the URL and the reason instead.

Pricing

Pay per event: a small start fee per run plus $0.0018 for each URL actually fetched and delivered. Rows we do not fetch are free: NO_SITEMAP, SITEMAP_UNUSABLE, invalid sites, fetch errors, timeouts (status TIMEOUT, no HTTP response received), URLs robots.txt disallows (status ROBOTS_DISALLOWED), URLs on another host (status EXTERNAL_HOST_SKIPPED) and refused internal addresses.

includeOk: false hides healthy results: a URL is delivered unless it came back as a clean OK (no issues, no error). You still receive every problem row and every free row — robots-disallowed, external-host-skipped, refused, and every error row (including CHECK_FAILED and NO_STATUS, which carry no issues list). Every URL is still checked, but you are only charged for fetched URLs delivered with issues — free rows never cost anything. If your spending limit is reached the run stops cleanly instead of doing unpaid work.

Good to know

• Own or authorised sites only. Public domain names only: IP addresses, localhost, private and internal hostnames are rejected, and a sitemap entry pointing at 169.254.169.254, 127.0.0.1, ::1 or *.internal is refused without a single request. Every connection, including each redirect hop, is made to an address validated as public at connect time; a redirect to a private or metadata address is refused rather than followed.
• robots.txt is honoured for the URLs we check, on every redirect hop: a disallowed URL (or a redirect target on the same host) is reported with status ROBOTS_DISALLOWED and never requested. Disallowed sitemap documents are not downloaded either, and that includes a declared sitemap whose redirect hop lands in a disallowed path: the hop is refused before it is dialled and the site gets one free SITEMAP_UNUSABLE row.
• Same host only. EXTERNAL_HOST_SKIPPED means the URL in the sitemap is on a different host than the one you gave. This is literal: if your sitemap uses www.example.com, pass www.example.com (an apex input example.com will report its www URLs as skipped).
• Polite by default: 4 concurrent requests per host, at least 100 ms between request starts, 10 s timeout, identifying apify-actor-fleet/1.0 User-Agent. Every individual request is paced — each redirect hop and each HEAD/GET attempt counts as a request, not just each URL check. Pacing is a promise about request starts (each start is reserved before any wait, so concurrent slots cannot start together); network connect time can shift the arrival time a server observes by a few milliseconds. HEAD first, with a GET (at most 8 KB read) when the server answers 403/405/501 or errors.
• Limits: maxUrlsPerSite default 200 (hard cap 5000), maxSites default 5 (hard cap 50), redirects followed 5. URLs are taken in sitemap order, so the cap truncates the tail of a very large sitemap.
• checkExternalLinks is not supported — this Actor never leaves the sitemap.
• A bot-protection wall may answer 403 to our requests; that is reported as a broken link with the real status, because it is what a crawler would see.

Implementation note (shared network guard)

src/lib/http.js is a deliberate superset of the fleet's shared network kit: it adds HEAD requests, a bounded body read (8 KB per link check, 20 MB per sitemap) and a redirect hop chain. It keeps every safety check from the shared kit unchanged: IP literals are validated on every hop, and each connection resolves through the address-validating lookup. Its loopback test escape (allowPrivateHost) accepts only 127.x / ::1, applies to the first hop's origin only (redirect hops to any other origin are refused), and production code never sets it. The fleet audit gate tracks this file as a declared fork and fails if it ever loses those guard calls.