Sitemap & Redirect Auditor

Audit your own website from its XML sitemap in one run. For every URL in the sitemap (and any extra URLs you add) you get the status code, the full redirect chain, redirect loops, the canonical tag compared with the URL, noindex from the meta robots tag or the X-Robots-Tag header, and whether robots.txt blocks it. Optionally it also checks the same-website links found on those pages, so you see broken internal links and links that point at redirects.

Built for SEO specialists, agencies and developers checking sites they run or have been asked to audit, for example after a migration or a redesign. Plain HTTP, no browser, so it is quick and cheap.

What it checks

• robots.txt: read first for every host, following RFC 9309 (our own user-agent group if there is one, otherwise *; longest match wins; * and $ wildcards). A 4xx answer means no rules; a 5xx or an unreachable robots.txt means everything is treated as disallowed, and nothing on that host is fetched.
• Sitemaps: the sitemap URL you give, or the Sitemap: lines in robots.txt, or /sitemap.xml. Sitemap index files are followed up to 3 levels and 50 sitemap files. Gzipped sitemaps are supported. Sitemaps on another website are not read.
• Each URL (one GET, redirects followed by hand up to 10 hops on the same website):
  • error pages (4xx/5xx) and URLs that don't answer;
  • redirects: hops, the chain with each status, loops, temporary redirects (302/303/307) to live pages;
  • canonical tag and whether it points to the URL itself;
  • noindex (meta robots or X-Robots-Tag);
  • sitemap problems: in the sitemap but redirecting, noindex, canonicalised elsewhere, or blocked by robots.txt.
• Internal links (optional): same-website links on the listed pages (rel="nofollow" links skipped), checked with a HEAD request (GET if the server refuses HEAD). One level only: links are never followed further.

Input example

{
  "websites": ["https://www.example.com"],
  "extraUrls": ["https://www.example.com/old-pricing"],
  "maxUrlsPerSite": 500,
  "checkInternalLinks": true,
  "maxLinksPerPage": 100
}

You can give a homepage or a sitemap URL, and up to 20 websites per run (they are audited one after another). extraUrls are used only when exactly one website is given. When internal links are on, the listed pages take up to half of maxUrlsPerSite and links the rest.

Output

One row per URL (type = url or link) and one summary row per website (also saved as SUMMARY in the run's key-value store).

Field	Meaning
url, source	The URL and where it came from: sitemap, input or link
inSitemap	Whether the URL is listed in the sitemap
statusCode, responseMs	First response and its time to headers
redirectHops, redirectChain, redirectLoop	Each hop as {url, status, to}
finalUrl, finalStatusCode	Where the chain ends
canonical, canonicalMatches	Canonical tag (resolved) and whether it equals the URL
robotsMeta, xRobotsTag, noindex	Indexing directives
robotsAllowed	false when robots.txt disallows the URL (it is then not fetched)
foundOn	For links: up to 3 pages that link to it
issues	Plain-English list of problems found
note, error	Why something was not followed or fetched

The summary row lists the robots.txt status, the sitemaps read (with entry counts), sitemap errors, URL counts, whether the URL cap was reached, stoppedEarly if the website asked us to stop, and a count per issue.

Only technical fields are saved. Page text, emails, phone numbers and names are never extracted or stored.

Limits and politeness

• At most 2 requests at a time to one website (a host and its www. twin share the limit) and at least 0.5 s between request starts, or the site's Crawl-delay up to 30 s. A site whose robots.txt asks for a longer Crawl-delay is not audited (the summary says why).
• Timeouts: 10 s to connect, 20 s per read, 45 s in total per request.
• Back-off: a 429 or 503 answer doubles the gap for that website (up to 30 s) and pauses it until its Retry-After (seconds or a date), then the request is tried once more. A Retry-After over 60 s, or three 429/503 answers in a row, stops the audit of that website (stoppedEarly in the summary). A 429 on robots.txt is treated like an unreachable robots.txt: nothing on that website is fetched.
• robots.txt is read once per host and read again if the run is still going 24 hours later.
• Page bodies are read up to 2 MB (only HTML, only for canonical/noindex/links), robots.txt up to 500 KB, sitemaps up to 50 MB (also after unzipping).
• Up to 10,000 URLs per website (default 500).
• No cookies (none are accepted or sent), logins, proxies or browser. Pages behind a login, captcha or bot check are reported with their status code and never bypassed.
• Only public web addresses on ports 80/443; private and reserved network addresses are refused. Platforms whose terms forbid automated access (for example Google, Amazon, LinkedIn, Facebook and other Meta sites, Instagram, X, TikTok, Yelp, Trustpilot, Indeed, Glassdoor, Zillow, Booking, Expedia, Tripadvisor, ticketing sites) are refused on every country domain.
• The user-agent is Mozilla/5.0 (compatible; SitemapRedirectAuditor/1.0; +https://apify.com/velveteen_driver/sitemap-redirect-auditor), and robots.txt rules for SitemapRedirectAuditor are honoured.

Pricing

$0.0015 per URL checked (event url-checked, so $1.50 per 1,000 URLs). Only URLs that were actually fetched are charged: URLs blocked by robots.txt, on another website or refused, and the summary rows, are free. Example: a 500-page sitemap with internal links off = $0.75. Set a maximum charge per run in the run options if you like; the run stops when it is reached and the status message says so.

Data source and responsible use

• The only data source is the website you supply. Use it on websites you own or are authorised to audit.
• robots.txt is respected for every request, including sitemaps and internal links. robots.txt is not permission to access a site, so make sure you are allowed to audit it.
• Not affiliated with any search engine. "Noindex" and "canonical" findings describe the page markup only; they don't report how any search engine has actually indexed the site.