Sitemap Extractor & Broken Link / Status Checker
Pricing
from $0.10 / 1,000 url extracteds
Extract every URL from a website's sitemaps (robots.txt discovery, sitemap indexes, .xml.gz, hreflang, images) and check each page's HTTP status, redirect chain and response time. Finds broken pages and redirects for SEO audits and site migrations.
A domain (example.com or https://example.com) to discover its sitemaps, or a direct sitemap URL (https://example.com/sitemap.xml, .xml.gz or .txt). Up to 100 entries.
[ "https://crawlee.dev"]Request every URL (HEAD, then GET if needed) and add status code, final URL, redirect chain, response time, content type and a broken flag. Turn off to only extract URLs from the sitemaps.
Follow up to 10 redirects and report the final status. When off, a redirect is reported as its own 3xx status with the target in the redirect chain.
Keep only URLs whose sitemap is on or after this date. Use a date (2025-01-31) or a relative age (7 days, 2 weeks, 6 months). URLs without a are skipped when this is set.
Keep only URLs that match this regular expression (case-insensitive). Example: /blog/
Drop URLs that match this regular expression (case-insensitive). Example: .(pdf|jpg)$|/tag/
Stop after this many URLs (after filters and de-duplication). Each URL is one result, and the limit also caps the cost.