Sitemap Extractor & Broken Link / Status Checker

Get every URL from a website's sitemaps and check each one's HTTP status, redirects and response time in one run. Enter a domain and the actor finds its sitemaps through robots.txt (or /sitemap.xml and /sitemap_index.xml). It follows sitemap indexes, reads .xml.gz and plain-text sitemaps, and keeps lastmod, changefreq, priority, image counts and hreflang alternates. It then checks every page: status code, final URL, the full redirect chain, content type, response time and a simple isBroken flag.

One broken child sitemap never fails the run. It's listed in the run summary with the reason, and you still get every URL from the other sitemaps. Results come as JSON, CSV or Excel, over the API, or from AI agents through the Apify MCP server.

What it does

• Finds sitemaps for you: reads Sitemap: lines in robots.txt, then tries /sitemap.xml and /sitemap_index.xml. You can also enter sitemap URLs directly.
• Reads every common format: <urlset>, nested <sitemapindex> files (up to 5 levels deep, 500 files per run), gzip (.xml.gz), plain-text sitemaps (one URL per line), the Google image extension and xhtml:link hreflang alternates.
• Tolerates broken XML: an unescaped & in a URL doesn't stop the parser. A file that's cut off still gives you every complete entry, and the summary marks that file as partial.
• Checks status politely: HEAD first, then GET when a server rejects HEAD. It follows up to 10 redirects, retries 429 and 5xx with backoff and honours Retry-After. It never sends more than 2 requests at once to one host, and its User-Agent names the actor.
• Filters: include or exclude regex, lastmod after a date or a relative age ("30 days"), and a maximum number of URLs. Duplicate URLs across sitemaps are removed.

Use cases

• SEO audits: find every 404, 410 and 5xx page listed in your sitemap, plus redirected URLs that shouldn't be in a sitemap at all.
• Site migrations and redirect checks: run it on the old site's sitemap with followRedirects on to see where each old URL ends up, or with it off to see the raw 301/302/308 and its target.
• Broken page monitoring: schedule a daily run and send isBroken: true items to Slack, email or a webhook through Apify integrations.
• Content inventory: export all URLs with lastmod to a spreadsheet, or keep only pages changed in the last 7 days to feed a crawler or an indexing tool.
• International SEO: check that every page lists its hreflang alternates.
• AI agents (Apify MCP, LangChain, OpenAI Agents): a small input and a flat, documented output. An agent can ask "which pages on example.com are broken?" and get structured answers.

Input

Field	Type	Default	Description
startUrls	string[]	["https://crawlee.dev"]	Domains (example.com) or sitemap URLs (.xml, .xml.gz, .txt). Up to 100.
checkStatus	boolean	true	Check each URL's HTTP status. Turn it off to only extract URLs.
followRedirects	boolean	true	Follow up to 10 redirects. When off, a redirect is reported as its own 3xx with the target in redirectChain.
lastmodAfter	string	none	Keep URLs whose <lastmod> is on or after this date: 2025-01-31, or 7 days, 2 weeks, 6 months. URLs without <lastmod> are skipped when this is set.
urlPattern	regex	none	Keep only URLs that match (case-insensitive).
excludeUrlPattern	regex	none	Drop URLs that match (case-insensitive).
maxUrls	integer 1–100,000	1000	Stop after this many URLs (after filters and de-duplication). This also caps the cost.
concurrency	integer 1–20	5	Checks running at the same time across all hosts. One host never gets more than 2 at once.

Example: blog posts changed in the last 30 days, with tag pages excluded:

{
    "startUrls": ["https://example.com"],
    "urlPattern": "/blog/",
    "excludeUrlPattern": "/tag/|/page/\\d+",
    "lastmodAfter": "30 days",
    "maxUrls": 5000
}

Example: extract URLs only (no requests to the pages):

{
    "startUrls": ["https://example.com/sitemap_index.xml"],
    "checkStatus": false,
    "maxUrls": 50000
}

Output

One dataset item per URL. This one is from a real run on apify.com, 2026-10-25:

{
    "url": "https://apify.com/alternatives/gmapsextractor-alternative",
    "sitemapUrl": "https://apify.com/sitemap/pages.xml",
    "sourceDomain": "apify.com",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "imageCount": 0,
    "hreflang": [],
    "statusCode": 200,
    "finalUrl": "https://apify.com/alternatives/outscraper-alternative",
    "redirectChain": [
        {
            "url": "https://apify.com/alternatives/gmapsextractor-alternative",
            "statusCode": 308,
            "location": "https://apify.com/alternatives/outscraper-alternative"
        }
    ],
    "responseTimeMs": 646,
    "contentType": "text/html",
    "isBroken": false,
    "checkMethod": "HEAD",
    "error": null,
    "checkedAt": "2026-10-25T07:05:11.653Z"
}

With hreflang alternates (from a multilingual site):

"hreflang": [
    { "lang": "en", "url": "https://www.icyanstudio.net/en" },
    { "lang": "ja", "url": "https://www.icyanstudio.net/ja" },
    { "lang": "zh-TW", "url": "https://www.icyanstudio.net/zh_tw" },
    { "lang": "x-default", "url": "https://www.icyanstudio.net/en" }
]

• statusCode, finalUrl, redirectChain, responseTimeMs, contentType, isBroken, checkMethod, error and checkedAt are present only when checkStatus is on.
• isBroken is true for status 400 and above, and for requests that failed (DNS, timeout, TLS, too many redirects). error says why.
• lastmod is exactly as written in the sitemap.

The dataset has two views: Status check and Sitemap data.

Run summary (key-value store OUTPUT)

Every run saves a summary: the sitemaps found and how each was found (robots.txt, default location, sitemap index), the status of each file (ok, partial or failed) with its errors, URL counts at each step, filter counts, counts by HTTP status, broken and redirected totals, and warnings. Example (shortened):

{
    "sitemapsFound": 19,
    "sitemapsRead": 2,
    "sitemapsFailed": 0,
    "sitemaps": [
        { "url": "https://apify.com/sitemap.xml", "foundVia": "robots.txt", "status": "ok", "type": "sitemapindex", "childSitemaps": 18, "errors": [] },
        { "url": "https://apify.com/sitemap/pages.xml", "foundVia": "sitemap index", "status": "ok", "type": "urlset", "urls": 1605, "errors": [] }
    ],
    "urlsReturned": 50,
    "statusCounts": { "200": 50 },
    "brokenCount": 0,
    "redirectedCount": 1,
    "stoppedAtMaxUrls": true,
    "warnings": ["Reached maxUrls (50); 17 more sitemap file(s) were not read."]
}

Pricing

Pay per event: you pay only for results.

Event	Price
URL extracted from a sitemap	$0.0005
URL status checked (only when checkStatus is on, and only for URLs actually requested)	$0.0005
Actor start	Apify's default

So 1,000 URLs cost $0.50 extracted only, or $1.00 extracted and checked. Set a maximum cost per run in Apify Console: the actor processes only as many URLs as that limit covers, then stops cleanly. You never pay for URLs that weren't returned.

Limits and notes

• Up to 100,000 URLs per run, 500 sitemap files per run, sitemap indexes nested up to 5 levels, and 50 MB per sitemap file (the sitemaps.org limit). Bigger files are read up to 50 MB and marked partial.
• One host gets at most 2 requests at a time, so a single-site sitemap is checked 2 URLs at a time. Speed depends on the site: in a test run, 50 apify.com URLs were checked in about 3 seconds.
• Some sites block automated requests (for example with Cloudflare bot protection) and return 403 or 429 to any non-browser client. Those URLs show up as broken with the real status code. The actor doesn't try to get around bot protection.
• robots.txt is used only to find sitemaps. URLs listed in the sitemap are checked whatever Disallow rules say, as in other SEO auditing tools.
• For safety, the actor only requests public http and https addresses. URLs and redirects that point at private, loopback or cloud-metadata addresses are refused and reported as broken with the reason.
• RSS and Atom feeds used as sitemaps are not supported yet.
• Only check sites you own or have permission to audit. Requests go out at a polite rate, but the target site's terms are your responsibility.

Feedback

Found a sitemap it can't read, or need a field? Open an issue on the actor's Issues tab with the sitemap URL.