Reliable Sitemap URL Extractor

Extract all URLs from a website's sitemap.xml. Enter a domain or sitemap URL and get a de-duplicated list with lastmod, changefreq and priority. Nested indexes, .xml.gz and plain-text sitemaps are followed, and URLs are recovered from malformed XML (flagged per row). A broken child sitemap doesn't stop the run: RUN_SUMMARY lists it with the reason. $0.10 per 1,000 URLs; sites with no sitemap cost $0.

Why this one: it finishes

Real-world sitemaps are messy. This Actor is built to keep going and tell you exactly what happened:

• Finds sitemaps automatically from robots.txt, then falls back to common paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, .gz, .txt)
• Follows nested sitemap indexes, with loop protection
• Reads gzipped sitemaps (.xml.gz or gzip-encoded responses) and plain-text sitemaps
• Recovers URLs from malformed XML (unescaped &, truncated files, junk before the root tag) and flags those rows
• A broken child sitemap (404, timeout, HTML page) is listed in the run summary; the rest of the site is still extracted
• An unreachable site or a site with no sitemap is reported in RUN_SUMMARY, and the run doesn't crash

Output

Dataset, one row per unique page URL: url, lastmod, changefreq, priority, sitemap, site, recovered_from_malformed_xml. Key-value store RUN_SUMMARY: per site, how sitemaps were found, sitemaps parsed and failed (with reasons), URLs, duplicates skipped, notes.

Pricing

$0.10 per 1,000 URLs (pay per event "url-extracted"). No platform-usage charges. Sites with no sitemap cost nothing. Set "max charge per run" in Apify to cap spend.

What it does not do

• It doesn't crawl pages or check each page's HTTP status; it only reads the sitemap files a site publishes for search engines. Use a crawler if a site has no sitemap.
• It doesn't bypass bot protection, logins or paywalls; such sites are reported as "no sitemap found" or as fetch errors.
• lastmod and the other fields are whatever the site declares.

Input

startUrls (domains or sitemap URLs), maxUrls (default 100,000 total), sameHostOnly, requestTimeoutSecs.

The example input runs shopify.com first, then github.blog, with Max URLs set to 5,000 (about $0.50). maxUrls is a total for the whole run and sites run in input order, so github.blog (more than 5,000 URLs in its sitemaps) stops when the run reaches 5,000. Its RUN_SUMMARY entry then shows a "stopped at maxUrls=…" note. That is expected.