Sitemap URL Extractor & Sitemap Validator

Give it domains (or sitemap URLs). It finds every sitemap and returns one row per unique page URL with lastmod, changefreq, priority and validation warnings.

Use cases

• Build a URL list for a crawl, a migration or a content audit.
• Find stale pages by lastmod.
• Check that a new sitemap is valid before submitting it to search engines.
• Feed page lists to RAG or monitoring pipelines.

What it does

• Discovers sitemaps from robots.txt Sitemap: lines, falling back to /sitemap.xml and /sitemap_index.xml (also when every sitemap robots.txt lists is broken; the broken ones are still reported).
• Follows sitemap indexes, reads gzip (.xml.gz) and plain-text sitemaps, removes duplicate URLs.
• Flags invalid lastmod dates, invalid changefreq, out-of-range priority, and URLs on a different host.
• Skips malformed entries (a <loc> or robots.txt Sitemap: line that is not a valid absolute URL) and reports each as a free row, while keeping every valid URL in the same file.
• Reports each broken sitemap (404, invalid XML, timeout) as its own row so you can fix it.
• Identifies itself with a named user agent and waits between sitemap files (default 0.5 s).
• A SUMMARY record per run: URLs per domain, capped with the reason (maxUrlsPerDomain, maxSitemapsPerDomain or charge_limit), sitemaps read and failed, and whether the fallback was used.
• Rows are saved in batches of up to 100; when your maximum charge for the run is reached, only the rows that were paid for are saved and the run stops.

Pricing (pay per event)

Event	Price
url (one unique URL row)	US$0.0005

Read errors are billed like a URL row: a sitemap that can't be fetched or parsed (404, invalid XML, timeout), and a domain where no readable sitemap is found, are each billed as one url event (US$0.0005), following Apify's pay-per-event guidance for input that looks valid but leads to an error. A domain is billed at most once for "no readable sitemap found", and not at all for it when one of its broken sitemaps was already billed. Still free: blocked or private addresses and DNS failures (nothing is fetched), invalid entries skipped inside a good sitemap, and unexpected errors on our side. Each row says not charged when it is free. Use maxUrlsPerDomain to cap spend.

Each run also has Apify's run-start charge of US$0.00005 per GB of run memory (US$0.00005 at the default 512 MB), charged when the run starts, even if every domain fails.

How to use

1. Open the actor in Apify Console and fill in the input form (or paste the JSON example below into the JSON editor).
2. Optional: set a maximum charge for the run under the run options to cap what a run can cost.
3. Click Start. Results appear in the Output tab as they are produced; export them as JSON, CSV, Excel or HTML.

You can also start runs from the Apify API or CLI, run the actor on a schedule (for example weekly), and send results to other apps with Apify integrations and webhooks.

Input example

{"domains": ["docs.apify.com"], "maxUrlsPerDomain": 1000}

Output example

{"domain": "https://docs.apify.com", "url": "https://docs.apify.com/api", "lastmod": null, "changefreq": null, "priority": null, "sitemap": "https://docs.apify.com/sitemap_base.xml", "sitemapKind": "urlset", "warnings": [], "status": "ok"}

FAQ

Does it crawl the website? No. It reads only robots.txt and the sitemap files, which sites publish for machines.

What if a site has no sitemap? You get one row saying no readable sitemap was found, with the locations tried. It is billed as one URL row (US$0.0005); a domain that is blocked, private or does not resolve is free.

How do I control cost? Set maxUrlsPerDomain. 1,000 URLs cost US$0.50. Each run adds Apify's run-start charge (US$0.00005 per GB of memory, US$0.00005 at the default 512 MB).

Can I give a sitemap URL directly? Yes, put the sitemap URL in domains or startUrls.

Limits: what this actor does not do

• It does not crawl pages or check that each URL responds (that is a link checker's job).
• No private or loopback targets; redirects are re-checked at every hop.
• Each sitemap file must download within 60 seconds and be at most 50 MB uncompressed (the sitemap protocol limit); otherwise it is reported as an error row (billed as one URL row) and the next one goes ahead.

Changelog

• October 8, 2026: a sitemap that can't be fetched or parsed, and a domain with no readable sitemap, are now billed as one URL row each (US$0.0005). Blocked, private and unresolvable targets stay free.
• 1.0 (October 2026): first public version.