XML Sitemap URL Extractor

---

🎯 What this scrapes

Give it a list of site URLs and it discovers every sitemap the site publishes — first by trying /sitemap.xml directly, then by parsing Sitemap: lines out of robots.txt (a 200 response on either is not treated as proof of a real sitemap; the Actor checks the Content-Type and looks like real sitemap XML before it commits to parsing it). Sitemap index files (<sitemapindex>) are expanded recursively up to a configurable depth, gzip-compressed sitemaps (.xml.gz) are decompressed transparently, and redirects are followed. Every leaf <url> entry becomes one dataset row tagged with the sitemap file and source site it came from.

🔥 What we handle for you

• 🛡️ Browser fingerprint rotation — curl-cffi impersonates real Chrome / Firefox / Safari TLS handshakes so the target sees a browser, not Python.
• 🌐 Residential proxy rotation via Apify Proxy — fresh session and exit IP on every block.
• 🔁 Retries with exponential backoff on 408 / 429 / 5xx — up to 5 attempts per page, Retry-After honoured.
• 🧱 Rate-limit-aware pacing — when the target pushes back, we slow down instead of getting banned.
• 🧊 Clean, typed dataset rows — Pydantic-validated, ISO-8601 timestamps, stable IDs, JSON / CSV / Excel export straight from the Apify Console.
• 💰 Pay-Per-Event pricing — you only pay for results that hit your dataset. No data, no charge.

💡 Use cases

• SEO content audits — pull a site's full published-URL inventory in one pass, no crawling required.
• Site migration checks — diff sitemap output before/after a replatform to confirm nothing silently dropped.
• Competitive research — see how many pages a competitor publishes and how often they update (lastmod, changefreq).
• Freshness monitoring — schedule runs and diff lastmod values to catch stale or newly-published pages.

⚙️ How to use it

1. Click Try for free at the top of the page.
2. Fill in the input form — most fields have sensible defaults.
3. Click Start. Output streams into the run's dataset.
4. Export from Storage → Dataset as JSON, CSV, or Excel — or fetch via the API.

📥 Input

Field	Type	Required	Default	Notes
siteUrls	array	yes	['https://www.cloudflare.com', 'https://apify.com']	Root or homepage URLs to scan, one per line. For each, the Actor tries /sitemap.xml directly and parses Sitemap:…
maxUrlsPerSite	integer	no	5000	Stop emitting rows for one site after this many URLs. Also lets the Actor stop fetching further sitemap files for that…
maxSitemapDepth	integer	no	3	How many levels of nested <sitemapindex> to follow. A depth-1 index pointing at leaf sitemaps is depth 1; an index of…
maxTotalUrls	integer	no	20000	Hard ceiling on rows emitted across every site in this run, regardless of maxUrlsPerSite. Bandwidth is our biggest…
proxyConfiguration	object	no	{'useApifyProxy': True}	Sitemaps are public, unauthenticated files, so this is mainly IP hygiene and rate-limit avoidance across many sites in…

Example input

{
  "siteUrls": [
    "https://www.cloudflare.com",
    "https://apify.com"
  ],
  "maxUrlsPerSite": 10,
  "maxSitemapDepth": 3,
  "maxTotalUrls": 20,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

📤 Output

Every row is one dataset item.

Field	Type	Notes
loc	string	The URL listed in the sitemap <loc> element.
lastmod	['string', 'null']	<lastmod> value, verbatim (ISO-8601 date or datetime per the sitemap protocol).
changefreq	['string', 'null']	<changefreq> value (e.g. daily, weekly) when the sitemap publishes one.
priority	['number', 'null']	<priority> value (0.0–1.0) when the sitemap publishes one.
sitemap_url	string	The specific sitemap file this row was read from (the leaf file, after index expansion).
source_site	string	The original input site URL this sitemap was discovered for.

Example output

{
  "loc": "https://apify.com/about",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://apify.com/sitemap/pages.xml",
  "source_site": "https://apify.com"
}

💰 Pricing

Pay-Per-Event — you pay only when these events fire:

Event	USD	What it is
actor-start	$0.2	One-off warm-up charge per run
result	$0.002	Per dataset item

Example: 1 000 results at the rates above ≈ $2.20. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

🚧 Limitations

• We read what a site's sitemap files declare — we don't crawl or discover pages a site doesn't list in a sitemap.
• The standard sitemap protocol caps a single leaf file at 50,000 URLs; large sites split across many files via a sitemap index, which we follow up to maxSitemapDepth.
• maxUrlsPerSite and maxTotalUrls are hard caps by design (bandwidth control) — raise them if you need a fuller export, within the input limits.

❓ FAQ

What if a site has no sitemap.xml or robots.txt Sitemap entry?

That site contributes zero rows and the run still SUCCEEDS — the status message reports how many of the input sites actually yielded a sitemap. This is a completed search that found nothing, not a failure.

Does it handle sitemap index files?

Yes — <sitemapindex> files are expanded recursively up to maxSitemapDepth. A visited-URL set stops circular or self-referencing indexes from looping.

Does it handle gzip-compressed sitemaps?

Yes, transparently — both .xml.gz URLs and sitemaps served with Content-Encoding: gzip (or just gzip magic bytes regardless of headers) are decompressed before parsing.

What happens if one site is unreachable or its sitemap is malformed?

That one site or sitemap file is skipped with a logged warning; the rest of the run continues. One bad target never fails the whole run.

Why did I get an HTML page instead of sitemap data for a site?

Some sites return a 200 HTML page (a redirect landing page, a CMS 404-as-200) at the sitemap path. The Actor checks the Content-Type and the actual root XML tag before parsing — an HTML shell is logged and skipped, not force-parsed as XML.

💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an issue on the Actor's Issues tab on Apify Console — we ship fixes weekly and we read every report.

---