Sitemap Extractor

Recursively crawls any sitemap.xml — including a sitemap index pointing at hundreds of child files — into a flat list of URLs. Over plain HTTP, no browser, no login: a sitemap is a file websites publish specifically so machines can enumerate their pages.

Handles both sitemap shapes, and both compression states

The sitemap protocol defines two document types, and large sites almost always use both:

• <sitemapindex> — a list of child sitemap files. This Actor traverses it breadth-first, recursing up to 4 levels deep (some very large sites nest indexes inside indexes).
• <urlset> — the actual page list, each entry optionally carrying lastmod, changefreq and priority.

Either may be gzip-compressed (sitemap.xml.gz) per the spec. This Actor detects that from the response's own magic bytes rather than trusting the URL extension or the Content-Type header, since servers are inconsistent about both.

What you get per URL

{
  "url": "https://techcrunch.com/press-release/example/",
  "path": "/press-release/example/",
  "lastmod": "2026-08-22",
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://techcrunch.com/sitemap-page-1.xml"
}

lastmod is passed through exactly as the site writes it — some use a bare date, others full ISO-8601 with time. sitemap_url tells you which child file a URL came from, useful when a site splits its sitemap by section (/blog/, /products/) or by date.

Filtering without extra requests

urlContains and pathPrefix filter client-side as the crawl streams in, so you can point this at a site's root sitemap index and pull out just /blog/ or /products/ without writing a separate crawler for each section.

Input

Field	Type	Notes
startUrl	string	A sitemap or sitemap index URL. Usually /sitemap.xml or /sitemap_index.xml, and referenced in /robots.txt if you need to find it.
urlContains	string	Keep only URLs containing this substring.
pathPrefix	string	Keep only URLs whose path starts with this.
maxItems	integer	Cap on URLs returned. 0 = unlimited.
maxSitemaps	integer	Cap on sitemap files fetched during traversal — independent of maxItems, since an index can have hundreds of children.
proxyConfiguration	object	Sitemaps have no anti-bot layer; leave this off unless a specific target needs it.

Known limits

• 50 MB uncompressed cap per file, per the sitemap protocol's own limit — a compliant sitemap never exceeds this, so hitting it means the file is non-standard.
• lastmod format varies by site and is passed through unparsed rather than guessed at, so downstream code should handle both a bare date and a full timestamp.
• Nested indexes stop at depth 4. No real-world site has needed to go deeper in testing, but a pathological case would be truncated rather than looping forever.

Local development

pip install -r requirements.txt
python test_local.py "https://apify.com/sitemap.xml" --max 20 --out sample_output.json
python test_local.py "https://techcrunch.com/sitemap_index.xml" --contains press-release --max 10

sample_output.json in this folder is real output from a live run, kept so the schema can be reviewed without running anything.