Sitemap Extractor Done Right

Give it a sitemap URL, get back one clean record per URL: loc, lastmod, changefreq, priority. Handles gzip, follows sitemap index files into their children with a depth cap so a nested index tree can never run away, and filters by URL pattern or by last-modified date. Built because the current store leader for this sits at 3.2 stars on real demand, which is a gap this Actor is built to close.

What it does

• Fetches each sitemapUrls entry with a plain HTTP GET.
• Detects gzip automatically by sniffing the response's magic bytes, so a literal .xml.gz file and a server that transfer-encodes a plain .xml body both just work, no configuration needed.
• Parses XML namespace-agnostically: whatever namespace URI (or none) a sitemap declares, <url>, <loc>, <lastmod> etc. are matched by local tag name, not by exact namespace string. Extra vendor tags (image, video, news extensions) are present in plenty of real sitemaps and are ignored cleanly rather than tripping the parser.
• When a file is a sitemap index (<sitemapindex>), follows its child <sitemap><loc> entries and processes each one the same way, recursively, up to maxDepth levels.
• Filters by urlPattern (regex, or a plain substring if the regex does not compile) and by respectLastmodAfter (a date floor on <lastmod>).
• Dedupes URLs across the entire run: the same URL discovered twice, whether from two different sitemap files or two different sitemapUrls entries, is written and charged once.
• Writes a summary record per sitemapUrls entry no matter what happened: clean run, partial run, or failure. status is one of ok, error (malformed or non-sitemap XML), too-many-nested (nesting exceeded maxDepth before any URL could be found), too-many-sitemaps (the entry has more sitemap files than maxSitemapFiles allows, before any URL could be found), or download-failed (the file itself could not be fetched).

What it deliberately does not do

• No content fetching. This Actor reads only the sitemap XML the site published. It never fetches the pages the sitemap lists.
• No runaway recursion or fan-out. Two independent guards, both enforced in code, not left to trust: maxDepth stops a sitemap index that points to indexes that point to indexes, and maxSitemapFiles stops a site with many shallow sitemap files (per-category or per-day sitemaps are common) from racking up one fetch and one charge per file even when maxUrlsPerSitemap is small, since that cap only counts URL records, not files visited. A cycle guard also stops an index that points back at itself. Any of the three stopping a branch shows up in the summary instead of the run hanging.
• No invented fields. lastmod, changefreq, and priority come from the sitemap or are null. A respectLastmodAfter filter excludes URLs with no <lastmod> at all, since there is nothing to verify them against; that is documented here, not a silent surprise.

Sitemaps and machine access

A sitemap is a file a site owner publishes at a predictable, public URL specifically so it can be parsed by software, per the sitemaps.org protocol that Google, Bing, and every other search engine already read the same way. This Actor processes only sitemap URLs you supply; it is not choosing what to crawl and it is not reading page content, only the index file the site itself published for exactly this purpose.

Output example

Real records from https://squareup.com/sitemap.xml, a live sitemap index whose children include both gzipped leaf sitemaps and further nested sitemap indexes:

A url record, from a gzipped child (sitemap.xml.gz, decompressed automatically):

{
  "recordType": "url",
  "sourceSitemapUrl": "https://squareup.com/jp/ja/sitemap.xml.gz",
  "url": "https://squareup.com/jp/ja/townsquare/omino",
  "lastmod": "2026-08-25T03:00:40.493Z",
  "changefreq": "weekly",
  "priority": "0.5",
  "discoveredAt": "2026-08-25T03:29:39+00:00"
}

The matching summary record for the top-level entry:

{
  "recordType": "summary",
  "url": "https://squareup.com/sitemap.xml",
  "status": "ok",
  "urlsFound": 18223,
  "sitemapsProcessed": 17,
  "error": null
}

Input

Field	Meaning
sitemapUrls	Sitemap files to parse: plain .xml, gzipped .xml.gz, or a sitemap index. Required.
maxUrlsPerSitemap	Total URL cap per sitemapUrls entry, shared across every child sitemap discovered under it. Default 5000, max 100000.
followSitemapIndexes	Follow child sitemaps listed by a sitemap index. Default on; off returns zero URLs for an index-only entry.
urlPattern	Keep only matching URLs. Tried as a regex first, falls back to a plain substring match if it does not compile.
maxDepth	Levels of nested sitemap indexes to follow. The entry itself is depth 1. Default 3.
maxSitemapFiles	Total sitemap FILES (not URL records) one entry may fetch, across its whole nested tree. Independent of maxUrlsPerSitemap, which only counts URL records. Default 500, max 5000.
respectLastmodAfter	YYYY-MM-DD. Keeps only URLs whose <lastmod> is on or after this date; URLs with no <lastmod> are excluded once this is set.

Typical uses

• Pull every URL a site has published, for a crawl budget or a content audit, without writing an XML parser.
• Feed a downstream scraper a clean, deduped, pattern-filtered URL list instead of every URL on the site.
• Track what changed recently with respectLastmodAfter on a schedule.
• Sanity-check a site's own sitemap: too-many-nested and error statuses surface a badly configured sitemap tree immediately.

Fair pricing

Pay per URL record returned and per sitemap file successfully fetched and parsed (index files and leaf files both count), once pay-per-event pricing is enabled. A sitemap that fails to download or does not parse as valid sitemap XML costs nothing. No subscription.