Sitemap URL Extractor — Nested Indexes, Gzip & Incremental lastmod

Give it a domain. Get back every URL the site publishes, with lastmod, changefreq and priority, and the sitemap each URL came from.

You do not need to know where the sitemap lives. The Actor reads robots.txt first, and if the site declares nothing it probes the usual locations. Then it follows the whole tree: sitemap indexes pointing at other indexes, .xml.gz archives, plain-text sitemaps, and RSS or Atom feeds when that is all a site has.

What it handles that simple extractors do not

Nested sitemap indexes	Follows index → index → urlset to a configurable depth, instead of stopping at the first file.
Gzip	Unpacks .xml.gz by inspecting the bytes, so it works even when the server sends the wrong Content-Type.
RSS / Atom	Falls back to feeds, the only "sitemap" many blogs and news sites publish.
Plain-text sitemaps	A file of one URL per line is valid per the sitemap protocol, and is read as such.
Duplicates	Real sitemaps repeat URLs. They are deduplicated, and the number skipped is reported, not hidden.
Incremental extraction	lastmodAfter returns only pages changed since a date.
Honest limits	If a cap or a depth limit stops the run early, it says so in the log and in the SUMMARY record.

Incremental crawling — the reason to use this

Re-crawling a 50,000-page site to find the 40 pages that changed is a waste. Set lastmodAfter to your last run's date and you get just the new and updated URLs, which you can feed straight into a content crawler:

{
  "startUrls": ["example.com"],
  "lastmodAfter": "2026-07-01"
}

URLs with no lastmod are dropped when this filter is set — if a site publishes no dates, there is nothing to compare, and silently returning everything would defeat the point.

Input

{
  "startUrls": ["apify.com"],
  "lastmodAfter": "2026-07-01",
  "includePattern": "/blog/",
  "excludePattern": "/tag/|/author/",
  "maxUrls": 10000,
  "maxDepth": 3,
  "outputSitemapList": false,
  "minExpectedResults": 0
}

• startUrls — a bare domain (example.com), a full sitemap URL, a sitemap index, an .xml.gz file, or a feed. Mix them freely in one run.
• includePattern / excludePattern — case-insensitive regular expressions over the full URL. An invalid expression fails the run immediately with the reason, rather than silently matching nothing.
• outputSitemapList — return the site's sitemap structure instead of its URLs. Useful on a large site before you extract from it.
• minExpectedResults — fail the run if fewer records come back than expected. See Scheduling below.

Output

One record per URL:

{
  "url": "https://apify.com/store/apify/website-content-crawler",
  "lastmod": "2026-07-14T00:00:00.000Z",
  "lastmodRaw": "2026-07-14",
  "changefreq": "weekly",
  "priority": 0.8,
  "site": "https://apify.com",
  "sourceSitemap": "https://apify.com/sitemap/actors1.xml",
  "sitemapKind": "urlset",
  "depth": 1
}

With outputSitemapList: true you get one record per sitemap instead — its kind, how it was discovered, and how many child sitemaps and URLs it holds.

Every run also writes a SUMMARY record to the key-value store: sitemaps read, sitemaps failed, records pushed, URLs filtered, duplicates skipped, whether the cap was hit, and the full sitemap map.

Typical uses

• Feed a content crawler a precise URL list instead of letting it discover links, so you crawl what you meant to and nothing else.
• Incremental re-crawls — only pages changed since your last run.
• SEO audits — count indexable pages, find sections missing from the sitemap, spot duplicate entries and missing lastmod.
• Site migration checks — extract the URL list before and after a move and diff the two.
• Agent tooling — a fast, cheap way for an AI agent to see what a website actually contains before deciding what to read.

Scheduling and health checks

Set minExpectedResults to roughly what you expect. If a sitemap moves, gets renamed or breaks, the run fails loudly instead of quietly returning nothing — which is what you want from a monitor. Pair it with a Schedule and Apify's failure notifications.

Limits

• Sitemap files above 60 MB uncompressed are skipped with an explicit error, which is well beyond the 50 MB the sitemap protocol allows.
• maxDepth defaults to 3 levels of nesting; deeper indexes are reported as not followed rather than dropped silently.
• Only robots.txt and the sitemaps themselves are fetched. The Actor never renders pages, so it needs no browser, no proxy and no captcha solving.

Pricing

Pay per event: a small charge to start the run, then a per-URL charge. You pay for URLs returned, not for compute time.