Sitemap URL Extractor Plus - Sitemap Index, Gzip, robots.txt, lastmod

Extract every page URL from a website's sitemaps, reliably. Give it a site (or a sitemap URL) and get a flat, deduplicated dataset of URLs with lastmod, priority and changefreq. You are charged only per unique URL extracted ($0.0004 each, $0.40 per 1,000 URLs on the Free plan; paid Apify plans get lower tiers); sitemap files, errors, duplicates and filtered-out URLs are free. A dead site, a 404 sitemap or a malformed URL never fails the run: it is logged and listed in the SUMMARY record.

Quick start

Input (startUrls is the only field you need):

{ "startUrls": ["https://docs.apify.com"], "maxUrls": 20 }

Output, one row per unique URL (illustrative shape):

{ "url": "https://docs.apify.com/platform", "lastmod": "2026-09-30", "priority": 0.8, "changefreq": "weekly", "sitemapUrl": "https://docs.apify.com/sitemap.xml", "source": "https://docs.apify.com" }

Pricing event: url-extracted, $0.0004 per unique URL ($0.40 per 1,000 on the Free plan; paid Apify plans get lower tiers); sitemap files, errors, duplicates and filtered-out URLs are free.

What it handles

• Sitemap indexes, followed recursively (configurable depth).
• Gzip sitemaps (sitemap.xml.gz), detected by content, even when the server mislabels them.
• robots.txt discovery: pass a home page and Sitemap: lines are found automatically; falls back to /sitemap.xml, /sitemap_index.xml, /sitemap.xml.gz.
• Plain-text sitemaps (one URL per line).
• Dedupe across all sitemaps and all start URLs.
• Filters: include/exclude regex, modifiedSince / modifiedUntil (date or 7d), onlyChanged monitoring, maxUrls cap (also your spend cap).
• Retries on network errors; 404/403 sitemaps are skipped and reported.

Input

startUrls (sites or sitemap URLs; aliases sitemapUrls, urls, sitemaps are merged with it), maxUrls (default 20), includePattern, excludePattern, modifiedSince, modifiedUntil, includeMissingLastmod, onlyChanged, maxSitemapDepth (default 5), checkPageStatus (adds an HTTP statusCode per URL via HEAD, no extra charge).

Switching from another sitemap actor? Keep your saved input and change only the Actor ID. See the Switching section below.

Drop-in for the input of apify/sitemap-extractor: startUrls as [{"url": ...}], maxCrawlingDepth (same as maxSitemapDepth) and checkPageStatus work as-is; proxyConfiguration, proxyRotation and maxRequestRetries are accepted and ignored, and no proxy is required.

Output per URL

url, lastmod, daysSinceModified, priority, changefreq, sitemapUrl (file it came from), source (your start URL). Missing fields are null.

Sitemap changed URLs: recently updated pages and lastmod filter

Use the sitemap lastmod filter to list only recently updated pages:

• modifiedSince: a date (2026-10-01) or a relative age (7d, 2w, 3m, 1y). modifiedUntil sets the upper bound.
• URLs with no lastmod are skipped by default (counted in skippedNoLastmod); set includeMissingLastmod: true to keep them.
• Every row has a status (ok, error or skipped); unreadable sitemaps appear as free error rows with a reason. The SUMMARY record holds counts (rows, ok, error, skipped) that equal the dataset rows.
• Each row gets daysSinceModified. The RUN_SUMMARY key-value record holds { total, kept, skippedNoLastmod }.

Sitemap changed URLs on a schedule (onlyChanged): the actor stores each URL's last seen lastmod in a named key-value store per site (sitemap-state-<host>). Every later run outputs only URLs that are new or whose lastmod changed, so a daily Apify schedule works as a sitemap change monitor and you pay only for changed URLs. The first run outputs all URLs as the baseline. To reset, delete that named store.

{ "startUrls": ["https://example.com"], "modifiedSince": "7d", "onlyChanged": true, "maxUrls": 1000 }

Use cases

SEO audits, content inventories, crawl seeding, change monitoring (modifiedSince, onlyChanged), site migration checklists, feeding a crawler or scraper a clean URL list.

FAQ

Does it follow a sitemap index (nested sitemaps)?

Yes. A <sitemapindex> is followed recursively down to maxSitemapDepth levels (default 5, max 10; maxCrawlingDepth is accepted as the same setting). Each sitemap file is read once. If the limit cuts off nested sitemaps, the SUMMARY record lists a "max sitemap depth reached" problem.

Can it read .xml.gz sitemaps?

Yes. A response that starts with the gzip header is decompressed whatever the URL or Content-Type says, so sitemap.xml.gz and mislabelled files both work, including gzipped files listed inside a sitemap index.

Do I need to know the sitemap URL? How does robots.txt discovery work?

No. Give a site such as https://example.com. The Actor reads its robots.txt and follows every Sitemap: line. If none is found it tries /sitemap.xml, and if that yields no URLs, /sitemap_index.xml and /sitemap.xml.gz. If you give a sitemap URL directly (ending in .xml, .txt, .gz or containing "sitemap"), robots.txt is skipped.

How do I get only URLs changed recently (lastmod filter)?

Set modifiedSince to a date (2026-10-01) or a relative age (7d, 2w, 3m, 1y); modifiedUntil sets the upper bound. URLs with no lastmod are skipped unless includeMissingLastmod is true. For a schedule, add onlyChanged to output only new or changed URLs on each run.

What does it cost?

$0.0004 per unique URL ($0.40 per 1,000 on the Free plan; paid Apify plans get lower tiers) extracted. Sitemap files, duplicates, errors and URLs removed by your filters are free; maxUrls caps both the output and your spend.

Honest limits

• Only fetches robots.txt and sitemap files that the site publishes for crawlers. It does not crawl pages, log in, or use proxies. Sites that block the request (403) are skipped.
• Sites without a sitemap return nothing; this is not a link crawler.
• Very large sites: raise maxUrls; each extracted URL is billed.

Switching from another sitemap actor

This Actor accepts the input field names of these Store actors, so a saved task or API call needs only the Actor ID changed. Field names were read from each actor's public input schema on 2026-10-21; output rows are this Actor's own shape (url, lastmod, priority, changefreq, sitemapUrl, source), so check any code that reads the dataset.

You used	Your input field	Here
apify/sitemap-extractor	startUrls ([{"url": ...}] or strings)	works as-is
apify/sitemap-extractor	maxCrawlingDepth	same as maxSitemapDepth
apify/sitemap-extractor	checkPageStatus	works as-is (adds statusCode, no extra charge)
apify/sitemap-extractor	proxyConfiguration, proxyRotation, maxRequestRetries	accepted, ignored (no proxy needed)
onescales/sitemap-url-extractor	sitemapUrls	works as-is (merged with startUrls)
pink_comic/sitemap-url-extractor	urls, maxResults, modifiedSince	urls and modifiedSince as-is; maxResults = maxUrls; autoDiscover accepted (discovery is always on); includeImages is not supported
mikolabs/sitemap-url-extractor	sitemapUrls, maxDepth, filterUrlPattern	maxDepth = maxSitemapDepth; filterUrlPattern = includePattern; requestTimeoutSecs accepted, ignored

Not affiliated with any of those actors. Pricing here: $0.0004 per unique URL, nothing for sitemap files, errors or duplicates.

Use from Claude, Cursor or any MCP client

Apify's MCP server can expose this actor as a tool. Use this URL (format from the Apify MCP docs, ?tools= takes a comma-separated list of actors):

https://mcp.apify.com/?tools=pennydata/sitemap-extractor-plus

Client config (mcpServers format; sign in with OAuth when the client asks):

{"mcpServers": {"apify": {
  "url": "https://mcp.apify.com/?tools=pennydata/sitemap-extractor-plus"
}}}

Example prompt: "List every URL in the sitemap of docs.apify.com, limited to 50 URLs."

Related actors

• Shot PDF: capture screenshots or PDFs of the URLs you extracted.
• Website Tech Stack Detector: checks the technology behind the site you mapped.
• RSS/Atom Feed Reader (bulk): reads the site's feeds when it has them.

Not affiliated with sitemaps.org or any site you extract from. Written with the assistance of an AI (Claude Sonnet 5.5).