Sitemap URL Extractor: sitemap.xml Parser, robots.txt & .gz

Every URL a site publishes, in one clean table, in seconds. Give this sitemap extractor a domain or a sitemap URL and get back every page URL listed in the site's XML sitemaps: one JSON row per URL with lastmod, changefreq and priority. It finds the sitemaps for you, follows sitemap indexes and unzips .gz files. $0.30 per 1,000 URLs; sites without a sitemap cost nothing.

• Finds sitemaps from robots.txt, then tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /sitemap.xml.gz
• Follows nested sitemap indexes, decompresses .gz, and reads plain-text sitemaps and RSS/Atom feeds
• Change monitoring: lastmodAfter returns only pages added or updated since a date
• Filters: include/exclude regex, same-domain only
• Polite: respects robots.txt, waits between requests to each host, retries 429/5xx with back-off
• Lightweight: downloads only the XML files, no browser

Quick start

1. Click Start.
2. Add a domain (https://htmx.org) or a sitemap URL (https://example.com/sitemap.xml) to Websites or sitemap URLs.
3. Optional: set Max URLs, a regex filter or a "modified after" date.
4. Run, then export the dataset as JSON, CSV or Excel.

{
  "startUrls": ["https://htmx.org"]
}

Sample output

From our test run on htmx.org (188 URLs in 0.4 s, local test). First 5 rows:

url	lastmodIso	path	sitemapUrl
https://htmx.org/	null	/	https://htmx.org/sitemap.xml
https://htmx.org/CVE-2026-3682-1/	null	/CVE-2026-3682-1/	https://htmx.org/sitemap.xml
https://htmx.org/api/	null	/api/	https://htmx.org/sitemap.xml
https://htmx.org/attributes/	null	/attributes/	https://htmx.org/sitemap.xml
https://htmx.org/attributes/hx-boost/	null	/attributes/hx-boost/	https://htmx.org/sitemap.xml

lastmodIso is null where the site's sitemap gives no lastmod for that page. The per-site report saved as OUTPUT in the key-value store:

{
  "site": "https://htmx.org",
  "sitemapsFetched": 1,
  "sitemapsFailed": 0,
  "urlsFound": 188,
  "urlsKept": 188,
  "discoveredVia": "robots.txt",
  "errors": []
}

On a site with a sitemap index, sitemapsFetched counts every child sitemap followed (in our docs.apify.com test: found via robots.txt, index followed, 1,500 URLs in 1.3 s).

Only pages changed since a date

{
  "startUrls": ["https://htmx.org"],
  "lastmodAfter": "2026-01-01",
  "dropUrlsWithoutLastmod": true
}

Result: 7 of 188 URLs kept (0.3 s, local test), sorted by date:

url	lastmodIso
https://htmx.org/essays/markdown-in-src/	2026-09-21T00:00:00.000Z
https://htmx.org/essays/working-with-ai/	2026-06-29T00:00:00.000Z
https://htmx.org/essays/universities-and-ai/	2026-06-11T00:00:00.000Z
https://htmx.org/essays/code-is-cheap/	2026-06-04T00:00:00.000Z
https://htmx.org/essays/mcp-apps-hypermedia/	2026-03-18T00:00:00.000Z
https://htmx.org/essays/yes-and/	2026-02-27T00:00:00.000Z
https://htmx.org/essays/paris-2024-olympics-htmx-network-automation/	2026-01-16T00:00:00.000Z

Without dropUrlsWithoutLastmod, URLs that have no lastmod are kept too (105 rows here), because the sitemap doesn't say when they changed.

Output fields

Field	Meaning
url, host, path	The page URL and its parts
lastmod / lastmodIso	As written in the sitemap / normalised to ISO 8601 UTC
changefreq, priority	From the sitemap, if present
imageCount, alternateCount	Image entries and hreflang alternates
sitemapUrl, sitemapDepth	Which sitemap the URL came from, and how deep in the index

Pricing

Event	Price (USD)
Actor start	$0.00005 per GB of memory (default 512 MB = $0.00005)
URL extracted (url-extracted)	$0.0003 ($0.30 per 1,000 URLs)

You pay only for URL rows delivered. Sites with no sitemap, and sitemap URLs that return 404, are reported in OUTPUT and not charged. Set "Max cost per run" to cap spending.

Use cases

• Get all URLs of a website (as listed in its sitemaps) for an SEO audit or content inventory
• Change monitoring: "what pages did this site add or update this month?" with lastmodAfter
• Migration checklists, and finding stale pages by lastmod
• Feed a crawler, scraper or RAG pipeline with a clean URL list
• Give an AI agent a site map before it browses

Input options

Field	Default	What it does
startUrls	(required)	Domains or sitemap URLs
maxUrls	10000	Total URL limit (0 = no limit; hard cap 1,000,000)
maxUrlsPerSite	0	Per-input limit (0 = none)
maxSitemaps	500	Safety limit on sitemap files per site
includePatterns / excludePatterns	[]	Regexes matched against each URL
lastmodAfter	(none)	ISO date; URLs with an older lastmod are dropped
dropUrlsWithoutLastmod	false	With a date filter, also drop URLs without lastmod
sameDomainOnly	true	Ignore URLs and child sitemaps on other hosts
respectRobotsTxt	true	Skip sitemap files robots.txt disallows
requestDelayMs	300	Delay between requests to one host

Use with AI agents (MCP)

https://mcp.apify.com/?tools=saif-tools/sitemap-url-extractor

Prompt: "List all blog URLs on example.com changed since 2026-06-01." The agent passes startUrls, includePatterns: ["/blog/"] and lastmodAfter. Keep maxUrls small for agent calls.

FAQ

How do I get all URLs of a website?

Enter the domain and run. The Actor finds the sitemaps (via robots.txt or the usual locations) and returns every URL they list. It doesn't crawl pages, so URLs missing from the sitemaps aren't included. To crawl pages and check their links, use the Broken Link Checker.

What if a site has no sitemap?

The run reports "No sitemap found" for that site in OUTPUT, and you aren't charged for it. In our tests, example.com had no sitemap and a sitemap URL that returned 404 was logged; neither was charged.

Can I export to CSV or Google Sheets?

Yes. Download the dataset as CSV, Excel or JSON from the Storage tab, or send it to Google Sheets with Apify's integrations.

How do I use it from n8n or Make?

Use the Apify node in n8n or the Apify app in Make (Zapier works too), choose "Run Actor", pick saif-tools/sitemap-url-extractor, then read the dataset items in the next step. Setup guides: https://docs.apify.com/platform/integrations

Can it handle huge sites?

It follows sitemap indexes up to maxSitemaps files per site and stops at maxUrls. Single sitemap files larger than 60 MB uncompressed are skipped (the protocol limit is 50 MB).

Does it use proxies or log in?

No. It reads public files only.

SEO toolkit

Pair it with the Broken Link Checker: the sitemap list shows what you tell search engines exists; the link checker shows what's actually broken on your pages.