Sitemap Extractor: Get All URLs from XML Sitemaps

A sitemap extractor for any website: get the full list of sitemap URLs as a clean dataset for SEO audits, site migrations, change monitoring or as the start list for a crawl.

Give it a website or a sitemap URL and get back every page URL the site publishes in its XML sitemaps, with lastmod, changefreq, priority, hreflang alternates and image counts. Sitemaps are discovered automatically from robots.txt, sitemap indexes are followed, and .xml.gz, plain-text, RSS and Atom sitemaps are read.

Quick start for AI agents and API users

• What it does: List every page URL a website publishes in its XML sitemaps (auto-found via robots.txt; indexes and .gz too).
• When to use it: For a site's full URL list or recently changed pages.
• Example input (the form's prefill; it finishes in under a minute):

{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 1000
}

• Example output: one real dataset row from run gTFrGDOf1X4rtFem0 (2026-10-23), with long fields shortened and empty fields left out. Fields not shown are listed in the output table below.

{
  "url": "https://crawlee.dev/blog",
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "sitemapType": "urlset",
  "site": "https://crawlee.dev",
  "scrapedAt": "2026-10-23T13:10:53Z"
}

• Cost (pay per event, Free-plan prices): result $0.0005. Failed items are not charged.
• From an AI agent: with the Apify MCP server, call call-actor with actor phaux/sitemap-url-extractor, or load it as its own tool with https://mcp.apify.com?tools=phaux/sitemap-url-extractor. Read results with get-dataset-items.

What does Sitemap URL Extractor do?

• Auto-discovery: reads the Sitemap: lines in robots.txt, then falls back to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.xml.gz.
• Follows sitemap indexes recursively (maxDepth, up to 10 levels).
• Reads every common format: .xml, .xml.gz, plain-text sitemaps, RSS 2.0 and Atom feeds, streamed up to the 50 MB protocol limit per file.
• Filters: include/exclude regex, "modified since", same-host only, and URL caps per site and in total.
• De-duplicates across all sitemaps and sites in the run, and returns one dataset item per unique URL.

Why use it?

• Fast: only sitemap files are downloaded, never the pages themselves. 8,000 URLs typically take a few seconds.
• Safe defaults: 1,000 URLs per site unless you raise the cap, and a time budget that finishes the run cleanly before its timeout, keeping everything saved so far.
• Pay only per URL saved: sitemap files, duplicates, filtered URLs and failed sitemaps are free.

Use cases: SEO audits and content inventories, finding new or updated pages (lastmodSince), seeding crawlers with URL lists, site migrations, and monitoring competitors' published pages.

How to use it

1. Add websites (https://example.com) or direct sitemap URLs to startUrls. With an empty input, the Actor runs a small example (crawlee.dev).
2. Optionally set includeUrlPatterns (for example "/blog/"), lastmodSince (for example "30 days"), and caps.
3. Set maxUrlsPerSite to 0 to get every URL of a large site.
4. Run the Actor and open the URLs dataset view, or download JSON, CSV or Excel.

Input

{
  "startUrls": ["https://crawlee.dev", "https://www.bbc.com/sitemaps/https-index-com-news.xml"],
  "includeUrlPatterns": ["/blog/"],
  "excludeUrlPatterns": ["/tag/"],
  "lastmodSince": "30 days",
  "sameHostOnly": false,
  "maxUrlsPerSite": 1000,
  "maxItems": 0,
  "timeBudgetSecs": 0
}

Field	Type	Default	Description
startUrls	array	["https://crawlee.dev"]	A website (https://example.com: sitemaps are discovered from its robots.txt, then /sitemap.xml) or a direct sitemap URL (.xml, .xml.gz, .txt, RSS/Atom). Leave empty to run the small example (crawlee.dev).
includeUrlPatterns	array	—	Optional. Keep only URLs matching at least one regular expression, e.g. '/blog/' or '\.html$'. Case-insensitive.
excludeUrlPatterns	array	—	Optional. Drop URLs matching any of these regular expressions, e.g. '/tag/' or '\?'.
lastmodSince	string	—	Optional. Keep only URLs whose
keepUrlsWithoutLastmod	boolean	true	Only used with 'Modified since'.
sameHostOnly	boolean	false	Drop URLs on other hosts than the start URL (www. is ignored).
includeAlternates	boolean	true	Add the xhtml:link hreflang alternates listed for each URL.
maxUrlsPerSite	integer	1000	Default 1,000 keeps runs quick and cheap. Set 0 for no limit (large sites can have hundreds of thousands of URLs).
maxItems	integer	0	0 = no limit. You are only charged for URLs saved.
maxSitemapsPerSite	integer	500	Safety cap on sitemap files read per site.
maxDepth	integer	5	How deep to follow nested sitemap indexes.
maxConcurrency	integer	3	Sites processed at once. Requests to one host are always sequential, at least 1 s apart (or robots.txt Crawl-delay).
timeBudgetSecs	integer	0	Stop reading new sitemaps after this many seconds and finish cleanly, keeping every URL saved so far. 0 = automatic: finish ~90 s before the run timeout so a run never times out.

Output example

One dataset item per unique URL:

{
  "url": "https://www.bbc.com/news/articles/crwyd02yz22vo",
  "lastmod": "2026-10-19T07:46:54Z",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "alternates": [],
  "sitemapUrl": "https://www.bbc.com/sitemaps/https-sitemap-com-news-1.xml",
  "sitemapType": "urlset",
  "site": "https://www.bbc.com",
  "scrapedAt": "2026-10-20T13:22:30Z"
}

Output fields

Field	Type	Description
url	string	Page URL listed in the sitemap.
lastmod	string or null
changefreq	string or null
priority	number or null
imageCount	integer	Number of image:image entries for the URL.
alternates	array	hreflang alternates: [{hreflang, href}].
sitemapUrl	string	Sitemap file the URL came from.
sitemapType	string	urlset, text, rss or atom.
site	string	Origin of the start URL.
scrapedAt	string	Run time (ISO 8601, UTC).

The run also saves SITEMAP_ERRORS (sitemaps or sites that couldn't be read, with a code: robots_disallowed, http_error, rate_limited, not_a_sitemap, too_large, no_sitemap_found, network_error, blocked_host, invalid_url, max_depth, time_budget) and RUN_STATS (per-site counts, how each site's sitemap was discovered, and stoppedByTimeBudget) in the key-value store.

How much does it cost?

Pay per event: you pay per URL saved, with no start fee. Apify platform usage is included.

Apify plan	Price per URL	Per 1,000 URLs
Free	$0.0005	$0.50
Bronze	$0.0003	$0.30
Silver	$0.0002	$0.20
Gold, Platinum, Diamond	$0.0001	$0.10

Example: the default 1,000 URLs per site cost $0.50 on the Free plan. Use maxItems to cap the cost of any run.

Use with AI agents, MCP and the API

• MCP: add https://mcp.apify.com?tools=phaux/sitemap-url-extractor to any MCP client and ask, for example, "list the blog URLs in crawlee.dev's sitemap".
• API: POST https://api.apify.com/v2/acts/phaux~sitemap-url-extractor/run-sync-get-dataset-items with {"startUrls": ["https://example.com"]} returns the URLs directly.
• Input and output fields are documented in the input and dataset schemas.

Data source and compliance

• The Actor reads only sitemaps the website publishes for crawlers, and the site's /robots.txt. It never fetches the pages themselves, never logs in and needs no API key.
• Each sitemap URL is checked against that host's robots.txt (RFC 9309) for the user agent PhauxSitemapBot. A disallowed sitemap is skipped and reported, and Crawl-delay is honoured.
• It identifies itself honestly (Mozilla/5.0 (compatible; PhauxSitemapBot/0.1; +https://apify.com/phaux/sitemap-url-extractor)), uses no proxies and no IP rotation, and doesn't try to get around bot protection. After an HTTP 429 it stops contacting that host for the rest of the run.
• Polite pacing: requests to one host are sequential and at least 1 second apart. Only public http(s) hosts are fetched, and every redirect hop is checked.
• No personal data: sitemaps list page URLs only.

Limits

• Only URLs that the site lists in its sitemaps are returned. Pages missing from the sitemaps aren't discovered (this Actor doesn't crawl links).
• 50 MB per sitemap file, maxSitemapsPerSite files per site (default 500, up to 5,000), and maxDepth index levels (default 5, up to 10).
• Very large sites can list hundreds of thousands of URLs; raise maxUrlsPerSite and the run timeout accordingly.

FAQ

• "no_sitemap_found"? The site doesn't advertise a sitemap in robots.txt and has none at the usual paths. Pass the sitemap URL directly if you know it.
• Only part of a site? Use includeUrlPatterns, for example "/products/".
• Run stopped early? RUN_STATS.stoppedByTimeBudget is true when the time budget was reached; everything found until then is kept. Raise the run timeout or timeBudgetSecs.
• Something isn't working? Open an issue on the Issues tab with the run ID.