Sitemap extractor: get all URLs from a website

This sitemap URL extractor gets all URLs from a website's XML sitemaps, with each URL's lastmod, changefreq and priority, the sitemap file it came from, and (optionally) its live HTTP status. Useful for SEO audits, site migrations, content inventories, change monitoring and feeding other crawlers a clean URL list.

How it finds the sitemaps

1. The Sitemap: lines in the site's robots.txt.
2. If robots.txt lists none: /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml.
3. Sitemap index files are followed up to 5 levels deep. Gzipped sitemaps (.xml.gz), files with a BOM, files with or without the sitemaps.org namespace, plain-text sitemaps (one URL per line) and slightly broken XML are read. Each URL is output once per site.

Input

Input	Default	What it does
startUrls	one prefilled site	Required website(s), or any URL on them. The prefill is an example; a blank API run is rejected before work starts.
maxUrlsPerSite	200	Maximum sitemap URLs from each site (1–50,000).
maxItems	200	Compatibility output cap per site; the lower of this and maxUrlsPerSite applies.
lastmodAfter	none	Optional YYYY-MM-DD sitemap lastmod filter.
urlPattern	none	Optional case-insensitive URL regex.
checkStatus	false	Opt in to HTTP status checks (HEAD, then GET if necessary).

{ "startUrls": [{ "url": "https://example.com" }], "maxUrlsPerSite": 200, "urlPattern": "/blog/", "checkStatus": false }

Output

One dataset row per URL:

field	meaning
site	the site's origin, e.g. https://example.com
url	the URL listed in the sitemap
lastmod, changefreq, priority	as written in the sitemap (empty when the sitemap leaves them out)
sitemapSource	the sitemap file that lists it
depth	0 for a sitemap found directly, 1+ for sitemaps reached through an index
status	HTTP status when status checking is on
error	why a site (or a status check) failed; a failed site gets one row with only site and error

The key-value store gets a SUMMARY-<host> record per site (sitemaps found and read, sitemaps that failed and why, URLs listed, duplicates, how many filters removed, whether the run stopped at your maximum) and a SUMMARY record listing them all.

How it behaves on other people's sites

Requests identify themselves as WeioBot, obey robots.txt for every host and redirect, are paced to about one request per second per site (slower if robots.txt asks), and back off on 429/503 answers. A sitemap that robots.txt blocks is not read; the summary says so. Only public addresses are contacted.

Limits

• No sitemap means 0 URLs and an error reason in the dataset and per-site summary; it does not guess pages.
• This is not a JavaScript crawler: it only reads published sitemap files, so JavaScript-only routes are not discovered.
• Robots.txt is obeyed for sitemap discovery, sitemap files, status checks, redirects and every contacted host.
• Up to 1,000 sitemap files and 50,000 URLs per site per run; a gzipped sitemap may unpack to at most 60 MB.
• Status checking is paced per site, so 1,000 URLs take roughly 20 minutes.
• Sitemaps list what a site wants indexed; pages missing from them are not discovered (this actor does not crawl links).

Pay per event

The default example is one site and up to 200 URLs: at most about $0.062 before optional status checks ($0.002 for one processed site + up to 200 × $0.0003 extracted URLs). A site that cannot be processed, including an invalid, unreachable or robots-refused site, is free. Enabling checkStatus adds $0.0005 only for each URL whose status is actually checked. The free-tier price for extracted URLs is $0.0003, below the comparable $0.0005 tier named in the market brief.

Use the URL list in n8n or Make

Use this actor when an SEO audit, migration, or content inventory needs the URLs a site already publishes—not a web crawl. Set your own output cap before the first paid run.

• n8n: add the Apify node, choose Run Actor, select weio/sitemap-url-extractor-bulk, and pass {"startUrls":[{"url":"https://example.com"}],"maxItems":100}. Follow it with Get Dataset Items and send the URL rows to a sheet, database, or audit step. Use your own Apify credential; do not put a token in workflow JSON.
• Make: add Apify > Run an Actor with the same input, then Get Dataset Items (or Watch Actor Runs for longer jobs). Set the dataset limit explicitly. Connect using your own Apify OAuth/token in Make.

The prefill is only an example. A blank API run is rejected before work starts. checkStatus is optional and only checks sitemap URLs that the site's robots rules allow.

Built and maintained by Weio, Inc. The actor was written and is operated by AI agents (Anthropic Claude), with a person accountable at Weio.