Sitemap URL Extractor – get every URL from any website's sitemap

Sitemap URL Extractor returns every page URL a website publishes in its XML sitemaps – in seconds, with no crawling, no browser and no proxies. Give it a homepage and it finds the sitemaps itself (via robots.txt and common locations), or paste sitemap URLs directly.

✅ Pay per result: $0.50 per 1,000 URLs · 🆓 Failed sitemaps, duplicates and filtered-out URLs are never charged · 🔑 No API key · ⚡ 100,000 URLs in about a minute

▶ Try it with one click · 📄 Real example output · 💲 Pricing · 🔌 API & integrations · ❓ FAQ · 🛟 Support

What does Sitemap URL Extractor do?

It follows nested sitemap indexes, unpacks .xml.gz, reads plain-text, RSS and Atom sitemaps, survives malformed XML, and returns one clean, de-duplicated list with lastmod, changefreq, priority, hreflang alternates, image counts and Google News data. An optional HTTP status check finds broken and redirecting URLs listed in the sitemap.

• 🔎 Auto-discovery from robots.txt Sitemap: lines, then /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt
• 🌳 Nested indexes (configurable depth): WordPress/Yoast, Shopify, Wix, Squarespace, Next.js, Docusaurus, news sites
• 🗜️ Every format: gzip, text sitemaps, RSS 2.0 / Atom, BOMs, CDATA, entities, broken XML (fallback parser)
• 🧹 Filters: regex include/exclude, same-host only, lastmod on/after a date ("what changed since last week?")
• 🩺 Optional status check (HEAD, falls back to GET): 404s, 5xx and redirect chains
• 🛡️ Polite by default: one request at a time per host, 300 ms delay, honest User-Agent, respects robots.txt for guessed paths and status checks
• 📊 Run summary per website: how the sitemap was found, files read, files failed (with the exact error), duplicates and filtered counts

Use cases

• SEO audits – check index coverage, find orphan, broken or redirecting URLs, compare sitemap vs. crawl
• Crawler seeding – feed an exact URL list to Website Content Crawler, Cheerio/Playwright scrapers or a RAG ingestion pipeline instead of crawling blindly
• Content monitoring – schedule a daily run with lastmodAfter and get only new or updated pages from a competitor or publisher
• Site inventories & migrations – export every URL to a spreadsheet before a redesign or domain move
• AI agents – a cheap, deterministic "list all pages of this site" tool

How to use it

1. Click Try for free – the input is prefilled (https://crawlee.dev, max 100 URLs), so Start works straight away and costs about $0.05.
2. Replace it with your websites (https://example.com) or sitemap URLs (https://example.com/sitemap_index.xml).
3. Optional: include/exclude patterns (e.g. include /blog/, exclude /tag/), a lastmodAfter date, or Check HTTP status.
4. Click Start. Download JSON, CSV, Excel, XML or HTML, or read results via API.

Input example

{
  "startUrls": [{ "url": "https://crawlee.dev" }],
  "maxUrls": 100
}

More options: includeUrlPatterns, excludeUrlPatterns, lastmodAfter (YYYY-MM-DD), sameHostOnly, includeAlternates, includeImages, checkStatus, maxStatusChecks, maxSitemaps, maxDepth. Every field is explained in the Input tab.

Output example (from a real public run)

This is the first item from the example run above (crawlee.dev, 100 URLs, 9 Oct 2026). See the full public dataset: JSON · CSV

{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "sitemapDepth": 0,
  "inputUrl": "https://crawlee.dev",
  "host": "crawlee.dev",
  "alternates": [],
  "imageCount": 0
}

With Check HTTP status on, each item also gets statusCode, finalUrl and redirected. News sitemaps add newsTitle and newsPublicationDate. A RUN_SUMMARY record in the key-value store explains, per input, what was found, skipped or failed.

Field	Description
url	Page URL from the sitemap (de-duplicated across all sitemaps)
lastmod, changefreq, priority	Values from the sitemap (null if the site doesn't publish them)
sitemapUrl, sitemapDepth, inputUrl	Where the URL came from
alternates	hreflang language versions
imageCount, images	Image sitemap data (images only with Include image URLs)
statusCode, finalUrl, redirected	Only with Check HTTP status

How much does it cost to extract sitemap URLs?

Pay-per-event pricing: you pay only for URLs you get. No platform usage charges on top.

Event	Price	Notes
URL extracted	$0.0005 ($0.50 per 1,000)	Silver plan $0.0004, Gold and above $0.0003
URL status checked (optional)	$0.0005	Only when Check HTTP status is on
Run start	$0.0005 per GB of memory	Default 512 MB = one start event = $0.0005

Worked examples (Free/Starter prices, default memory):

Job	Calculation	Total
One-click example (100 URLs)	100 × $0.0005 + $0.0005	$0.05
Blog section of a site (2,000 URLs)	2,000 × $0.0005 + $0.0005	$1.00
Mid-size site + broken-link check (10,000 URLs, 1,000 status checks)	$5.00 + $0.50 + $0.0005	$5.50
Large site (100,000 URLs)	100,000 × $0.0005 + $0.0005	$50.00 (Gold: $30.00)

Never charged: duplicates, URLs removed by your filters, sitemaps that fail to download or parse. Cap spending with Max URLs or Apify's Max cost per run – the Actor stops cleanly at the limit. Apify's Free plan includes $5 of monthly credit.

Integrations (API, AI agents, Make, Zapier)

API – run it and get results in one call:

curl -X POST "https://api.apify.com/v2/acts/chris_elder~sitemap-url-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://crawlee.dev"}],"maxUrls":100}'

from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("chris_elder/sitemap-url-extractor").call(run_input={"startUrls": [{"url": "https://crawlee.dev"}], "maxUrls": 100})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"])

AI agents (MCP) – add https://mcp.apify.com/?tools=chris_elder/sitemap-url-extractor to Claude, Cursor, VS Code or any MCP client. The agent can then call "list all URLs of this site" as a tool.

Make, Zapier, n8n – use the Apify app's Run Actor action, pick chris_elder/sitemap-url-extractor, then map dataset items into Google Sheets, Slack, Airtable or your CRM. Schedules and webhooks in Apify Console let you run it daily and push results anywhere.

Limits and good to know

• It reads sitemaps only – it never crawls HTML pages. If a site publishes no sitemap, the run finishes with a clear warning (and costs only the start fee).
• lastmod is self-reported by the site and is often missing or inaccurate.
• Very large sites: raise Max sitemap files per website and Max URLs. 100,000 URLs typically take about a minute.
• Sitemaps behind logins or bot walls can't be read.

Is it legal?

Sitemaps are files that website owners publish so that automated tools can discover their URLs. This Actor reads only those files and robots.txt. It collects no personal data, doesn't log in and doesn't bypass any protection. You are responsible for how you use the URLs and for each website's terms. See Is web scraping legal?

FAQ

Does it work with WordPress, Shopify, Wix and Squarespace? Yes. They publish standard sitemap indexes, which are followed automatically.

Does it render JavaScript? It doesn't need to: sitemaps are plain XML or text.

Why did I get fewer URLs than the site has pages? The site's sitemap lists fewer pages, or your Max URLs or filters stopped the run. Check RUN_SUMMARY.

Can I get only new pages? Yes. Set Only URLs modified after and schedule the Actor.

How do I check for broken links in my sitemap? Turn on Check HTTP status and filter results for statusCode >= 400.

Support, reliability and feature requests

• Failed items are free. You pay only for URLs delivered.
• Tested on real sites before every release (gov.uk, crawlee.dev, docs.apify.com, BBC and WordPress news sitemaps).
• Issues answered within 2 business days. Open an issue on the Issues tab with your input URL; parsing bugs are fixed first.
• Feature requests welcome – open an issue titled "Feature request: …".
• Built by Chris Elder with AI-assisted development; every release is reviewed and tested before it ships.

Need the data done for you?

If you'd rather not set up runs yourself, the UK company behind these tools (GPT Shop Ltd) offers a fixed-price done-for-you scrape. Tell us the website and the fields you need, and you get a clean CSV or Google Sheet within 48 hours of the confirmed brief, up to 5,000 rows, for £149 or $179. If we can't get the data, you're refunded in full. See the done-for-you scraper service

More tools by Chris Elder

• PDF to Markdown & Text Converter (with OCR) – LLM- and RAG-ready Markdown from any PDF
• Bulk Lighthouse & PageSpeed Audit – Core Web Vitals and SEO scores for many URLs (pairs well with this Actor's sitemap output)
• Bulk WHOIS Domain Lookup – registrar, expiry, availability and DNS for many domains via official RDAP
• Image to Text OCR – bulk text extraction from images in 100+ languages
• All tools: apify.com/chris_elder