Sitemap URL Extractor: Every URL from robots.txt, Sitemap Indexes & .xml.gz (Pay Only per Unique URL)

Give it a domain and get back every URL the site publishes in its sitemaps, de-duplicated, with lastmod, changefreq and priority. Sitemaps are found automatically through robots.txt and common locations, nested sitemap indexes are followed, and gzip (.xml.gz), plain-text, RSS and Atom sitemaps all work. You pay only for unique URLs delivered. Duplicates, filtered URLs, failed sitemaps and errors cost $0.

Use it as a sitemap scraper / sitemap.xml parser, a "list all pages of a website" tool, a URL finder for RAG and LLM ingestion, or a new-page monitor for competitors. No browser, no crawling, so it's fast and cheap.

Quick start (about 30 seconds)

1. Enter one or more domains in Websites or sitemap URLs (for example nodejs.org).
2. Optional: add an Include URL pattern like /blog/ or a Modified since date.
3. Click Start and export the URL list as CSV, Excel or JSON, or pipe it into another Actor.

What can it do?

• 🔎 Auto-discovery: reads Sitemap: lines in robots.txt, then tries /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz and /sitemap.txt
• 🗂️ Sitemap indexes: follows nested indexes (WordPress, Yoast, Shopify, Next.js and so on) up to a depth you choose
• 🗜️ Gzip, text, RSS & Atom: .xml.gz is detected by content, not just by file name
• ♻️ De-duplication across all sitemaps and inputs
• 🎯 Filters: include/exclude regex, same-domain only, and "modified since" by lastmod
• 🖼️ Optional image sitemap entries and hreflang alternates
• 🛡️ Polite and safe: per-host delay, honors robots.txt Crawl-delay and Retry-After, retries 429/5xx with backoff, and blocks private/internal addresses
• ⚡ Fast and light: pure HTTP with no browser. In our tests it pulled about 172,000 URLs from 108 sitemaps in 30 s, and over 1 million URLs in a single run

Who uses it?

• SEO: audit indexable pages, find orphan pages, and compare sitemap URLs with crawled URLs
• Content & competitor monitoring: get new URLs since a date (modifiedSince) on a schedule
• Scraping pipelines: feed product or article URLs into any other Actor
• AI/RAG: build a URL list of docs sites for ingestion

Popular recipes

Goal	Input
All blog posts of a competitor	"includeUrlPatterns": ["/blog/"]
New pages published this month (schedule weekly)	"modifiedSince": "2026-10-01"
All product URLs of a Shopify store	"includeUrlPatterns": ["/products/"]
Docs URLs to feed a RAG / LLM pipeline	"startUrls": ["docs.example.com"], "sameDomainOnly": true
Every language version of a page	"includeAlternates": true
Quick sample of a huge site	"maxUrls": 1000

How much does it cost?

Pay per unique URL delivered: $0.40 per 1,000 on Free, $0.30 on Starter, $0.20 on Scale, and $0.10 on Business. There are no platform-usage fees on top.

URLs	Free-plan price
1,000	$0.40
10,000	$4.00
100,000	$40.00

Big sites can have millions of URLs, so set Max URLs or Max total charge on the run. The Actor stops cleanly at the cap.

Input

{
  "startUrls": ["https://www.djangoproject.com", "nodejs.org", "https://example.com/sitemap_index.xml"],
  "includeUrlPatterns": ["/blog/"],
  "modifiedSince": "2026-01-01",
  "maxUrls": 50000
}

Inputs can be domains, page URLs, robots.txt URLs, or direct sitemap/RSS URLs.

Output

{
  "url": "https://nodejs.org/en/blog",
  "lastmod": "2026-10-21T12:15:15.044Z",
  "changefreq": "always",
  "priority": null,
  "host": "nodejs.org",
  "sitemapUrl": "https://nodejs.org/sitemap.xml",
  "sitemapFormat": "xml",
  "input": "https://nodejs.org",
  "extractedAt": "2026-10-22T00:58:23.391Z"
}

A SUMMARY record in the key-value store lists, per input, where its sitemaps were found, which sitemaps failed and why, and the duplicate and filter counts. Export as JSON, CSV or Excel, or call it via API, MCP, Make, Zapier or n8n.

Use it from code (API)

curl -X POST "https://api.apify.com/v2/acts/blond_terrarium~sitemap-url-extractor-pro/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["nodejs.org"],"maxUrls":1000}'

Python:

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("blond_terrarium/sitemap-url-extractor-pro").call(run_input={"startUrls": ["nodejs.org"], "includeUrlPatterns": ["/blog/"]})
urls = [i["url"] for i in client.dataset(run["defaultDatasetId"]).iterate_items()]

Use it from AI agents (MCP)

This Actor is available as a tool through the Apify MCP server (https://mcp.apify.com). In Claude, Cursor or ChatGPT, ask: "List every URL in the sitemap of example.com that contains /docs/ using blond_terrarium/sitemap-url-extractor-pro." It's a cheap first step before scraping or summarizing a site, so an agent doesn't have to crawl blindly.

Chain it with other Actors

• Sitemap → content for RAG: pass the URLs to a website-content / markdown crawler.
• Sitemap → screenshots: send URLs to Bulk Website Screenshot.
• Sitemap → link audit: run Broken Link Checker on the same site.
• Works with Zapier, Make, n8n, Google Sheets and webhooks.

How it compares

	This Actor	Crawling the site	Basic sitemap parsers
Finds sitemaps via robots.txt + common paths	✅	n/a	often only /sitemap.xml
Nested sitemap indexes + .xml.gz	✅	n/a	partial
De-duplicated, with lastmod/changefreq/priority	✅	no lastmod	varies
Speed on a 100k-URL site	seconds	hours	varies
Pay only for unique URLs delivered	✅	pay per page crawled	varies

FAQ

A site returned no URLs. Why? Some sites publish no sitemap at all (python.org, for example). The SUMMARY record shows "No sitemap found" for those, and you aren't charged.

Does it crawl pages? No. It only reads sitemaps, which keeps it fast and cheap. To crawl links, use a crawler or a broken-link checker.

Can I pass a sitemap URL directly? Yes, https://example.com/sitemap_index.xml, a robots.txt URL or an RSS feed URL all work.

Can I monitor new pages? Yes. Save a Task with modifiedSince and add a Schedule; or diff two runs.

How do I cap spend on giant sites? Set Max URLs or Max total charge. The Actor stops cleanly.

Is it allowed? Sitemaps are published so machines can read them. The Actor identifies itself honestly, respects robots.txt when probing, and is for public data only.