Sitemap URL Extractor — Bulk Sitemap.xml Scraper

Extracts every URL from a website's sitemap.xml, auto-discovering the real sitemap via robots.txt and following nested sitemap index files. Returns a clean dataset in JSON/CSV/Excel.

What it does

• Finds the real sitemap for any domain (it doesn't assume /sitemap.xml: it first looks for the Sitemap: line in robots.txt, which is where many large sites declare it).
• Follows nested sitemap indexes (sitemaps that point to other sitemaps) up to 5 levels deep.
• Deduplicates URLs repeated across sitemaps.
• Caps the run safely with maxUrls and an internal ceiling of 50 sitemaps processed per run, so a huge site can't trigger an unpredictable cost.
• If an individual sitemap fails (404, timeout), it's reported as an error and the run continues with the rest — one broken link doesn't crash the whole run.

Who it's for

• SEO teams auditing indexation coverage or building a crawl budget.
• Data/agent pipelines that need the full list of URLs on a site before processing them (custom scraping, comparative sitemap generation, change monitoring).

Input

Field	Type	Required	Description
startUrl	string	Yes	Site homepage (e.g. https://example.com) or a direct URL to a .xml file.
maxUrls	integer	No (default 1000)	Maximum number of URLs to return, between 1 and 20000.

{
  "startUrl": "https://example.com",
  "maxUrls": 1000
}

Output

One item per URL found:

{
  "url": "https://example.com/blog/post-1",
  "sourceSitemap": "https://example.com/sitemap-posts.xml"
}

The run's key-value store (RUN-SUMMARY) also holds a summary: sitemaps processed, how the sitemap was discovered (robots.txt, default, or direct input), and per-sitemap errors, if any.

Pricing

Pay-per-event, provisional until calibrated against real compute (see build/README.md → "Calibrar precios"). Starting reference: USD 2.00 per 1,000 URLs extracted (03-plan.md §2).

How to call it

curl "https://api.apify.com/v2/acts/cuantic-data~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrl": "https://example.com", "maxUrls": 500}'

Source terms and limits

See ./TERMS.md: sitemap.xml and robots.txt are public files designed to be read by automated programs. The Actor doesn't follow Disallow, doesn't visit site pages (only sitemap files), and doesn't extract personal data.

Known limitation (disclosed, not hidden): compressed sitemaps (.xml.gz) aren't supported in this version — if your site only publishes the compressed sitemap, the Actor reports it as an error instead of failing silently.

FAQ

Why does it return 0 URLs? The site has no robots.txt with a Sitemap: line and no /sitemap.xml at the root. Check the run summary (RUN-SUMMARY) for the exact error per sitemap.

Found a bug? Email cuanticwindows@gmail.com — we reply within 72 hours.

---

See build/README.md for how to run tests and publish this Actor.