🗺️ Sitemap to URL Crawler — sitemap.xml to URL List Extractor (RAG, SEO, AI Agents)

Instantly turn any website's sitemap.xml into a clean, complete URL list. This sitemap.xml to URL list crawler recursively follows every nested Sitemap Index file and returns a deduplicated dataset of every public page URL — with the SEO metadata (lastmod, changefreq, priority) each URL carries. It is the fastest, cheapest first step for RAG pipelines, LLM training datasets, AI browsing agents, and SEO audits.

Before you can scrape, summarize, or embed a site's content, you need to know every page it has. Guessing sitemap paths, hand-parsing XML, and chasing nested sitemap indexes is tedious and error-prone. Paste a domain (or a direct sitemap URL), and this Actor discovers the sitemap entry point, walks the full index tree, and streams out one row per URL — no API key, no login, no browser overhead. Point it at https://example.com and it will automatically try https://example.com/sitemap.xml for you.

🏆 Why this sitemap crawler?

recursive sitemap-index crawling · deduplicated URL list · lastmod / changefreq / priority metadata · raw HTTP (no browser = blazing fast & cheap) · auto-discovery from a bare domain · no API key / no login · perfect RAG & LLM discovery layer · export to JSON / CSV / Excel

---

✨ What this Actor does

• 🔁 Recursive sitemap-index crawling — automatically resolves Sitemap Index files and every nested sitemap reference (e.g. sitemap-posts.xml, sitemap-products.xml) so you never miss a section.
• 🧭 Smart auto-discovery — paste a bare domain like https://example.com and the Actor tries the standard sitemap.xml entry point for you.
• 🏷️ SEO metadata extraction — captures lastmod, changefreq, and priority for every URL when the sitemap provides them.
• 🧹 Deduplicated URL list — emits a clean, one-row-per-URL dataset with duplicates removed, ready for the next pipeline stage.
• ⚡ Blazing fast & cost-efficient — uses raw HTTP requests with no browser automation, so it flies through huge sitemap indexes at minimal cost.
• 🌐 Multiple start URLs — pass several domains or sitemap links in one run and merge the output.
• 🎚️ Result caps — set maxUrls to control output size and cost on very large e-commerce catalogs.
• 🛡️ Proxy-ready — route requests through Apify Proxy to reduce IP blocks and throttling.
• 📤 Export anywhere — download the URL list as JSON, CSV, or Excel, or pull it via the REST API and feed it straight into a content crawler or vector DB.

🚀 Quick start (3 steps)

1. Configure — add one or more Start URLs: a domain (https://www.nytimes.com) or a direct sitemap (https://example.com/sitemap.xml). Optionally set maxUrls.
2. Run — click Start. The Actor discovers and walks every sitemap, then streams URLs into the dataset.
3. Get your data — export the deduplicated URL list as JSON, CSV, or Excel from the Output / Dataset tab, or pull it via the API into your RAG / SEO pipeline.

📥 Input

Give the Actor a domain or a sitemap URL and (optionally) a cap on how many URLs to return. Here are three ready-to-run scenarios:

Scenario A — Auto-discover from a bare domain

{
  "startUrls": [
    { "url": "https://www.nytimes.com" }
  ],
  "maxUrls": 10000,
  "proxyConfiguration": { "useApifyProxy": true }
}

Scenario B — Point straight at a sitemap index (big e-commerce catalog)

{
  "startUrls": [
    { "url": "https://example.com/sitemap_index.xml" }
  ],
  "maxUrls": 5000,
  "proxyConfiguration": { "useApifyProxy": true }
}

Scenario C — Several domains in one run

{
  "startUrls": [
    { "url": "https://example.com" },
    { "url": "https://another-site.com/sitemap.xml" }
  ],
  "maxUrls": 20000
}

Field	Type	Description
startUrls	array	Required. One or more website URLs or direct sitemap.xml links. A bare domain triggers auto-discovery of the sitemap entry point.
maxUrls	integer	Cap the number of URLs extracted per run — useful for gauging volume or controlling cost on huge sites. Default 10000.
proxyConfiguration	object	Apify Proxy settings, recommended to avoid IP blocks and throttling on large sitemaps. Defaults to Apify Proxy.

📤 Output

Every run produces a clean URL List dataset — one row per extracted URL, carrying whatever SEO metadata the sitemap declared. Here is a trimmed, realistic sample (these are exactly the fields the Actor emits):

[
  {
    "url": "https://example.com/blog/ai-trends-2026",
    "lastmod": "2026-01-10",
    "changefreq": "weekly",
    "priority": "0.8"
  },
  {
    "url": "https://example.com/products/wireless-headphones",
    "lastmod": "2026-06-28",
    "changefreq": "daily",
    "priority": "1.0"
  },
  {
    "url": "https://example.com/about",
    "lastmod": "2025-11-02",
    "changefreq": "monthly",
    "priority": "0.5"
  }
]

💡 Use cases

• 🧠 RAG pipelines — generate a complete URL list to feed into a website-to-Markdown crawler and your vector DB ingestion.
• 🤖 AI agents — hand your agent a full map of a domain so it can browse and retrieve the right pages instead of crawling blindly.
• 📊 SEO audits — pull sitemap metadata (lastmod, changefreq, priority) at scale to spot stale, low-priority, or mis-prioritized pages.
• 🔍 Competitor analysis — enumerate every blog post, product page, category, and landing page a competitor publishes.
• 🔔 Content monitoring — re-run daily and diff lastmod to detect new and updated pages, then scrape only what changed.
• 🚚 Site migration & QA — validate sitemap coverage and quickly find missing sections before or after a move.

👥 Who uses it

RAG & LLM engineers · AI agent builders · SEO specialists · growth & marketing teams · content strategists · competitive-intelligence analysts · web-migration & QA engineers · data engineers building ingestion pipelines

💰 Pricing

This Actor uses Apify's pay-per-result model — you pay for the URLs it extracts plus the underlying platform compute, with no monthly subscription. Because it runs on raw HTTP with no browser, it is one of the cheapest ways to build a large URL list. Use maxUrls to cap output on very large sites and keep costs predictable. See the Pricing tab on the Actor's Apify page for exact, up-to-date figures.

❓ Frequently Asked Questions

How do I extract all URLs from a sitemap.xml?

Paste a domain or a direct sitemap.xml URL as a Start URL and run the Actor. It recursively crawls the sitemap — including nested Sitemap Index files — and returns every public page URL with lastmod, changefreq, and priority when available.

Is this a sitemap API alternative?

Yes — think of it as an on-demand sitemap parsing API. Instead of writing XML-parsing and index-following code yourself, you send a domain and get back a structured, deduplicated URL dataset you can query or export.

Can I use it without an API key or login?

Yes. There's no API key, token, or login. The Actor only reads publicly available sitemap files, so you can build URL lists straight away.

How do I export sitemap URLs to CSV or JSON?

After a run finishes, open the dataset and export the URL list as CSV, JSON, or Excel directly from Apify (or pull it via the REST API). The output is a clean, deduplicated list ready for downstream pipelines — no code required.

Is it legal to crawl a sitemap?

Sitemaps are published by site owners specifically to be read by crawlers, and this Actor only reads that public file. You remain responsible for respecting each site's Terms of Service and applicable law when you use the resulting URLs.

How much data can I get in one run?

As much as the sitemap contains — from a handful of pages to hundreds of thousands on large e-commerce catalogs. Use maxUrls to cap output; start with 5000 for a first scan, then raise it once your pipeline is stable.

Can I use it as the discovery layer for RAG and LLM training?

Yes — that's the core use case. It handles multi-sitemap architectures and index files, producing a complete URL list you can feed into RAG ingestion (e.g. a website-to-Markdown crawler), vector DBs, or LLM training-set preparation.

The sitemap is missing URLs I can see on the site — why?

Not every site lists every page in its sitemap; that's common. For full coverage, combine this Actor (high precision) with an internal-link crawler (high recall) such as the Website Link Graph Crawler below.

Some lastmod / changefreq / priority fields are empty. Is that a bug?

No — those three fields are optional in the sitemap protocol, so many sites omit them. The url field is always populated; the metadata fields are filled only when the sitemap declares them.

🔗 More website & lead-gen tools by logiover

Actor	What it does
Wayback Machine URL Extractor	Historical/archived URLs for any domain from the Internet Archive CDX API
Website Link Graph Crawler	Crawl internal links and map a site's full link graph
Website SEO Audit Crawler	Full on-page SEO audit across an entire site
Broken Link Checker	Crawl a site and find dead links with HTTP status codes
URL to Markdown	Convert any page to clean, RAG-ready Markdown
Website Text & Markdown Crawler	Crawl a whole site into clean text/Markdown for LLMs
Bulk URL Status Checker	Check HTTP status codes for a list of URLs in bulk
Website Contact Scraper	Extract emails, phones, and socials from websites
Subdomain Finder	Enumerate a domain's subdomains
JSON-LD Schema & Meta Tag Extractor	Pull structured data and meta tags at scale
Website Tech Stack Detector	Detect the technologies a site is built with
Website Change Monitor	Detect and track changes to any web page

👉 Browse all logiover scrapers on Apify Store — 180+ actors across real estate, jobs, crypto, social media & B2B data.

⏰ Scheduling & integration

• Schedule — use Apify Schedules to re-crawl a sitemap daily or weekly and detect new pages by diffing lastmod between runs.
• Export — download any run as JSON, CSV, or Excel, or query the dataset via the REST API.
• Automate — chain results into Make, n8n, or Zapier to push fresh URLs into a content crawler, Google Sheet, or vector-DB ingestion job.
• API & webhooks — trigger runs and pull results with the Apify API, and attach webhooks so the next pipeline step fires automatically when a run finishes.

⭐ Support & feedback

Have a feature request or hit an issue? Open a ticket on the Actor's Issues tab and include your start URL, a sample sitemap URL, and the approximate URL count you expect. If this Actor is useful, a ★★★★★ review on its Apify Store page is hugely appreciated and helps others find it.

⚖️ Legal

This Actor only reads publicly available sitemap files — documents that site owners publish specifically for crawlers to consume. It performs no login and accesses no private data. You are responsible for using the extracted URLs in compliance with each website's Terms of Service, robots.txt guidance, applicable laws, and (where relevant) GDPR.

---

📝 Changelog

2026-07-06

• ✨ README overhaul: richer output sample, ready-to-run example scenarios, cross-promo links, and clearer quick-start.

2026-07-01

• Maintenance pass: re-verified end-to-end on live data and confirmed successful runs within the 5-minute quality window on the default input.
• Sharpened Store metadata (SEO title & description) and expanded the FAQ with high-intent, long-tail questions for easier discovery in Google and Apify Store search.
• Added ready-to-run example tasks that cover common real-world use cases.

2026-06-15

• Reliability pass: re-verified end-to-end on live data with real-world inputs. Routine maintenance build.

2026-06-07

• Docs: added coverage for exporting sitemap URLs to CSV/JSON, sitemap index crawling for RAG, and building URL lists for LLM training.

2026-06-05

• 🛡️ Reliability fix: results are no longer dropped by strict output validation — runs now complete cleanly even at high volume (thousands of results).
• ⚡ Stability & performance hardening; fresh rebuild.

2026-06-04

• Verified live & refreshed build — reliability/maintenance pass.