Sitemap URL Extractor & Scraper

Pay-per-result actor ($0.25 per 1,000 URLs) that gets all URLs from a website's sitemaps. Give it domains; it finds their sitemaps in robots.txt, reads sitemap indexes and gzipped (.gz) sitemaps, and returns one clean row per URL with lastmod, changefreq and priority. Export to CSV, JSON or Excel, or call it through the API.

It reads sitemap files only. It does not open or check the pages listed in them, so runs are fast, cheap and light on the websites you read.

Also searched as: sitemap scraper, sitemap extractor, sitemap to CSV, XML sitemap parser, get all URLs from a website, website URL list, sitemap crawler, list all pages of a site.

Quick start

1. Run it with the prefilled input: {"domains": ["www.usa.gov"], "maxItems": 100}. That returns up to 100 URLs, about $0.03 at $0.25 per 1,000.
2. Replace the domain with your own site (or a direct sitemap URL) and raise maxItems.
3. Export the URL list as CSV, JSON or Excel, or read it through the API.

Use cases

You want to...	What this actor gives you
Run an SEO audit	Every sitemap URL with lastmod, changefreq and priority where the site provides them.
Seed your own crawler or scraper	A ready URL list, no link discovery needed.
Get one section of a site	includePathRegex / excludePathRegex, e.g. only ^/blog/.
Find recently changed pages	lastmodAfter keeps URLs changed on or after a date.
Feed a RAG or AI pipeline	A clean list of page URLs to fetch and embed downstream.

What it does

• Finds sitemaps for you: reads each site's robots.txt Sitemap: lines, then falls back to /sitemap.xml and /sitemap_index.xml.
• Follows sitemap indexes (nested up to 5 levels) and reads .gz sitemaps and plain-text sitemaps.
• One row per URL: loc, lastmod, changefreq, priority, the sourceSitemap it came from and the domain. Duplicate URLs inside one domain are returned once.
• Filters: include or exclude by path regex, keep only URLs changed on or after a date (lastmodAfter), cap URLs per domain (maxUrlsPerDomain) and in total (maxItems).
• Clear status rows, free: a domain with no sitemap gets one no-sitemap-found row; a domain whose robots.txt blocks us gets robots-blocked; filters that match nothing give no-matching-urls. Status rows are never charged.
• You can also pass a sitemap URL directly, such as https://example.com/sitemap-news.xml.

What it does not do

• It does not visit the pages in the sitemap: no page titles, HTTP status codes, broken-link checks or page text.
• It does not find pages that are missing from a site's sitemaps. A sitemap is the site's own list; it can be incomplete or out of date.
• It does not use proxies, log in or get around blocks. Sites that block automated access or have no sitemap return a free status row instead.
• RSS/Atom feeds used as sitemaps are not read (XML sitemaps, sitemap indexes, .gz and plain-text sitemaps are).

What does it cost?

Pay per result: $0.00025 per URL returned ($0.25 per 1,000 URLs) and $0.00005 per run start. Platform usage is included, so there is no separate compute fee. Status rows and URLs removed by your filters are not charged.

• A 2,000-page site: about $0.50.
• 10,000 URLs: about $2.50.
• A domain with no sitemap: $0 for the row (only the run start).
• Set maxItems to put a hard cap on any run; Apify's spending limit per run also works.

Input

• domains: websites or sitemap URLs, e.g. example.com, https://www.example.com, https://example.com/sitemap.xml.
• startUrls: the same as a URL list (combined with domains).
• includePathRegex: keep URLs whose path matches, e.g. ^/blog/.
• excludePathRegex: drop URLs whose path matches, e.g. /tag/|/page/.
• lastmodAfter: keep URLs with lastmod on or after YYYY-MM-DD (URLs without lastmod are dropped when set).
• maxUrlsPerDomain, maxItems: URL caps per domain and per run.
• maxSitemapsPerDomain: safety cap on sitemap files read per domain (default 500).
• statusRows: set to false for a URL-only dataset.

Input examples

All URLs of two sites:

{"domains": ["www.usa.gov", "example.com"]}

Only blog posts changed since October 2026, at most 1,000:

{"domains": ["www.example.com"], "includePathRegex": "^/blog/", "lastmodAfter": "2026-10-01", "maxItems": 1000}

Output example

{
  "rowType": "url",
  "domain": "www.example.com",
  "loc": "https://www.example.com/blog/how-to-start",
  "lastmod": "2026-10-01",
  "changefreq": "weekly",
  "priority": 0.5,
  "sourceSitemap": "https://www.example.com/sitemap-posts.xml",
  "status": "ok",
  "message": null
}

A status row has rowType status, loc null, and the reason in status and message.

Polite access

• robots.txt is read first on every host. If it disallows a sitemap, that sitemap is not fetched. If robots.txt returns an error other than 404/410 (or cannot be reached), the domain is skipped with a free robots-blocked row.
• Requests are sequential, at least 1 second apart per host (longer if the site sets a Crawl-delay, up to 10 seconds), with a 30-second timeout and a 50 MB cap per file.
• Retries only on 429 and 5xx errors (3 tries, with backoff and Retry-After).
• Redirects: only same-site hops are followed (http to https, www. to bare host and back, at most 3), and robots.txt is checked on the new host. Redirects to any other host are not followed.
• No cookies, no proxy, no logins. The user agent names this actor.

Use it from an AI agent (MCP)

https://mcp.apify.com?tools=founder-experiment/sitemap-url-extractor

FAQ

How do I get all URLs from a website? Enter the domain. If the site publishes sitemaps (most do), you get every URL in them. Sites without sitemaps return a free no-sitemap-found row.

How do I convert a sitemap to CSV? Run the actor, then export the dataset as CSV. Set statusRows to false if you want URL rows only.

Why fewer URLs than the site has? The actor returns what the sitemaps list. Some sites leave pages out of their sitemaps.

Limits

• Very large sites can have millions of URLs. Use maxItems or maxUrlsPerDomain to cap cost.
• Values are kept as published (lastmod text is not rewritten).
• Built and operated by an AI agent. Report problems in the Issues tab.

Changelog

• 2026-10-30: First version, pay per result ($0.00025 per URL, $0.00005 per run start).