Sitemap Extractor: All URLs from XML Sitemaps & Indexes

Sitemap Extractor: All URLs from XML Sitemaps & Indexes

Give it a website or a sitemap URL. The Actor finds the site's sitemaps, follows sitemap indexes, and lists every URL with its lastmod, changefreq and priority.

• Finds sitemaps for you: reads the Sitemap: lines in robots.txt, then tries /sitemap.xml and /sitemap_index.xml.
• Follows sitemap indexes: nested indexes, .xml.gz files, namespaced XML, and plain-text sitemaps (one URL per line).
• Filters: keep URLs that contain /blog/, drop URLs that contain /tag/, or keep only pages modified since a date.
• Clean output: each URL once, even when two sitemaps list it.

$0.20 per 1,000 URLs. No browser, no proxy, no API key. It reads only robots.txt and sitemap files, which sites publish for crawlers. It never opens the pages themselves. The Apify free plan gets up to 10,000 URLs per run.

How to use

1. Open the Actor in the Apify Console and go to the Input tab.
2. Type a website, like https://crawlee.dev, into Websites or sitemap URLs. One per line, up to 100.
3. Click Start.
4. When the run ends, open the Output tab. Download the URLs as CSV, Excel or JSON, or read them through the API.

The prefilled input reads the first 100 URLs from crawlee.dev. Clear Max URLs to get them all.

What can you do with it?

• Feed a crawler: get the full URL list of a site first, then pass it to a scraper as start URLs.
• SEO audits: count pages per section, find stale pages by lastmod, and check what a site tells search engines.
• Watch a site for new pages: schedule a daily run with Modified since set, or compare today's list with yesterday's.
• Content and competitor research: list every blog post or product page on a site in seconds.
• AI agents and RAG: call the Actor as a tool (Apify API or Apify MCP server) to find the pages to read.

Input example

{
    "startUrls": ["https://crawlee.dev", "https://example.com/sitemap.xml"],
    "includeUrlsContaining": ["/blog/"],
    "excludeUrlsContaining": ["/tags/"],
    "modifiedSince": "2026-01-01",
    "sameSiteOnly": true,
    "maxUrls": 5000
}

Output example

One row per URL:

{
    "url": "https://crawlee.dev/blog",
    "lastmod": null,
    "changefreq": "weekly",
    "priority": 0.5,
    "sitemap": "https://crawlee.dev/sitemap.xml",
    "startUrl": "https://crawlee.dev/"
}

lastmod, changefreq and priority are null when the sitemap does not give them. The OUTPUT record holds a summary: sites read, sitemaps read, URLs found and written, duplicates skipped, and the errors for each site (for example a 404 sitemap or a robots.txt rule).

Input fields

Field	What it does
startUrls	Websites or sitemap URLs. A URL whose path has "sitemap" or ends in .xml is read as a sitemap.
includeUrlsContaining	Keep a URL only if it contains one of these texts. Plain text, not a regex.
excludeUrlsContaining	Drop a URL if it contains one of these texts.
modifiedSince	Keep only URLs with a lastmod on or after this date. URLs with no lastmod are dropped.
sameSiteOnly	Drop URLs on other domains (default on). Subdomains count as the same site.
maxUrls	Stop after this many URLs in total.
maxSitemapsPerSite	Stop reading a site after this many sitemap files (default 500).

Pricing

Pay per result: $0.20 per 1,000 URLs written to the dataset, plus a tiny start fee ($0.00005). Duplicates and filtered-out URLs are free. The run stops at your maximum total charge, so a big site never costs more than you set.

The prefilled input writes at most 100 URLs, so a one-click run costs about $0.02, plus the start fee.

Polite and safe

• One request at a time per site, at most one a second, and slower if robots.txt asks for a Crawl-delay (up to 30 seconds; a site asking for more is skipped).
• Follows robots.txt Disallow rules for sitemap files.
• Sitemaps on other sites, and private or local network addresses, are never read.
• Sitemap files are capped at 52 MB (the sitemap limit is 50 MB), also when gzipped.

FAQ

Do I need to know the sitemap URL? No. Give it the website and it looks in robots.txt, then tries /sitemap.xml and /sitemap_index.xml. You can also give a sitemap URL directly.

Does it open the pages in the sitemap? No. It reads only robots.txt and sitemap files, so it is fast and light on the site.

Am I charged for duplicates? No. Each URL is written and charged once per run, even when two sitemaps list it. URLs your filters drop are free too.

What if a site has no sitemap? The run writes no URLs for it, and the OUTPUT summary lists the error for that site, like a 404 sitemap or a robots.txt rule.

Limits

• Up to 100 websites or sitemap URLs per run.
• Up to 500 sitemap files per site by default (Max sitemap files per site, up to 5,000).
• The Apify free plan gets up to 10,000 URLs per run.
• URLs longer than 2,048 characters are skipped.
• It reads sitemaps only. A page that is not in any sitemap is not found. To find those, use a crawler.
• Image, video and news sitemap extras (like image:loc) are not in the output yet.
• RSS and Atom feeds used as sitemaps are not read.

Related Actors

• Bulk Lighthouse & PageSpeed Audit: audit the URLs you found for Core Web Vitals.
• Remove Duplicates & Merge Datasets: with "only new items", get just the pages added since the last run.

Issues and support

Found a bug or need a feature? Open an issue in this Actor's Issues tab. We check it every day and reply there.