Sitemap URL Extractor – All URLs from sitemap.xml & robots.txt

Extract page URLs from a website's sitemaps. Finds sitemaps via robots.txt, common paths and homepage links, then reads XML and .xml.gz sitemaps, sitemap indexes, text sitemaps, RSS and Atom feeds. Returns lastmod, changefreq, priority, images, videos and hreflang.

Sitemaps are streamed, so 50 MB+ files work. Gzip is detected even when the server sends the wrong headers, broken XML is parsed as far as it goes, and every run saves a per-sitemap report of what was found and what failed.

What you get

• One row per URL with the sitemap it came from, its domain and (optionally) lastmod, changefreq, priority, image:image, video:video, news:news, xhtml:link hreflang alternates, and feed titles and publish dates.
• Sitemap discovery – robots.txt Sitemap: lines, /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /news-sitemap.xml, /sitemap_news.xml, /sitemap.txt, plus <link rel="sitemap"> and RSS/Atom feeds from the homepage.
• Formats and edge cases handled:
  • gzip-compressed sitemaps served as text/xml, application/octet-stream or anything else (magic-byte detection), double-gzip, deflate
  • http → https and www redirects, relative <loc> links, BOM, CDATA, HTML entities
  • truncated or malformed XML (keeps every complete URL)
  • timeouts, 429 and 5xx errors (exponential-backoff retries, mid-stream resume without duplicates)
  • soft-404 pages that return HTML instead of XML (detected and skipped)
• Filters – include / exclude regex, same-domain only, de-duplication, global and per-site limits.
• Speed – plain HTTP and a streaming SAX parser, no browser. In our tests: about 500 URLs in 2 seconds, a 50 MB sitemap with 4,600 product URLs plus images and hreflang in about 20 seconds.

Use cases

• SEO audits – list all indexable pages, find stale content by lastmod, check hreflang coverage and image sitemaps.
• Crawling seed lists – feed URLs into Website Content Crawler, Cheerio/Playwright scrapers or your own crawler instead of crawling blindly.
• Content & competitor monitoring – schedule daily runs and diff the URL list to spot new products, articles or landing pages.
• Migrations & redirects – export the full URL inventory before a site migration.
• E-commerce intelligence – collect all product URLs (and product images) from a store's sitemap.
• AI / RAG pipelines – get a clean list of documentation or blog URLs to ingest.

How to use

1. Click Try for free.
2. Add one or more websites (https://example.com or just example.com) or direct sitemap/feed URLs to Websites or sitemap URLs.
3. Optionally set Max URLs, regex filters, or turn off metadata for a lean URL list.
4. Click Start. Download the results as CSV, Excel, JSON, XML or HTML, or fetch them via API.

Input

Field	Default	Description
startUrls	https://apify.com	Domains or direct sitemap / sitemap index / .xml.gz / .txt / RSS / Atom URLs.
maxUrls	500	Total URL limit (0 = unlimited).
maxUrlsPerSite	0	Optional cap per start URL so one big site doesn't use the whole budget.
followSitemapIndex	true	Follow child sitemaps in sitemap indexes.
includeRobotsTxtDiscovery	true	Use Sitemap: lines from robots.txt.
tryCommonPaths	true	Probe common sitemap locations (missing ones are skipped silently).
discoverFromHomepage	true	If robots.txt has no sitemap, look for <link rel="sitemap"> and RSS/Atom feeds on the homepage.
urlIncludeRegex / urlExcludeRegex	–	Case-insensitive regex filters on page URLs, e.g. /blog/ or `/tag/
sameDomainOnly	false	Drop URLs on other domains (subdomains are kept).
includeMeta	true	lastmod, changefreq, priority, images, videos, news, alternates, title.
dedupe	true	Return each URL once (duplicates are never charged).
saveSummary	true	Save a per-sitemap report to the key-value store as OUTPUT.
maxConcurrency, maxSitemaps, requestTimeoutSecs, maxRetries, proxyConfiguration		Advanced tuning.

Example:

{
    "startUrls": [{ "url": "https://www.ikea.com/sitemaps/prod-et-EE_1.xml" }, { "url": "wordpress.org" }],
    "maxUrls": 5000,
    "maxUrlsPerSite": 2500,
    "urlExcludeRegex": "/(tag|author)/"
}

Output

One dataset row per URL. Real example from IKEA's product sitemap (lists shortened):

{
    "url": "https://www.ikea.com/ee/et/p/yxstaby-vitriinkarp-maengupuldile-valge-70620011/",
    "sourceSitemap": "https://www.ikea.com/sitemaps/prod-et-EE_1.xml",
    "domain": "www.ikea.com",
    "lastmod": "2026-10-07T05:23:26.915Z",
    "changefreq": "weekly",
    "priority": 1,
    "images": [
        { "url": "https://www.ikea.com/ee/et/images/products/yxstaby-vitriinkarp-maengupuldile-valge__1505544_pe1008573_s5.jpg", "title": null, "caption": null }
    ],
    "alternates": [
        { "hreflang": "en-ee", "url": "https://www.ikea.com/ee/en/p/yxstaby-gaming-controller-display-box-white-70620011/" },
        { "hreflang": "de-de", "url": "https://www.ikea.com/de/de/p/yxstaby-schaukasten-fuer-gaming-controller-weiss-70620011/" }
    ],
    "videos": [],
    "news": null,
    "title": null,
    "sitemapType": "urlset"
}

With includeMeta: false you get a lean list:

{ "url": "https://apify.com/", "sourceSitemap": "https://apify.com/sitemap/pages.xml", "domain": "apify.com" }

• sitemapType is urlset, rss, atom or text.
• videos items: title, description, thumbnailUrl, contentUrl, playerUrl, duration, publicationDate...
• news: publicationName, language, publicationDate, title, keywords.
• For RSS/Atom feeds, lastmod is the item's publish/update date and title its headline.

Run summary (key-value store OUTPUT)

{
    "totalUrls": 500,
    "stopReason": "maxUrls (500) reached",
    "durationSecs": 1.7,
    "sitemapsProcessed": 9,
    "sitemapsFailed": 0,
    "duplicatesSkipped": 0,
    "httpRequests": 10,
    "sites": [{ "input": "https://apify.com", "robotsTxtSitemaps": ["https://apify.com/sitemap.xml"], "urls": 500 }],
    "sitemaps": [
        { "sitemapUrl": "https://apify.com/sitemap.xml", "discoveredVia": "robots.txt", "type": "sitemapindex", "urls": 0, "childSitemaps": 16, "status": "done" },
        { "sitemapUrl": "https://apify.com/sitemap/pages.xml", "discoveredVia": "index:https://apify.com/sitemap.xml", "type": "urlset", "urls": 500, "status": "stopped" }
    ]
}

Pricing

Pay per event: $0.0003 per URL = $0.30 per 1,000 URLs. Platform usage is included.

• Only unique URLs saved to the dataset are charged – duplicates, filtered URLs and sitemap index entries are free.
• Set Maximum cost per run in the run options; the Actor stops cleanly at your budget.

Examples: 500 URLs = $0.15 · 10,000 URLs = $3 · 100,000 URLs = $30.

Use with AI agents (MCP)

Add the Actor to the Apify MCP server (https://mcp.apify.com?tools=openharvest/sitemap-url-extractor) or let agents find it with search-actors ("sitemap"). An agent can pass a bare domain; it does not need to know where the sitemap is.

Example call-actor input for "List the blog posts on example.com that were updated this year":

{
    "startUrls": [{ "url": "example.com" }],
    "urlIncludeRegex": "/blog/",
    "maxUrls": 500
}

The agent reads url and lastmod and filters by date.

FAQ

The site has no sitemap. What happens? The Actor tries robots.txt, the common paths and the homepage's sitemap/RSS links. If none exist, the run ends with 0 URLs (and $0 charged) and the OUTPUT report shows exactly what was tried. For sites without any sitemap, use a crawler such as Website Content Crawler. (Some very large sites – e.g. Amazon, Reddit, Etsy, GitHub – deliberately do not publish their sitemaps publicly.)

Can it handle huge sitemaps? Yes. Sitemaps are streamed and parsed on the fly (SAX), so 50 MB files and indexes with thousands of child sitemaps work with low memory. Use maxSitemaps and maxUrls to control the size of a run.

Does it support .xml.gz? Yes – and also gzip files served with the wrong content type or without a .gz extension.

Can I extract only blog posts / product pages? Use urlIncludeRegex, e.g. /blog/ or /products/, and urlExcludeRegex to drop tag/author pages.

Does it work with RSS and Atom feeds? Yes. Paste the feed URL directly, or let the Actor discover feeds from the homepage.

Some sites block me – what can I do? Enable Apify Proxy in the input (residential proxies help for sites behind strict bot protection). Retries are automatic.

Can I schedule it and get only new URLs? Schedule the Actor and compare datasets between runs (e.g. via the API, Google Sheets or Make/Zapier integrations).

Is it legal?

Sitemaps, robots.txt and RSS feeds are files that website owners publish specifically so machines can read them. This Actor only reads those public files and does not log in or bypass any protection. You are responsible for how you use the URLs – respect each website's terms of service and robots.txt rules when you crawl the pages themselves.

Related actors

• RSS Feed Reader & Feed Finder – RSS/Atom to JSON, Full Text
• Website SEO Audit (coming soon)
• Website Tech Stack (coming soon)

Support

Questions, bugs or feature requests? Open an issue on the Issues tab with the website you tried.