Sitemap URL Extractor: get every page of any website

Give it a domain and get back every URL the site lists in its XML sitemaps, with last-modified date, change frequency, priority and image count. It finds the sitemaps for you (robots.txt first, then the usual locations), follows sitemap indexes to any depth, and reads .xml.gz and plain-text sitemaps.

USD 0.50 per 1,000 URLs. Sites without a sitemap are reported free.

What people use it for

• SEO audits. List every indexable page and spot stale content by lastmod.
• Crawl seeding. Feed a complete URL list into a scraper, a broken-link checker or an AI/RAG ingestion pipeline, instead of crawling link by link.
• Content monitoring. Schedule it daily with Modified since to catch a competitor's new and updated pages, products or articles.
• Site migrations. Snapshot all URLs before and after a redesign to build redirect maps.
• Catalog sizing. Count products, posts or pages on any site in seconds.

Sample output

{
  "site": "ghost.org",
  "url": "https://ghost.org/about/",
  "lastmod": "2026-08-06T10:10:45-05:00",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "sitemapUrl": "https://ghost.org/sitemap-pages.xml"
}

Real runs: vercel.com gave 7,383 URLs in under a second. A news sitemap index on bbc.com gave 1,688 URLs from 4 sitemaps.

How to use it

1. Add one or more domains (example.com) or sitemap URLs (https://example.com/sitemap.xml) to Websites or sitemap URLs.
2. Optionally narrow the results:
   • Only URLs containing: /blog/ or /products/.
   • Skip URLs containing: /tag/, ?page=.
   • Modified since: 2026-09-01.
3. Click Start, then export as CSV, JSON or Excel, or read the dataset through the Apify API.

For monitoring, create a schedule (daily or weekly), set Modified since to a recent date and connect the results to Google Sheets, Slack or a webhook.

What it handles

• Sitemap indexes, nested to any depth, with each sitemap file read once.
• .xml.gz compressed sitemaps and plain-text sitemaps (one URL per line).
• Image sitemaps (imageCount, and image URLs if you turn on Include image URLs).
• hreflang alternates, returned as alternates.
• News sitemaps (newsTitle).
• Duplicate URLs across sitemaps, counted once.

Pricing

What	Price
URL extracted	USD 0.0005 (USD 0.50 per 1,000)
Site with no sitemap, or unreachable	Free

Set a maximum cost per run in the run options and the actor stops cleanly when it gets there. Max URLs per site caps the spend on very large sites.

Limits

• The actor identifies itself. Every request carries the user agent Mozilla/5.0 (compatible; sitemap-url-extractor/1.0; +https://apify.com/JTPalms/sitemap-url-extractor), with plain request headers. It never pretends to be a browser. Some sites block bots; a blocked request is reported as an error, not hidden.
• It only returns URLs the site lists in its sitemaps. Pages missing from the sitemap are not found; use a crawler for those.
• Up to 500 sitemap files per site by default (raise it under Advanced).
• Some sites block bots and answer HTTP 403 (BBC's news sitemaps do); those errors are listed in the run summary. A proxy under Advanced only helps when a site blocks data-center addresses, not bots.

Related actors

• URL Status Checker: Bulk HTTP Status Code & Redirect Checker: Bulk URL status and HTTP status code checker.
• Broken Link Checker: Find Dead Links on Any Website: Crawl a website and find every broken link, image, script and stylesheet: 404s, 5xx, DNS and TLS errors, redirect loops and redirected...
• SEO Audit & AI Search Readiness Checker (GPTBot, llms.txt): Crawl a site and audit every page for SEO basics (title, meta, H1, canonical, hreflang, noindex, alt text, JSON-LD, status, speed) plus...
• RSS Feed Reader & Scraper: Atom, JSON Feed & New-Item Monitor: RSS feed reader and scraper.

FAQ

How does it find the sitemap? It reads Sitemap: lines in robots.txt. If there are none, it tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.txt.

Is it legal? Sitemaps are files that websites publish so machines can list their pages. The actor reads only those public files and robots.txt. It does not log in and collects no personal data.

Something missing or wrong? Open an issue with the site.