Sitemap URL Extractor & Checker – Get All URLs of Any Website

Get every URL a website lists in its sitemaps in seconds. Just enter a domain: the Actor finds the sitemaps itself (from robots.txt and the usual locations), follows sitemap indexes recursively, reads gzip-compressed, XML, RSS/Atom and plain-text sitemaps, and returns one clean row per URL. Optionally it checks the HTTP status of every URL, so you can find broken links, redirects and blocked pages in your sitemap. No browser and no proxy are needed, so it is fast and cheap: 200,000 URLs from apify.com in about 60 seconds in our test.

Why this Actor

• 🔍 Auto-discovery. Uses Sitemap: lines in robots.txt, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and more.
• 🌲 Sitemap indexes of any depth, e.g. heise.de with 200+ monthly sitemaps.
• 📦 All formats. XML urlset, sitemap index, .gz, RSS 2.0, Atom, TXT, plus image, video, news and hreflang extensions.
• 🗓️ Smart date filter. With lastmodWithinDays, child sitemaps whose own lastmod is older are skipped entirely. On heise.de, 204 of 206 sitemap files were skipped, so a run that would read 12,600 URLs finished in 3 seconds.
• ✅ Status check included. A HEAD request with a GET fallback (many servers break HEAD) adds statusCode, finalUrl, redirected, contentType, isOk and possiblyBlocked (bot-protection challenge detected).
• 🎯 Include / exclude patterns. Plain text, wildcards, or regex with the re: prefix.
• 🧾 Per-website summary (SUMMARY record): sitemaps found, processed and skipped, errors, URLs found, filtered and saved.
• 💸 Free to use right now – you only pay the tiny Apify platform usage (no browser, no proxy).

Use cases

• SEO audits: find 404s, redirects and non-HTML URLs listed in a sitemap
• Content inventory and migrations: a full URL list with last-modified dates
• Feed a crawler or an LLM/RAG pipeline: crawl only the URLs that really exist, or only those changed in the last N days
• Competitor monitoring: which pages did a competitor publish or update this week?
• Indexing tools: submit fresh URLs to IndexNow or Google Search Console

Input

Field	Description
startUrls	Websites (https://example.com) or direct sitemap/feed URLs
includePatterns / excludePatterns	e.g. /blog/, *.pdf, `re:/20(25
lastmodWithinDays / lastmodAfter	Date filter on <lastmod>
keepUrlsWithoutLastmod	Keep URLs without a date when filtering (default on)
checkStatus	Check the HTTP status of each URL (default off)
sameDomainOnly	Drop URLs on foreign domains (default on)
tryCommonPaths	Probe common sitemap paths if robots.txt has none (default on)
maxUrls, maxSitemaps, concurrency	Limits

{
  "startUrls": [{"url": "https://wordpress.org/news/"}],
  "includePatterns": ["/news/20"],
  "lastmodWithinDays": 30,
  "checkStatus": true,
  "maxUrls": 5000
}

Output (real example)

{
  "url": "https://wordpress.org/news/2014/04/smith/",
  "lastmod": "2014-04-15T21:49:40Z",
  "lastmodIso": "2014-04-15T21:49:40+00:00",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "videoCount": 1,
  "alternates": [],
  "newsTitle": null,
  "sitemapUrl": "https://wordpress.org/news/video-sitemap-1.xml",
  "sitemapType": "urlset",
  "website": "https://wordpress.org/news/",
  "path": "/news/2014/04/smith/",
  "statusCode": 200,
  "finalUrl": "https://wordpress.org/news/2014/04/smith/",
  "redirected": false,
  "contentType": "text/html",
  "isOk": true,
  "checkError": null,
  "possiblyBlocked": false
}

Export as JSON, CSV, Excel, XML, or via API, webhooks, Make, n8n or Zapier.

Pricing

The Actor is currently free – you only pay Apify platform usage, which is very low because no browser or proxy is used (a 20-URL test run with status check cost about $0.0001). A fair pay-per-result price may be introduced later; Apify announces such changes at least 14 days in advance. Set Max cost per run to cap your spend.

Notes

• Only URLs listed in sitemaps are returned. This Actor does not crawl links on pages.
• Some websites protect pages with bot challenges. These are flagged with possiblyBlocked: true. Lower concurrency for such sites.
• Respect each website's terms of service. The Actor reads only sitemap files that the website publishes for machines, and it sends at most one lightweight request per URL when checking status.

---

🇩🇪 Deutsch: Sitemap-URL-Extraktor & Statusprüfung

Liest alle URLs aus den Sitemaps einer Website aus. Die Sitemaps werden automatisch über die robots.txt und typische Pfade gefunden. Sitemap-Indizes, .gz, RSS/Atom und TXT werden unterstützt. Optional wird für jede URL der HTTP-Status geprüft (404, Weiterleitungen, Bot-Schutz).

Ideal für: SEO-Audits, Relaunch und Migration, Content-Inventur, Wettbewerbsbeobachtung ("Was wurde diese Woche neu veröffentlicht?") und als Vorstufe für Crawler oder KI/RAG-Pipelines.

Besonderheiten: Der Datumsfilter überspringt komplette Teil-Sitemaps, deren lastmod älter ist (das ist sehr schnell). Dazu kommen Include/Exclude-Muster (auch Regex), eine Zusammenfassung pro Website und die Nutzung ist derzeit kostenlos (nur geringe Apify-Plattformkosten).