Sitemap URL Extractor: All Website URLs and Status Codes

This sitemap URL extractor finds every sitemap of a website and lists every URL in them. Give it a website (example.com) and it reads the Sitemap: lines in robots.txt, checks the usual places such as /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml, follows every sitemap index and unpacks .gz files. Give it a sitemap link and it starts there. Each URL comes back with its lastmod date, changefreq, priority, the sitemap it came from, the number of images and videos, hreflang alternates and Google News title and date. Turn on the status check to find broken and redirecting URLs.

• Finds sitemaps for you. No need to know where a site keeps them. robots.txt and nine common paths are checked, and nested indexes are followed up to six levels deep.
• Handles real-world sitemaps. Gzip files, image, video and news extensions, hreflang xhtml:link alternates, plain text sitemaps (one URL per line) and RSS or Atom feeds used as sitemaps.
• Filters before you pay. Keep only /blog/, drop *?page=*, use a regular expression, or keep only URLs modified in the last 30 days. Filtered URLs are not saved and cost nothing. With a date filter, sitemap files that are older than the cut-off are skipped without being downloaded.
• Optional status check. Each URL gets a HEAD request (with a GET fallback for servers that refuse HEAD) and you get its status code, where it redirects and the final status.

What data you get

Field	What it holds
site	The website the URL was found for
url	The page URL from the sitemap
lastmod, changefreq, priority	The sitemap values, when the site provides them
sourceSitemap	The sitemap file the URL was listed in
imageCount, images	Number of images from image sitemaps, and their URLs with Include image URLs
videoCount	Number of videos from video sitemaps
alternates	hreflang alternates: language code and URL
newsTitle, newsPublishedAt	Title and publication date from Google News sitemaps
statusCode, finalStatusCode, finalUrl, redirected, statusError	With Check the HTTP status: the URL's own status (301, 404...), the status after redirects, where it ends up, and the network error if it could not be reached

The run's key-value store gets a SUMMARY record with, for each site: every sitemap found (type, how it was found, URL count, child sitemaps), the common paths that had no sitemap, errors such as HTTP 403 or a sitemap that returned a web page, and how many URLs were skipped as duplicates or by your filters. With the status check on, it also counts URLs by status code.

How to use it

1. Add websites or sitemap URLs to Websites or sitemap URLs. Many sites in one run are fine.
2. Set Max URLs to cap the run. Each saved URL is one result.
3. Optional: add Only URLs matching or Skip URLs matching patterns, a date in Only URLs modified after, or turn on Check the HTTP status of each URL.
4. Run it and download the list as CSV, Excel or JSON, or call it from the API, Make, Zapier, n8n or an AI agent through the Apify MCP server.

Patterns come in four kinds:

Pattern	Matches
/blog/	URLs whose path starts with /blog/
*/products/*	A glob on the full URL; * matches anything
/[0-9]{4}/[0-9]{2}/$/	A regular expression between slashes (it must use a regex symbol, so /blog/ stays a path)
pricing	URLs that contain the word

Example input

{
  "startUrls": [
    "https://wordpress.org/news",
    "https://www.theguardian.com/sitemaps/news.xml",
    "https://www.deepl.com/sitemap.xml"
  ],
  "maxUrlsPerSite": 100,
  "checkStatus": true,
  "includePatterns": ["/news/", "*theguardian.com/*", "/de/"]
}

Example output (from a real run, one URL per site, alternates shortened)

[
  {
    "site": "https://www.theguardian.com",
    "url": "https://www.theguardian.com/media/commentisfree/2026/oct/30/liberal-senator-sarah-henderson-mistake-abc-gaza-stringer-ntwnfb",
    "lastmod": "2026-10-30T02:44:00Z",
    "changefreq": "hourly",
    "priority": null,
    "sourceSitemap": "https://www.theguardian.com/sitemaps/news.xml",
    "imageCount": 1,
    "videoCount": 0,
    "alternates": [],
    "newsTitle": "Liberal senator admits mistake after grilling ABC execs about Gaza journalist’s ‘Hamas’ ID",
    "newsPublishedAt": "2026-10-30T02:37:22Z",
    "statusCode": 200,
    "finalStatusCode": 200,
    "finalUrl": "https://www.theguardian.com/media/commentisfree/2026/oct/30/liberal-senator-sarah-henderson-mistake-abc-gaza-stringer-ntwnfb",
    "redirected": false,
    "statusError": null
  },
  {
    "site": "https://www.deepl.com",
    "url": "https://www.deepl.com/de/translator",
    "lastmod": null,
    "changefreq": null,
    "priority": null,
    "sourceSitemap": "https://www.deepl.com/p-JidqPmeasZMtTlNiGpFf0gzd.xml",
    "imageCount": 0,
    "videoCount": 0,
    "alternates": [
      { "hreflang": "x-default", "url": "https://www.deepl.com/translator" },
      { "hreflang": "pt-br", "url": "https://www.deepl.com/pt-BR/translator" }
    ],
    "newsTitle": null,
    "newsPublishedAt": null,
    "statusCode": 200,
    "finalStatusCode": 200,
    "finalUrl": "https://www.deepl.com/de/translator",
    "redirected": false,
    "statusError": null
  },
  {
    "site": "https://wordpress.org",
    "url": "https://wordpress.org/news/prototype-of-the-wayfinder-in-action/",
    "lastmod": "2023-06-27T16:50:02Z",
    "changefreq": null,
    "priority": null,
    "sourceSitemap": "https://wordpress.org/news/video-sitemap-1.xml",
    "imageCount": 0,
    "videoCount": 1,
    "alternates": [],
    "newsTitle": null,
    "newsPublishedAt": null,
    "statusCode": 301,
    "finalStatusCode": 200,
    "finalUrl": "https://wordpress.org/news/files/2023/06/Prototype-of-the-Wayfinder-in-action.mp4",
    "redirected": true,
    "statusError": null
  }
]

That run found 300 URLs across the three sites, checked all of them in 23 seconds, and flagged 4 that redirect.

How much does it cost?

You pay $0.001 per URL saved, plus $0.0005 once per run. The status check is included at no extra cost. URLs removed by your filters, duplicates listed in more than one sitemap, and sites where no sitemap could be found cost nothing.

1,000 URLs cost $1.00, and 10,000 URLs cost $10.00. Set Max URLs to cap a run on a large site.

Use cases

• SEO audits: get the full list of URLs a site submits to Google, then check which of them return 404 or redirect. Sitemaps should only list final 200 URLs.
• Content inventories and migrations: export every page with its last modified date before a redesign or a CMS move, and check the redirects afterwards.
• Feed other tools: use the URL list as input for a crawler, a screenshot tool, a page speed test or an AI extraction run.
• Track new pages: schedule a run with Only URLs modified after set to 1 day to get the pages a competitor or news site added or changed since yesterday.
• hreflang checks: see which language versions each page declares.

FAQ

What if a site has no sitemap? The Actor checks robots.txt and the common paths. If none of them has a sitemap, the site is listed in SUMMARY with that reason and costs nothing. It does not crawl the site's links to build a URL list; for that, use a crawler.

Does it follow sitemap indexes on other domains? Yes. Sitemap URLs listed in robots.txt or in an index are followed wherever they point, and page URLs on other hosts are kept, because a sitemap can list pages on a subdomain or another domain the site owns.

What if I give a website with a path, like example.com/blog? robots.txt is read at the domain root, so you get every sitemap of the domain, and /blog/sitemap.xml and similar paths are checked too. Add /blog/ to Only URLs matching to keep just that section.

Why is lastmod empty for some URLs? The site did not include it. lastmod, changefreq and priority are optional in the sitemap protocol, and many sites leave them out. With a date filter, URLs without lastmod are dropped unless you turn on Keep URLs without a lastmod date.

How fast is the status check, and will it get me blocked? It runs 10 checks in parallel by default (up to 50). One test run checked 300 URLs on three sites in 23 seconds. If a site answers 429 Too Many Requests, the Actor waits and retries, but lower Status checks in parallel for sites that rate-limit. A site that blocks datacenter traffic may answer 403 to every request; try Apify Proxy for those.

What does the status check not tell me? It reads only the HTTP status and redirects. It does not render JavaScript, so a page that returns 200 but shows "not found" in the browser (a soft 404) is reported as 200.

Are there size limits? A sitemap file can hold up to 50,000 URLs by the protocol, and the Actor reads files of that size. It stops after 10,000 sitemap files per site as a safety limit.

Other Actors by this developer

• SEO Site Audit: crawls a site and lists broken links, redirect chains, missing titles and descriptions, canonical, hreflang and sitemap errors.
• Website Change Monitor: watch the pages from your sitemap and get the exact lines that changed, with alerts by email, Slack or webhook.
• Lighthouse and PageSpeed Audit: Lighthouse scores and Core Web Vitals for many URLs, no PageSpeed API key.
• Website Tech Stack Detector: the CMS, analytics, frameworks and hosting behind a list of sites.
• Website Screenshot and PDF: full-page screenshots or PDFs of a list of URLs.