Sitemap URL Extractor: All Pages, Lastmod & Status

Get every page URL a website lists in its sitemaps, with lastmod dates, change frequency, priority, hreflang alternates, image counts and Google News titles. Just type a domain: sitemaps are found automatically from robots.txt and common paths, and sitemap indexes (including .xml.gz) are followed all the way down.

Fast and cheap: a 3,000-URL site takes about a second.

What you can do with it

• Crawl planning: feed a clean, de-duplicated URL list into a content scraper instead of crawling blindly.
• Change monitoring: schedule it with Changed on or after to get only new or updated pages (new blog posts, products, job pages, news articles).
• SEO audits: find broken links and redirects listed in the sitemap (Check HTTP status), pages with missing lastmod, and hreflang coverage.
• Competitor research: see how many product, blog or landing pages a competitor has, and how fast they publish.
• News tracking: read news sitemaps for headline + publication time, minutes after publishing.

Input

Field	What it does
urls	Domains, homepages, section URLs (https://www.bbc.com/news) or sitemap URLs. One per line.
onlyUnderPath	For a section URL, return only pages under that path (on by default).
includeUrlPatterns / excludeUrlPatterns	Keep or drop URLs containing these strings.
lastmodAfter	Only pages changed on or after this date.
checkStatus	Adds statusCode, finalUrl and httpLastModified per URL (broken links, redirects, server-side change date).
maxUrlsPerSite, maxItems	Caps to control cost. maxItems is shared fairly: every site gets an equal share, and sites with fewer URLs pass the rest on.

Example:

{ "urls": ["apify.com", "https://www.bbc.com/sitemaps/https-index-com-news.xml"], "includeUrlPatterns": ["/news/"], "lastmodAfter": "2026-10-01" }

Output

One item per URL:

{
  "url": "https://wordpress.org/news/2026/09/some-post/", "lastmod": "2026-09-01T12:37:54Z",
  "changefreq": "weekly", "priority": 0.8, "alternates": [{"hreflang": "de", "url": "https://..."}],
  "imageCount": 2, "newsTitle": null, "newsPublishedAt": null,
  "statusCode": 200, "finalUrl": "https://...", "httpLastModified": "Tue, 01 Sep 2026 12:37:54 GMT",
  "site": "https://wordpress.org", "sitemap": "https://wordpress.org/news/wp-sitemap-posts-post-1.xml"
}

statusCode/finalUrl/httpLastModified appear only with Check HTTP status. A per-site summary (URLs returned, sitemaps read, errors, and whether Max URLs in total cut a site short) is saved to the run's key-value store as SUMMARY.

Pricing

Pay per URL, with lower prices on higher Apify plans: $0.40 per 1,000 URLs on the free plan, down to $0.08 per 1,000 on Gold and above. Filters and caps apply before you are charged. A run where no sitemap can be found fails clearly instead of silently returning an empty dataset. A filtered run with no new pages succeeds with 0 results.

Limits and notes

• Reads only what a site publishes in its sitemaps. Pages not listed there won't appear (it doesn't crawl links).
• Up to 300,000 URLs and 1,000 sitemap files per site, and 60 MB per sitemap file (the sitemap spec allows 50 MB). Memory use stays flat whatever the site size.
• Section URLs: if a site lists nothing under the path in 100 sitemap files in a row, the search for that section stops there (noted in SUMMARY).
• With Changed on or after, URLs without a lastmod date are skipped.
• Public data only. Status checks send one light request per URL; use caps for very large sites.

Feedback and requests

We welcome feedback: a field or option you wish this had, a change to how it works, or a scraper you would like us to build. Open an issue in the Issues tab of this page and tell us what you need. We read every one, and we build the requests that come up most. If this actor works well for you, a short review helps other people find it.