Sitemap URL Extractor – All Page URLs + Lastmod

Get every page URL of a website from its sitemaps, with the last-modified date, change frequency and priority the site publishes, and optionally each URL's HTTP status. Give it a website (https://example.com) or a sitemap URL; it finds the sitemaps through robots.txt or /sitemap.xml, follows sitemap indexes and .gz files, and returns one clean, deduplicated row per page.

Plain HTTP, no browser, so even sites with tens of thousands of pages are read in seconds to minutes.

Who uses it

• SEO specialists: audit what a site submits to search engines, find 404s and redirects listed in the sitemap, and compare sitemap coverage with your crawl.
• Content and competitor research: list every blog post, product or docs page of a competitor, and see what was added or updated recently with Only pages modified on or after.
• Developers and data teams: get the URL list to feed a scraper, a link checker, a migration or an AI/RAG pipeline, without crawling the whole site.
• Monitoring: schedule it daily with a date filter to get only new or updated pages.

How to use it

1. Put one or more websites or sitemap URLs in Websites or sitemap URLs (up to 100).
2. Optionally narrow the output: Only URLs matching (e.g. /blog/), Skip URLs matching, Only pages modified on or after, and the URL limits. Turn on Check each URL's HTTP status to find broken links and redirects.
3. Run it and download the URLs as JSON, CSV or Excel, or pass them to another Actor or tool.

Input example

{
  "urls": ["https://docs.apify.com", "https://example.com/post-sitemap.xml"],
  "includePatterns": ["/blog/", "/academy/"],
  "excludePatterns": ["/tag/"],
  "lastmodAfter": "2026-09-01",
  "maxUrls": 5000,
  "maxUrlsPerSite": 2000,
  "checkStatus": false
}

How sitemaps are found for a website: first the Sitemap: lines of its robots.txt, otherwise /sitemap.xml, then /sitemap_index.xml. If a site keeps its sitemap somewhere else, enter that sitemap URL directly.

Output example

{
  "url": "https://docs.apify.com/academy",
  "site": "docs.apify.com",
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "lastmod": "2026-09-30T10:00:00+00:00",
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "httpStatus": 200,
  "finalUrl": null,
  "statusNote": null,
  "extractedAt": "2026-10-07T19:30:00Z"
}

Field	Type	Description
url	string	Page URL listed in the sitemap
site	string	Host of the website or sitemap you entered
sitemapUrl	string	The sitemap file that lists this URL
lastmod	string / null	Last modification date as published in the sitemap (W3C datetime)
changefreq	string / null	always, hourly, daily, weekly, monthly, yearly or never
priority	number / null	0.0 to 1.0, as published
imageCount	integer	Images listed for this URL in an image sitemap
httpStatus	integer / null	HTTP status (only with Check each URL's HTTP status)
finalUrl	string / null	Where the URL redirects to (only with the status check; null when no redirect)
statusNote	string / null	Why a URL was not checked or the check failed
extractedAt	string	ISO 8601 UTC

Sitemaps that could not be read are listed in the run's key-value store as ERRORS with the reason (NO_SITEMAP, NOT_FOUND, NOT_A_SITEMAP, ROBOTS_DISALLOWED, ROBOTS_UNREACHABLE, BLOCKED, FETCH_FAILED, TOO_MANY_SITEMAPS, TOO_DEEP, REFUSED), and a per-site summary as SUMMARY (sitemaps read, URLs found, URLs saved).

Use it with

• Schedules: save the input as a Task with a modified on or after date and run it daily or weekly.
• Other Actors: pass the URLs to a scraper or to a change monitor as its start URLs.
• Make, Zapier, n8n, Google Sheets, Slack: through the Integrations tab.
• API and code: the Apify API, the JavaScript and Python clients, or CSV/JSON/Excel downloads.
• AI agents: the Apify MCP server exposes this Actor as a tool; the input is a list of websites.

Pricing

Pay per URL: $0.30 per 1,000 URLs on the Apify Free plan; on paid plans $0.27 (Starter), $0.18 (Scale) and $0.09 (Business and higher) (the price for your plan is shown on the Pricing tab). The examples below use the Free-plan price. Only saved URLs are charged; URLs removed by your filters, duplicates and sitemaps that fail are free. The status check is included in the price.

• A 1,000-page blog → about $0.30.
• A 20,000-page shop → about $6.00.
• 50 competitor sites, new pages only, checked weekly (about 2,000 URLs/month) → about $0.60/month.

A small start fee of $0.00005 per run applies. Set a maximum cost per run in the run options and the Actor stops cleanly when it is reached; Max URLs per run does the same by count.

Limits and notes

• Only what the site publishes in its sitemaps. Pages that are not in any sitemap are not found (this Actor does not crawl links). Sitemaps follow the sitemaps.org protocol: XML urlset and sitemapindex, gzip, plain-text lists, RSS 2.0 and Atom.
• Up to 100,000 URLs per run, 500 sitemap files and 5 index levels per site, 50 MB per sitemap (the protocol's own limit).
• robots.txt is respected (RFC 9309, including * and $ rules): sitemaps and status checks that it disallows are skipped. If robots.txt itself can't be read (server error), the site is skipped to be safe.
• No bot-check bypass, no proxies, no logins. A Cloudflare/DataDome-style challenge, a 401/403 or a persistent 429 on a sitemap is reported as BLOCKED and not charged. Private or internal addresses are refused.
• Politeness: at most two requests per second per website, five websites in parallel.
• lastmod is shown exactly as the site publishes it; some sites set it to the build time of the whole site, so it is only as good as the site's sitemap.

FAQ

Is it legal to extract sitemaps? Sitemaps are files that websites publish specifically so that crawlers can find their pages, and this Actor reads only those files (plus, if you ask, the HTTP status of the listed pages) where robots.txt allows it, at a polite rate, without logging in or getting around any protection. It outputs URLs and dates, no personal data. What you do with the pages afterwards depends on each site's terms; if in doubt, ask a lawyer.

Why did I get NO_SITEMAP? The site lists no sitemap in robots.txt and has none at /sitemap.xml or /sitemap_index.xml. Look for it in the site's footer or in Google Search Console and enter the sitemap URL directly.

Why are there fewer URLs than pages on the site? The site's sitemaps don't list them all, or your filters or limits removed them. The SUMMARY record shows how many URLs each site's sitemaps contained and how many were saved.

Can it find broken links? Yes, for URLs in the sitemap: turn on Check each URL's HTTP status and filter the output for httpStatus 4xx/5xx or a non-empty finalUrl (redirects).

Does it work on WordPress, Shopify, Wix and Yoast sitemaps? Yes: they are standard sitemap indexes (often sitemap_index.xml or sitemap.xml) and are followed automatically.

Changelog

See CHANGELOG.md.

Support

Found a problem or need a feature? Open an issue on the Actor's Issues tab. We read every one.