Sitemap URL Extractor: All Site URLs, Lastmod & News

Give it a list of websites (or sitemap addresses) and get back every page URL the site lists in its XML sitemaps, one row per URL, with the details the sitemap gives: last modified date, change frequency, priority, and image, news and video information when the sitemap has it.

It's the fast way to get a full list of a site's pages for an SEO audit, a content inventory, a migration checklist, a crawl seed list, change monitoring, or feeding pages into another scraper, without crawling the site page by page.

You pay per URL saved: $0.50 per 1,000 URLs. Websites with no sitemap, blocked by robots.txt or unreachable still get one free row saying why.

What it does

• Finds the sitemaps for you. It reads the Sitemap: lines in the site's robots.txt first. If those give nothing readable, it tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml, in that order, and stops at the first real sitemap. Or give it a sitemap address directly, like https://example.com/sitemap_news.xml.
• Follows sitemap indexes down to 4 levels, reading child sitemaps in the order the index lists them.
• Reads what sites actually publish: XML sitemaps, sitemap indexes, gzip-compressed .xml.gz files, gzip or deflate transfer compression, plain-text sitemaps (one URL per line), and RSS or Atom feeds listed as sitemaps.
• Extension data when present: image count and image URLs (image:image), Google News title, publication date, publication name and language (news:news), and video count, title and thumbnail (video:video).
• Filters before you pay: keep only URLs containing some text (/blog/) or matching a regular expression, drop others, or keep only URLs changed after a date. Filtered-out URLs are not charged.
• Copes with messy sitemaps: URLs with unescaped &, stray whitespace, BOMs, XSL stylesheets, cut-off files (URLs before the break are kept, the half URL is dropped) and HTML "soft 404" pages served as /sitemap.xml (recognized and skipped).
• Repeats removed: a URL listed in two sitemaps of the same site is saved once.

Compared with crawling the site

	This tool	Crawling the site's pages to collect URLs
What is downloaded	Only robots.txt and the sitemap files	Every page of the site
Requests for a 10,000-page site	Usually 2 to 10	About 10,000
Last modified, priority, news and image data	Yes, as the site declares them	Not usually available
Pages the site doesn't link to	Included if the sitemap lists them	Missed
Pages missing from the sitemap	Missed	Found if linked
Page content (titles, text)	No (feeds give a title)	Yes

If you need what's on each page, use a crawler. If you need the list of pages, the sitemap is the site's own answer and much cheaper to read.

Input

Field	What it does
Websites or sitemap addresses	One per line: example.com, https://www.example.co.uk/, or a sitemap or feed address like https://example.com/sitemap_index.xml. A bare domain tries https, then https://www., then http.
Max URLs per website	Reading stops once this many URLs of a site are saved (default 10,000, max 1,000,000).
Only URLs containing	Keep URLs containing any of these texts (case-insensitive). Start an entry with re: for a regular expression, e.g. re:/20(25|26)/.
Leave out URLs containing	Drop URLs containing any of these. re: works here too.
Changed on or after	Keep URLs whose lastmod (or feed date) is on or after a date (2026-01-31) or a relative time (30 days). URLs without a date are left out, and child sitemaps the index says last changed before the date are skipped without downloading them.
Max sitemap files per website	Most sitemap files read per site, indexes included (default 200).
Websites at once	Parallel websites (default 5). Each website still gets at most one request per second.

Example input:

{
  "startUrls": ["nodejs.org", "allbirds.com", "https://www.theguardian.com/sitemaps/news.xml"],
  "maxUrlsPerSite": 5000,
  "includeFilter": ["/products/", "re:/20(25|26)/"],
  "excludeFilter": ["?page="],
  "lastmodAfter": "90 days"
}

Output

One row per URL (values illustrative):

{
  "url": "https://www.example-news.com/2026/11/election-night/",
  "lastmod": "2026-11-02T06:00:00+00:00",
  "changefreq": "hourly",
  "priority": 0.8,
  "imageCount": 2,
  "imageUrls": ["https://www.example-news.com/img/results.jpg", "https://www.example-news.com/img/crowd.jpg"],
  "newsTitle": "Election night: results as they come in",
  "newsPublicationDate": "2026-11-02T05:30:00+00:00",
  "newsPublicationName": "Example News",
  "newsLanguage": "en",
  "sitemapUrl": "https://www.example-news.com/news-sitemap.xml",
  "sitemapType": "urlset",
  "site": "example-news.com"
}

• lastmod is normalized to ISO 8601 (2026-11-02T06:00:00+00:00, or 2026-11-02 for a date only, keeping the site's own time zone). A date the tool can't read is kept as the site wrote it. null when the sitemap gives none.
• changefreq and priority are null when missing; a priority outside 0.0 to 1.0 is dropped.
• imageCount/imageUrls (first 10), newsTitle/newsPublicationDate/newsPublicationName/newsLanguage and videoCount/videoTitle/videoThumbnailUrl appear only when the sitemap has them. Feed rows have a title.
• sitemapType is urlset (XML sitemap), text, rss or atom.

A website where no URL could be read gets one free row with url: null, a status and an error saying why:

• no_sitemap_found: robots.txt lists no readable sitemap and none of the usual locations has one.
• blocked_by_robots: the site's robots.txt asks tools like this one not to read its sitemaps (or answered with a server error, which per the standard means stay away).
• unreachable: the site didn't answer.
• no_urls: sitemaps were read, but they list no URLs, or none passed your filters.
• invalid_url: the input isn't a usable public web address.

The run also saves a SUMMARY record: totals, the list of sites without a sitemap, and for each site the sitemap files read (with any problem per file), URLs listed, saved, filtered and repeated, and whether the URL cap, time limit or spending cap was reached.

Download results as JSON, CSV or Excel, or send them to Google Sheets, Zapier, Make or n8n.

Pricing

Pay per event: $0.0005 per URL row ($0.50 per 1,000 URLs). No start fee and no charge per sitemap file, so a site with 100 small sitemaps costs the same per URL as a site with one big one. Rows for sites without URLs are free, and URLs your filters drop are never charged.

Set a maximum charge per run in the run options: rows are saved and charged in batches, and the run stops exactly at the limit (a batch is cut to what's left) and says so in the log and the SUMMARY.

Good to know

• Sitemaps show what the site chooses to list. Some sites list only part of their pages, some list pages that no longer exist, and lastmod dates are only as accurate as the site keeps them.
• Limits per website: 4 levels of sitemap indexes, 200 sitemap files by default (up to 5,000), 100 MB read from one file (the sitemap standard's own limit is 50 MB), 500 MB per website in total, and 20 minutes per website. A file cut at a limit still gives the URLs read before it, and the SUMMARY says which file was cut.
• Order: URLs come in the order the sitemaps list them, so with a URL cap you get the first sitemaps first. Use lastmodAfter to get recent pages from big archives.
• URLs on other hosts are kept as listed (some sites list a blog. or shop. subdomain in the main sitemap); filter them with Leave out URLs containing if you don't want them.

Responsible use

• It reads only public sitemap files that websites publish for search engines and crawlers. It never logs in and never downloads the pages themselves.
• It respects robots.txt for every file it reads, waits at least a second between requests to the same website (longer if robots.txt asks for a crawl delay), retries politely on 429 and server errors, and names itself in its User-Agent (SitemapUrlExtractorBot).
• It refuses local and private network addresses (localhost, private IP ranges, .internal names and domains that resolve to them).
• Sitemap files are parsed as a stream with safety limits: XML entity declarations (the "billion laughs" trick) are refused, compressed files are inflated in small steps with a size cap, and nothing in a file can make the tool fetch anything outside the sitemap tree.

This Actor isn't affiliated with or endorsed by any of the websites it reads.

More tools from the same developer

• Broken Link Checker: 404s, Redirects & Dead Links
• SEO Audit Tool: On-Page Checks, Crawl & Broken Links
• Bulk URL Status Checker: Redirect Chain & Response Time
• Wayback Machine Scraper: Snapshots, Changes & Page Text
• Website Screenshot & PDF: Full Page, Mobile, Batch URLs
• Domain WHOIS, DNS & SSL Checker: Expiry, SPF & DMARC