Sitemap URL Extractor
Pricing
Pay per usage
Sitemap URL Extractor
Get every page URL a website lists in its sitemaps. Finds sitemaps via robots.txt and common paths, follows nested indexes, reads .gz, text, news and RSS sitemaps, and filters by date or pattern.
Pricing
Pay per usage
Rating
0.0
(0)
Developer
Pipwort Data
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
14 hours ago
Last modified
Categories
Share
Get every page URL a website publishes in its sitemaps, as a clean table you can export to CSV, Excel or JSON.
Give it a home page or a sitemap link. It reads robots.txt, checks the usual sitemap locations, follows nested sitemap indexes, and returns one row per page with its last-modified date.
What it handles
- Discovery:
Sitemap:lines inrobots.txt, then/sitemap.xml,/sitemap_index.xml,/wp-sitemap.xml,/sitemap.txtand similar. - Formats: XML sitemaps and sitemap indexes (nested up to 10 levels),
.gzfiles, plain-text sitemaps, and RSS or Atom feeds used as sitemaps. - Extras: image sitemaps (
image:image), hreflang alternates (xhtml:link) and Google News sitemaps (title and publication date). - Filters: last-modified date range, include and exclude patterns (regular expressions), same-domain only, and per-site or total limits.
- Reliability: retries with backoff on timeouts, 429 and 5xx responses, a hard time limit per request, and safe XML parsing. A broken sitemap is logged and skipped, and the rest of the run carries on.
Common uses
- Building a crawl list for SEO audits, content inventories or site migrations
- Feeding page lists to an AI or RAG pipeline
- Tracking new or updated pages on a site over time (use the date filter on a schedule)
Input
| Field | What it does |
|---|---|
startUrls | Home pages or direct sitemap URLs (required) |
lastmodAfter / lastmodBefore | Keep only pages modified in a date range |
includePatterns / excludePatterns | Regular expressions, for example /blog/ |
maxUrlsPerSite / maxUrlsTotal | Stop after this many URLs (0 means no limit) |
includeImages, includeAlternates | Add image URLs and language alternates |
sameDomainOnly | Drop URLs on other domains (on by default) |
Output
A SUMMARY record in the run's key-value store lists, for each site, the sitemaps found, any that failed and why, and how many URLs were returned. If a site publishes no sitemap, the run says so plainly and you are not charged for any results.
Pricing
Pay per result: US$0.50 per 1,000 URLs, plus a small start fee of US$0.005 per run. You can cap the cost of any run with the "Maximum cost per run" setting. The Actor stops cleanly when it reaches your limit.
Good to know
- The Actor reads only
robots.txtand the sitemaps a site publishes for automated tools. It does not load page content. - Some sites block automated requests to their sitemaps (you'll see
HTTP 403in the summary). The Actor does not try to get around these blocks. lastmodis only as accurate as the website makes it. Many sites leave it out. UsekeepUrlsWithoutLastmodto decide what to do with those URLs.
Support
Found a sitemap that doesn't parse? Open an issue on the Actor's Issues tab with the URL, and we'll look at it.
