Sitemap URL Extractor (index, gzip, robots.txt discovery)
Pricing
from $0.20 / 1,000 url extracteds
Sitemap URL Extractor (index, gzip, robots.txt discovery)
Give it domains or sitemap URLs. It finds sitemaps through robots.txt and common paths, follows sitemap indexes (including .xml.gz), and returns every page URL with lastmod, changefreq and priority. Optional filters by path pattern and lastmod date.
Pricing
from $0.20 / 1,000 url extracteds
Rating
0.0
(0)
Developer
Honest Agent Kit
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
13 days ago
Last modified
Categories
Share
Sitemap URL Extractor
Give it domains or sitemap URLs and get every page URL back as a dataset (JSON, CSV or Excel).
- Finds sitemaps automatically from robots.txt, then /sitemap.xml and /sitemap_index.xml
- Follows sitemap indexes, reads gzip (.xml.gz) and plain-text sitemaps, and removes duplicates
- Each row includes the URL, lastmod, changefreq, priority and the sitemap it came from
- Filters: a path regex (for example
/blog/) and "modified after" a date
Good for: SEO audits, content inventories, change monitoring (run it weekly with modifiedAfter), and feeding URL lists to other scrapers.
Pricing
Pay per event: url-extracted for each URL row returned. Error rows (no sitemap found, invalid input) are free. Set Max URLs to cap your cost. See PRICING.md.
Limits
It reads only what the site publishes in its sitemaps. It does not crawl pages, render JavaScript or get around blocks.
Built by a retired military veteran. Built with AI coding assistance and tested offline.
