Sitemap URL Extractor — robots.txt, Sitemap Index, gzip

Sitemap URL Extractor: robots.txt, gzip, Changes

succinct_glider/sitemap-urls

Extract every URL from a website's XML sitemaps: found through robots.txt, sitemap indexes and .gz files followed, lastmod and date filters, URL patterns, and new, changed or removed URLs since your last run. Reads sitemap files only, never the pages.

Sitemap URL Extractor

rowfeed/sitemap-url-extractor

Extract every URL from any site's sitemaps: robots.txt discovery, sitemap-index recursion, gzip, plain-text and RSS/Atom sitemaps, lastmod/regex filters. Plain HTTP, no browser, no proxies.

Sitemap Extractor & URL Change Monitor

automa-flow/sitemap-extractor-monitor

Extract every URL from XML, gzip, text and nested sitemaps, with robots.txt discovery. Compare repeat checks for added URLs, publisher lastmod updates and confirmed removals. An incomplete sitemap tree never produces false removals. Export for SEO audits and incremental crawlers.

Sitemap URL Extractor — All Page URLs from a Website

power_on/sitemap-url-list

Input: domains or sitemap URLs. Output: one row per page URL with its lastmod, changefreq and priority, plus a report row per sitemap file read. Finds sitemaps via robots.txt, follows nested indexes, unpacks gzip. Filter by URL pattern or by date. Hard cap of 25,000 URLs per run.

Sitemap URL extractor that reads robots.txt, sitemap indexes, .xml.gz and plain-text sitemaps and returns one row per URL with lastmod, changefreq, priority — plus a new/removed diff between runs.

Sitemap URL Extractor

leorochasantos/sitemap-extractor

Extract every URL from any XML sitemap — support sitemap indexes, gzip, robots.txt discovery, and metadata extraction. Returns clean structured output: one URL per row.

Extract every page URL from the sitemap.xml of any website, in bulk. Give the Actor a domain and it finds the sitemaps through robots.txt, follows sitemap indexes recursively, reads .xml.gz files, removes duplicates and returns one clean row per URL with lastmod, changefreq, priority and the...

Best Damn Sitemap URL Extractor

josh99smith/sitemap-url-extractor

Extract every URL from a website's XML sitemaps, including sitemap indexes, gzip files and robots.txt discovery, with lastmod, priority and hreflang. Give it a domain, get the whole map back, at a tiny price per URL.

XML Sitemap URL Extractor & Checker

scrapesignal_labs/sitemap-url-extractor

Extract URLs from XML sitemaps, sitemap indexes, robots.txt, and gzip files. Filter pages and optionally check HTTP status codes for SEO audits.

Sitemap URLs: every URL of a website from its XML sitemaps

steadydata/sitemap-urls

Sitemap URL extractor: every URL a website lists in its XML sitemaps, up to 100 sites per run: sitemaps are found through robots.txt and the usual paths, sitemap indexes and gzip are followed, and each URL comes with its sitemap, last modification date, change frequency and priority. Pay per URL.