Pricing

Pay per usage

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

vøiddo

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Example output row

{
  "domain": "vercel.com",
  "url": "https://vercel.com/blog/nextjs-14",
  "lastmod": "2024-03-15",
  "changefreq": "weekly",
  "priority": 0.8,
  "source": "https://vercel.com/sitemap-blog.xml"
}

How to use

Input

Field	Type	Default	Description
`domains`	`string[]`	`["stripe.com","shopify.com","vercel.com"]`	Domains to crawl — no scheme, no trailing slash
`maxUrlsPerDomain`	`integer`	`2000`	Hard cap on URLs returned per domain
`followSitemapIndex`	`boolean`	`true`	Recursively follow `<sitemapindex>` child links (up to depth 5)

Minimal run

{
  "domains": ["example.com"],
  "maxUrlsPerDomain": 500,
  "followSitemapIndex": true
}

Output fields

Field	Type	Notes
`domain`	string	Input domain
`url`	string	Discovered URL from `<loc>`
`lastmod`	string	ISO date, `null` if absent
`changefreq`	string	e.g. `weekly`, `null` if absent
`priority`	float	0.0–1.0, `null` if absent
`source`	string	Sitemap file the URL was found in

Pricing

Event	Cost	When charged
`url_extracted`	$0.0001 per URL	Once per run, total = URLs pushed

A 2 000-URL run costs $0.20. Unused budget is not charged — if a domain has only 300 URLs you pay for 300.

Buyer

SEO teams auditing crawl coverage — verify every page is in the sitemap.
Content operations checking lastmod staleness across thousands of URLs.
Competitive intelligence — map a competitor's full URL structure.
QA pipelines validating sitemap health after deploys.
Link-building researchers finding indexable pages at scale.

Source

Crawl order per domain:

GET https://{domain}/robots.txt — parse all Sitemap: lines.
If none found, fall back to GET https://{domain}/sitemap.xml.
For each sitemap URL: fetch + parse XML.
If <sitemapindex>, enqueue each <sitemap><loc> (up to depth 5).
If <urlset>, emit one row per <url> until maxUrlsPerDomain is reached.

All requests use a polite User-Agent and are paced at 250–600 ms between calls. 404 and empty responses are skipped gracefully.

Sitemap to URL Crawler — Extract Sitemap.xml URLs for RAG

logiover/sitemap-to-url-crawler

Extract all URLs from any sitemap.xml recursively. Export sitemap URLs to CSV/JSON for RAG pipelines, SEO audits, and LLM training datasets.

Logiover

Sitemap URL Extractor

wiry_kingdom/sitemap-url-extractor

Extract every URL from any website's sitemap.xml with lastmod, changefreq, priority. Recursively expands sitemap index files, reads robots.txt, handles gzipped sitemaps. SEO audits, content migration, site inventory, competitor research.

Mohieldin Mohamed

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.

Daniel Brenner

Sitemap URL Extractor

seemuapps/sitemap-extractor

Extract every URL from a website's sitemap.xml. Recursively walks nested sitemap indexes and returns loc, lastmod, changefreq, and priority for each page.

Andrew

Sitemap URL Finder

thescrapelab/sitemap-target-url-extractor

Find and export URLs from any website’s robots.txt and sitemaps. Enter a domain or website URL, optionally filter matching URLs by text, and get clean dataset rows with the URL, domain, path, source sitemap, and match details.

Inus Grobler

Sitemap Crawler - XML Sitemap URL Extractor

miccho27/sitemap-crawler

Extract all URLs from XML sitemaps (including sitemap index) and optionally audit each page

Tatsuya Mizuno

Sitemap Finder & URL Extractor · Crawl Any XML Sitemap

corent1robert/sitemap-detector

Find and crawl XML sitemaps from any website. Follows sitemap indexes, handles gzip, and exports every page URL with source file and lastmod into a clean dataset. No config needed.

Corentin Robert

Sitemap & URL Discovery - Find All URLs on Any Site

santamaria-automations/sitemap-url-discovery

Discover every URL on any website by parsing sitemap.xml, robots.txt, and sitemap indexes. Extract URLs with last modified dates, change frequency, and priority. Perfect for SEO audits, content analysis, crawling preparation, and site mapping.

Ale

Sitemap to URL List Extractor

scrapeworks/sitemap-to-urls

Extract every URL from any website's sitemap as clean JSON. Handles sitemap indexes (recursive) and gzipped sitemaps automatically. Includes lastmod, priority, and changefreq.

Nicolas van Arkens

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.