Sitemap URL Extractor — Robots.txt & Sitemap.xml Parser
Pricing
Pay per event
Extracts the URLs a website lists in its XML sitemaps (sitemap indexes, gzip and text sitemaps, robots.txt discovery), up to your Max URLs cap, with filters and a clear per-site report.
Site homepages (e.g. https://example.com) or direct sitemap URLs (e.g. https://example.com/sitemap.xml). A homepage/site URL triggers robots.txt Sitemap: discovery and common-path probing. A direct sitemap URL is fetched by itself, and ONLY that file (plus its sitemap-index children, if any) -- it never expands into a whole-site discovery, so you're never charged for URLs you didn't ask for.
Stop once this many URLs have been written to the dataset across all start URLs (0 = unlimited -- can run up an unbounded bill on a large site; set a cap unless you really want everything). Each URL costs $0.001, so the default of 5,000 caps a run at about $5 (plus the $0.005 start fee).
Only keep extracted URLs matching at least one of these regular expressions. Leave empty to keep all.
[]Drop extracted URLs matching any of these regular expressions.
[]ISO date (e.g. 2025-01-01). Only keep URLs whose is on or after this date. See 'Keep URLs without lastmod' for URLs missing a lastmod value.
Only matters when 'Modified since' is set. Off (default): URLs with no usable (missing, or a placeholder like 0000-00-00) are dropped and not charged, so you only pay for URLs known to match your date. On: keep them too (their date is unknown).
For a homepage/site-root start URL: use robots.txt's 'Sitemap:' lines for discovery (in addition to any common-path probing). Has no effect on a direct sitemap URL, which is always fetched by itself. robots.txt is always fetched and its Disallow rules are always honoured regardless of this setting; it only controls whether declared sitemaps are added to the discovery list.
For a homepage/site-root start URL: also probe well-known paths (/sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt) at its origin, regardless of what robots.txt declares (some sites keep a stale legacy sitemap at a common path alongside a different declared one). Has no effect on a direct sitemap URL, which is always fetched by itself.
Include <xhtml:link rel="alternate" hreflang=...> entries for each URL in the output.
Include image count (and image URLs) from image:image entries in the output.
Maximum number of sitemap/robots.txt requests in flight at once, across all sites.