Pricing

from $0.30 / 1,000 urls

Try for free

Go to Apify Store

Sitemap URL Extractor: All Pages from robots.txt & Sitemaps

Try for free

Extract every URL a website publishes in its XML sitemaps: robots.txt discovery, nested indexes, gzip, hreflang, images, news. Fast and cheap — no browser. $0.30 per 1,000 URLs.

Pricing

from $0.30 / 1,000 urls

Rating

0.0

(0)

Developer

Andrew Shamah

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Sitemap URL Extractor: All Website Pages from robots.txt & XML Sitemaps

Turn any domain into a complete list of sitemap URLs — every page the site publishes in its XML sitemaps, with lastmod, changefreq, priority, hreflang alternates, image/video counts and Google News titles. No browser needed: it only reads robots.txt and sitemap files, so it is fast (~170k URLs / 30s in our tests) and cheap.

$0.30 per 1,000 URLs ($0.0003 each). Error rows are free.

What it handles

Discovery from robots.txt Sitemap: lines, then common fallbacks (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, .gz, .txt)
Nested sitemap indexes (up to 6 levels), gzip, plain-text, RSS/Atom-as-sitemap
Extensions: hreflang, image, video, news title
Include/exclude globs or regex, modified after date, only-new monitoring mode

Fast free-tier try

Prefill uses a tiny sitemap and a low URL cap so a Try Actor run finishes in seconds:

{
  "startUrls": ["https://www.sitemaps.org/sitemap.xml"],
  "maxUrlsTotal": 50,
  "maxUrlsPerSite": 50
}

Example output

Field	Example
`url`	`https://www.example.com/products/123`
`lastmod`	`2026-10-27T22:49:48.000Z`
`changefreq` / `priority`	`daily` / `0.8`
`imageCount` / `videoCount`	`1` / `0`
`newsTitle`	Headline when present
`sitemap`	Source sitemap URL
`site`	Origin

{
  "url": "https://www.nytimes.com/2026/10/27/opinion/example.html",
  "lastmod": "2026-10-27T22:49:48.000Z",
  "imageCount": 1,
  "videoCount": 0,
  "newsTitle": "Example headline",
  "sitemap": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
  "site": "https://www.nytimes.com",
  "error": null
}

Pricing

Actor start + $0.0003 per URL written. Default safety cap maxUrlsTotal=100000 (set 0 for no limit). Use Maximum cost per run to stop cleanly.

Limits

Only URLs listed in sitemaps (not a full crawler)
lastmod is whatever the site reports
Follows robots.txt Disallow when probing fallback paths

SEO & web suite

Documents suite

Jobs suite

Also by us

Sitemap Extractor: Website → All URLs (sitemap.xml parser)

boxbox10/sitemap-extractor

Give it a website. Get every URL from its sitemap — loc, lastmod, changefreq, priority — as one clean record per URL. Auto-discovers sitemap.xml, robots.txt Sitemap: directives, and nested sitemap indexes. Perfect for SEO audits, crawl seeding, and URL discovery.

Marvin Eguilos

Sitemap URL Extractor — All Page URLs from a Website

power_on/sitemap-url-list

Input: domains or sitemap URLs. Output: one row per page URL with its lastmod, changefreq and priority, plus a report row per sitemap file read. Finds sitemaps via robots.txt, follows nested indexes, unpacks gzip. Filter by URL pattern or by date. Hard cap of 25,000 URLs per run.

Power On Labs

Sitemap URL Harvester

mahogany_songbird/sitemap-url-harvester

Collect URLs from XML sitemaps for SEO and crawling.

Britton Furness

Sitemap URL Extractor — Robots.txt & Sitemap.xml Parser

tarnlight/sitemap-url-extractor

Extracts the URLs a website lists in its XML sitemaps (sitemap indexes, gzip and text sitemaps, robots.txt discovery), up to your Max URLs cap, with filters and a clear per-site report.

Tarnlight

Sitemap Extractor – All URLs from sitemap.xml + Status Check

gazidev/sitemap-url-extractor

Extract every URL from any website's sitemap.xml: auto-discovery via robots.txt, nested sitemap indexes, .xml.gz, image/news/video/hreflang tags and lastmod. Filter by date or glob, get only new URLs, and optionally check HTTP status, redirects and canonical per URL.

Cemal Atakli

Sitemap URL Extractor

sootesting/sitemap-url-extractor

Extract URLs from XML sitemaps in bulk. Supports sitemap indexes, auto-discovers nested sitemaps, and outputs structured JSON. Pay-per-event: per 1,000 URLs extracted.

soot

Sitemap URL Extractor - XML Sitemap Scraper

benthepythondev/sitemap-url-extractor

Extract URLs from XML sitemaps and sitemap indexes. Get URL, lastmod, changefreq, priority and source sitemap.

Ben

Sitemap URL Finder

thescrapelab/sitemap-target-url-extractor

Find useful URLs from robots.txt and sitemaps, then output deduplicated URL inventory rows with source sitemap, domain, path, filter, and crawl summary data for SEO and scraping workflows.

Inus Grobler

Sitemap URL Extractor

appreciative_duct/sitemap-url-extractor

Extract all URLs from website sitemaps. Finds sitemaps in robots.txt, follows sitemap indexes, and reads gzip, news, image, and video sitemaps.

Ross AI

Sitemap URL Extractor

sampon/sitemap-url-extractor

List every URL a website publishes in its XML sitemaps, found automatically through robots.txt. Filter by text or lastmod date, or get only the URLs added since your last run.