Pricing

Pay per usage

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl

Discover every URL a site exposes via its public sitemap chain. Reads robots.txt, follows Sitemap declarations, recursively descends sitemap-index files, extracts URLs with lastmod, changefreq, priority.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

vøiddo

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Sitemap URL Extractor — All URLs by Domain API through CLI

The Apify CLI is the official tool that allows you to use Sitemap URL Extractor — robots.txt + sitemap.xml Crawl locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl API in Python

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl API in JavaScript

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl OpenAPI definition

Sitemap URL Extractor — robots.txt + sitemap.xml Crawl API

Sitemap to URL Crawler — Extract Sitemap.xml URLs for RAG

logiover/sitemap-to-url-crawler

Extract all URLs from any sitemap.xml recursively. Export sitemap URLs to CSV/JSON for RAG pipelines, SEO audits, and LLM training datasets.

Logiover

Sitemap URL Extractor

wiry_kingdom/sitemap-url-extractor

Extract every URL from any website's sitemap.xml with lastmod, changefreq, priority. Recursively expands sitemap index files, reads robots.txt, handles gzipped sitemaps. SEO audits, content migration, site inventory, competitor research.

Mohieldin Mohamed

Sitemap & URL Extractor — Get Every URL of a Website

dataquarry/sitemap-url-extractor

Get every URL of a website: parses sitemap.xml and sitemap-indexes (discovered via robots.txt or the default location), with a same-site crawl fallback when there's no sitemap. Returns each URL + lastmod. No API key.

Daniel Brenner

Sitemap URL Extractor

seemuapps/sitemap-extractor

Extract every URL from a website's sitemap.xml. Recursively walks nested sitemap indexes and returns loc, lastmod, changefreq, and priority for each page.

Andrew

Sitemap URL Finder

thescrapelab/sitemap-target-url-extractor

Find and export URLs from any website’s robots.txt and sitemaps. Enter a domain or website URL, optionally filter matching URLs by text, and get clean dataset rows with the URL, domain, path, source sitemap, and match details.

Inus Grobler

Sitemap Crawler - XML Sitemap URL Extractor

miccho27/sitemap-crawler

Extract all URLs from XML sitemaps (including sitemap index) and optionally audit each page

Tatsuya Mizuno

Sitemap Finder & URL Extractor · Crawl Any XML Sitemap

corent1robert/sitemap-detector

Find and crawl XML sitemaps from any website. Follows sitemap indexes, handles gzip, and exports every page URL with source file and lastmod into a clean dataset. No config needed.

Corentin Robert

Sitemap & URL Discovery - Find All URLs on Any Site

santamaria-automations/sitemap-url-discovery

Discover every URL on any website by parsing sitemap.xml, robots.txt, and sitemap indexes. Extract URLs with last modified dates, change frequency, and priority. Perfect for SEO audits, content analysis, crawling preparation, and site mapping.

Ale

Sitemap to URL List Extractor

scrapeworks/sitemap-to-urls

Extract every URL from any website's sitemap as clean JSON. Handles sitemap indexes (recursive) and gzipped sitemaps automatically. Includes lastmod, priority, and changefreq.

Nicolas van Arkens

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.