Pricing

from $0.10 / 1,000 urls

Try for free

Go to Apify Store

Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod

Try for free

Extract every URL from any site's sitemaps: robots.txt discovery, sitemap-index recursion, gzip, plain-text and RSS/Atom sitemaps, lastmod/regex filters. Plain HTTP, no browser, no proxies.

Pricing

from $0.10 / 1,000 urls

Rating

0.0

(0)

Developer

Rowfeed

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

Sitemap URL Extractor: Robots.txt, Index & Gzip API through CLI

The Apify CLI is the official tool that allows you to use Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod locally, providing convenience functions and automatic retries on errors.

Install Apify CLI

Using installation script (macOS/Linux):

$curl -fsSL https://apify.com/install-cli.sh | bash

Using installation script (Windows):

$irm https://apify.com/install-cli.ps1 | iex

Using Homebrew:

$brew install apify-cli

Using npm:

$npm install -g apify-cli

Other API clients include:

Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod API in Python

Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod API in JavaScript

Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod OpenAPI definition

Sitemap URL Extractor – Robots.txt, Index, Gzip & Lastmod API

Sitemap URL Extractor — Nested Indexes, Gzip & lastmod

relevate/sitemap-urls

Turn any website into a clean list of its URLs. Finds the sitemap from robots.txt, follows nested sitemap indexes, unpacks .xml.gz, reads RSS/Atom feeds, and can return only pages changed since a date — the cheap way to re-crawl just what is new.

Relevate

Sitemap URL Extractor & Auditor (index-aware, status checks)

plotbench/sitemap-url-auditor

Extract every URL from XML sitemaps — sitemap-index and gzip aware — with lastmod/changefreq/priority, plus optional robots-respecting HTTP status checks for broken pages. No API key, no code — export CSV/JSON/Excel or pipe to Sheets, Zapier, Make, n8n.

Plotbench Studio

Sitemap URL Extractor

leorochasantos/sitemap-extractor

Extract every URL from any XML sitemap — support sitemap indexes, gzip, robots.txt discovery, and metadata extraction. Returns clean structured output: one URL per row.

Leonardo Santos

XML Sitemap URL Extractor & Checker

scrapesignal_labs/sitemap-url-extractor

Extract URLs from XML sitemaps, sitemap indexes, robots.txt, and gzip files. Filter pages and optionally check HTTP status codes for SEO audits.

ScrapeSignal Labs

Sitemap URL Extractor — All Page URLs from a Website

power_on/sitemap-url-list

Get every page URL from a website's sitemaps. Finds sitemaps in robots.txt and common paths, follows sitemap-index files, unpacks gzip, and reports the status of each one. Filter by URL pattern or lastmod date. One row per URL, ready to feed a crawler.

Power On Labs

Sitemap Extractor & URL Change Monitor

automa-flow/sitemap-extractor-monitor

Sitemap extractor API for complete XML, gzip, text and nested sitemap inventories. Export normalized URLs or monitor new, publisher-lastmod-changed and conservatively confirmed removed URLs. Failed or partial trees never manufacture mass removals.

Vadim Bezrukov

Sitemap Finder · URLs from Any XML

corent1robert/sitemap-detector

Find and crawl XML sitemaps from any website. Follows sitemap indexes, handles gzip, and exports every page URL with source file and lastmod into a clean dataset. No config needed.

Corentin Robert

Sitemap Scraper

clintsa/sitemap-scraper

Extract all URLs from any website's sitemap.xml with robots.txt discovery, gzip support, full metadata, and automatic retries

Andy Besos

Sitemap Checker & Broken Link Finder

aiqlabs/sitemap-checker

Find every sitemap a site publishes, expand sitemap indexes, and audit each URL: broken links, redirect chains, URLs blocked by the site's own robots.txt, XML errors, duplicates and invalid lastmod. One row per URL. Respects robots.txt.

Ai-Q Labs

XML Sitemap Scraper & URL Extractor API - SEO Crawler

pink_comic/sitemap-url-extractor

Parse XML website sitemaps and extract URLs for SEO crawls, audits, migrations, and RAG indexing. Auto-discovers sitemap files from robots.txt, follows nested indexes, and exports URL, lastmod, priority, changefreq, and image metadata.