Pricing

$1.00 / 1,000 urls

Try for free

Go to Apify Store

Sitemap URL Extractor: all URLs, lastmod, indexes, gzip

Try for free

Give it a website or sitemap URL and get every URL in its sitemaps: lastmod, changefreq, priority, image/video/hreflang counts. Finds sitemaps via robots.txt, follows sitemap indexes and .gz files, dedupes, and filters by regex or modified date. Reads only public sitemap files; no page scraping.

Pricing

$1.00 / 1,000 urls

Rating

0.0

(0)

Developer

Northpine Studio

Actor stats

Bookmarked

Total users

Monthly active users

17 days ago

Last modified

Sitemap URL Extractor

Give it a website or a sitemap URL and get every URL listed in its sitemaps, one dataset row each.

What it does

Finds sitemaps from robots.txt (Sitemap: lines), falling back to /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap.xml.gz.
Follows sitemap index files (up to 5 levels), reads .xml, .xml.gz and plain-text sitemaps.
Output fields: url, lastmod, changefreq, priority, images, videos, alternates (hreflang links), sitemap (source file), site.
Dedupes URLs across files, filters by include/exclude regex and by lastmodAfter.
Key-value store SUMMARY: sitemaps fetched, URLs found/emitted, duplicates, per-file errors.

Input

Field	Meaning
`startUrls`	Websites or direct sitemap URLs
`maxUrls`	Stop after N URLs (default 10000)
`maxSitemaps`	Cap on sitemap files fetched (default 200)
`include` / `exclude`	Regexes on the URL
`lastmodAfter`	`YYYY-MM-DD`; keep only newer URLs
`delayMs`	Delay between fetches (default 300)

Example: { "startUrls": ["https://example.com"], "include": ["/blog/"], "lastmodAfter": "2026-01-01" }

Terms and scope

Reads only the public sitemap protocol files (sitemaps.org) that sites publish for crawlers. It does not fetch or scrape the pages themselves. Requests are sequential and rate-limited with a declared user agent. You are responsible for how you use the URLs, including the target site's terms.

Develop

npm install && npm test (offline unit tests), apify push to deploy.

Sitemap Extractor

apify/sitemap-extractor

This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.

Apify

403

3.2

(8)

Sitemap URL Extractor

onescales/sitemap-url-extractor

Provide a website link to a sitemap.xml and the app will extract and list all URLs in the sitemap as well as additional data in the sitemap (i.e. https://onescales.com/sitemap.xml).

One Scales

652

5.0

(3)

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

Andok

Wayback Machine URL Extractor - Archived URLs

logiover/wayback-machine-url-extractor

Extract every archived URL of any domain from the Internet Archive's Wayback Machine (CDX API). Recover lost or old pages, build redirect maps and run OSINT, with date and status filters. No API key, export to CSV or JSON.

Logiover

Website URL Crawler & Link Extractor

maged120/get-urls-pro

Crawl any website and extract every URL with its depth, parent page and anchor text. Build a sitemap or audit internal links. Works on static and JavaScript sites.

Maged

222

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Sitemap URL Harvester

mahogany_songbird/sitemap-url-harvester

Collect URLs from XML sitemaps for SEO and crawling.

Britton Furness

Sitemap URL Extractor - List All URLs in a Sitemap

dltik/sitemap-url-extractor

Extract every URL from any XML sitemap, with lastmod, changefreq and priority. Resolves sitemap indexes recursively. Pass a sitemap.xml or just a site root to auto-discover its sitemaps. Pure HTTP, no browser — fast and cheap.

Walid

Sitemap URL Extractor

gifted_wagon/sitemap-url-inventory

Traverse public sitemap indexes into a deduplicated URL inventory with source sitemap, lastmod, changefreq, priority, media counts, and partial-failure evidence.