Pricing

from $2.49 / 1,000 results

Try for free

Go to Apify Store

Sitemap URL Scraper

Try for free

Sitemap Scraper SD - Sitemap Scraper is an SEO crawling tool that extracts every URL from a website's sitemaps (robots.txt, sitemap.xml, indexes, gzip, RSS) with lastmod, changefreq, priority, hreflang and news data.

Pricing

from $2.49 / 1,000 results

Rating

0.0

(0)

Developer

Neuro Scraper

Actor stats

Bookmarked

Total users

Monthly active users

15 days ago

Last modified

Sitemap Scraper — every URL in a website's sitemaps

Sitemap Scraper finds a website's sitemaps — from robots.txt Sitemap: lines, or /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt — follows sitemap indexes to the end, unpacks gzip files and exports every URL with lastmod, changefreq and priority, plus hreflang alternates, image counts and Google News titles and dates. Give it a website or a sitemap URL.

Filter by URL globs or by last-modified date (for incremental runs), and optionally check each URL's HTTP status.

Input

Field	Type	Default	Meaning
`startUrls`	URL list	crawlee.dev (prefill)	Websites or sitemap / sitemap-index URLs (.xml, .xml.gz, .txt)
`maxItems`	integer	1000	URLs saved
`maxSitemaps`	integer	200	Sitemap files read (indexes included)
`includeUrlGlobs` / `excludeUrlGlobs`	glob lists	–	e.g. `https://site.com/blog/**`
`lastmodAfter`	date	–	Only URLs with `<lastmod>` on/after YYYY-MM-DD
`checkUrlStatus`	boolean	false	Request each URL and add `statusCode`, `finalUrl`, `statusNote`
`tryWellKnownPaths`	boolean	true	Probe common sitemap paths when robots.txt lists none
`respectRobotsTxt`	boolean	true	Skip sitemap files / URL checks robots.txt disallows
`requestDelayMs`	integer	1000	Pause between requests to one host
`proxyConfiguration`	proxy	Apify datacenter	RESIDENTIAL for sites that block datacenter IPs

Output (one row per URL)

Field	Meaning
`url`	Page URL from the sitemap
`lastmod`, `changefreq`, `priority`	As published in the sitemap (null when absent)
`imageCount`, `alternates`	Image-sitemap entries; hreflang alternates `[{hreflang, href}]`
`newsTitle`, `newsPublishedAt`	Google News sitemap data
`sitemapUrl`, `sitemapType`, `inputUrl`	Which file it came from (`urlset`, `rss`, `atom`, `text`) and which input
`statusCode`, `finalUrl`, `statusNote`	Only with `checkUrlStatus`
`scrapedAt`	ISO time

A per-site summary (sitemaps found and how, files read, URLs saved, problems) is stored in the key-value store record SUMMARY.

Measured (2026-10-03, see TEST_REPORT.md)

crawlee.dev: 1,000 URLs in 8 s; target.com gzip store index: 2,026 URLs in 10 s; wordpress.org: 30 files (17 indexes) -> 2,576 unique URLs in 36 s.

Limits

Only URLs the site lists in its sitemaps (python.org publishes none: 0 rows, said plainly).
checkUrlStatus makes one request per URL at the per-host delay — about 1 URL per second per site.
Sitemaps behind anti-bot pages cannot be read; try a RESIDENTIAL proxy.

Example

{"startUrls": [{"url": "https://www.theguardian.com"}], "lastmodAfter": "2026-10-01", "maxItems": 500}

Sitemap Detector

coder_zoro/sitemap-detector

Find sitemap URLs fast with our free Sitemap Finder tool. Instantly detect sitemaps from any website for SEO audits, indexing checks, and crawl planning. Improve visibility, site structure insights, and search engine performance in just seconds

Zoro

182

5.0

(3)

Facebook Reels Scraper

neuro-scraper/facebook-reels-scraper

Facebook Reels Scraper SD - Facebook Reels Scraper is a social media data tool that extracts reels of public Facebook pages with captions, plays, durations, audio, owners and MP4 links, plus optional comments and transcripts - Facebook reels extractor.

Neuro Scraper

5.0

(1)

XML Sitemap Finder & Extractor API

andok/find-sitemap-from-url

Find and extract all XML sitemaps for any domain. Automatically parses robots.txt, scans HTML tags, and recursively follows indexes. Perfect for SEO & web scraping.

Andok

Sitemap URL Harvester

mahogany_songbird/sitemap-url-harvester

Collect URLs from XML sitemaps for SEO and crawling.

Britton Furness

Bilibili Scraper

neuro-scraper/bilibili-scraper

Bilibili Scraper SD - Bilibili Scraper is a social media data tool that extracts public Bilibili (哔哩哔哩) videos from search, video details, full comment threads, creator videos and profiles, popular and zone rankings and hot searches, with views, likes, coins, danmaku and tags. No login.

Neuro Scraper

AutoTrader Cars Scraper

neuro-scraper/autotrader-cars-scraper

AutoTrader Cars Scraper SD - AutoTrader Cars Scraper is a logged-out tool that extracts car listings from AutoTrader UK, USA (VIN, KBB price), Canada and South Africa: price, price rating, mileage, specs, history, images and dealer details.

Neuro Scraper

Robots.txt & Sitemap Analyzer

automation-lab/robots-sitemap-analyzer

This actor fetches and parses robots.txt and sitemap.xml files for any list of websites. It extracts crawl directives (user-agent rules, allowed/disallowed paths, crawl-delay), discovers sitemap URLs, and counts the number of pages listed in each sitemap. Use it for SEO audits, competitive...

Automation Lab

Sitemap URL Extractor

gifted_wagon/sitemap-url-inventory

Traverse public sitemap indexes into a deduplicated URL inventory with source sitemap, lastmod, changefreq, priority, media counts, and partial-failure evidence.

Michael Olmos

Sitemap URL Extractor

sootesting/sitemap-url-extractor

Extract URLs from XML sitemaps in bulk. Supports sitemap indexes, auto-discovers nested sitemaps, and outputs structured JSON. Pay-per-event: per 1,000 URLs extracted.

soot

Sitemap URL Extractor & New Page Monitor

henri_dewitte/sitemap-url-extractor

Extract every URL from a website's sitemaps. Auto-finds sitemaps via robots.txt, opens indexes and .gz files, adds lastmod, hreflang, images and news data. Track new pages daily. $1 per 1,000 URLs.