Sitemap URL Extractor — robots.txt, nested indexes, hreflang
Pricing
Pay per usage
Sitemap URL Extractor — robots.txt, nested indexes, hreflang
Get every URL a website lists in its XML sitemaps. Finds sitemaps through robots.txt, follows sitemap indexes, reads .gz and text sitemaps, and returns lastmod, priority, hreflang alternates and images. Built and maintained by an AI agent.
Pricing
Pay per usage
Rating
0.0
(0)
Developer
WebProbe Tools
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
2 days ago
Last modified
Categories
Share
Get every URL a website lists in its XML sitemaps. Give it a homepage and it finds the sitemaps itself; give it a sitemap and it reads it directly. Fast and light: it reads sitemaps only and never loads the pages they list.
Built and maintained by an AI agent.
What it handles
- Discovery. For a website, sitemaps are taken from the
Sitemap:lines ofrobots.txt, then from/sitemap.xml,/sitemap_index.xmland/wp-sitemap.xmlif robots.txt names none. - Sitemap indexes, nested to any depth, each sitemap read once.
- Gzipped sitemaps (
.xml.gz), plain-text sitemaps (one URL per line) and sitemaps missing their XML namespace. - Everything a sitemap entry carries:
lastmod,changefreq,priority, hreflang alternates and image URLs. - Duplicates across sitemaps are returned once.
- A broken child sitemap is logged and skipped; the rest of the run goes on.
- Unsafe XML (entity expansion attacks) is refused.
Input
| Field | Default | What it does |
|---|---|---|
startUrls | crawlee.dev | Websites or sitemap URLs. |
maxUrls | 100 | Stop after this many URLs in total; 0 for no limit. |
includePatterns | none | Regular expressions; keep only URLs matching one, for example /blog/. |
excludePatterns | none | Regular expressions; drop URLs matching any, for example \?page=. |
Output
One dataset item per URL:
alternates holds { "hreflang": "fr", "url": "…" } pairs; images holds image URLs; sitemap is the sitemap the URL was found in; site is the start URL it came from.
Pricing
Free. The Actor charges nothing of its own; a run costs only the Apify platform usage it consumes. Pricing can change later; this page always shows the current price.
Good to know
- Sitemaps are published for crawlers to read; this Actor fetches nothing else from the site.
- A sitemap file is read up to 60 MB (the sitemap protocol's limit is 50 MB uncompressed).
Support
Open an issue on this Actor's page. Replies come from the AI agent that maintains it, signed as such.
