Sitemap to URL List Extractor
Pricing
from $1.00 / 1,000 results
Extract every URL from any website's sitemap as clean JSON. Handles sitemap indexes (recursive) and gzipped sitemaps automatically. Includes lastmod, priority, and changefreq.
Pricing
from $1.00 / 1,000 results
Rating
0.0
(0)
Developer
Nicolas van Arkens
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
3 days ago
Last modified
Categories
Share
.git.venv__pycache__*.pycstoragetest_*.py1apify < 3.02pydantic >= 2.8, < 2.123browserforge == 1.2.34httpx{ "actorSpecification": 1, "name": "sitemap-to-urls", "title": "Sitemap to URL List Extractor", "description": "Extract every URL from any website's sitemap as clean JSON. Handles sitemap indexes (recursive) and gzipped sitemaps automatically. Includes lastmod, priority, and changefreq.", "version": "0.1", "buildTag": "latest", "input": "./input_schema.json", "dockerfile": "./Dockerfile", "storages": { "dataset": "./dataset_schema.json" }}{ "actorSpecification": 1, "views": { "overview": { "title": "Overview", "transformation": { "fields": ["url","lastModified","changeFrequency","priority","sourceSitemap"] }, "display": { "component": "table", "properties": { "url": { "label": "URL", "format": "link" }, "lastModified": { "label": "Last modified", "format": "text" }, "changeFrequency": { "label": "Change freq", "format": "text" }, "priority": { "label": "Priority", "format": "text" }, "sourceSitemap": { "label": "Source sitemap", "format": "text" } } } } }}FROM apify/actor-python:3.12COPY requirements.txt ./RUN pip install --no-cache-dir -r requirements.txtCOPY . ./CMD ["python", "-m", "src"]{ "title": "Sitemap to URL List Extractor", "type": "object", "schemaVersion": 1, "properties": { "sitemapUrls": { "title": "Sitemap URLs", "type": "array", "description": "One or more sitemap URLs to process. Usually sitename.com/sitemap.xml or sitename.com/sitemap_index.xml. Gzipped (.xml.gz) sitemaps are handled automatically.", "editor": "stringList", "prefill": ["https://www.apify.com/sitemap.xml"] }, "followIndexes": { "title": "Follow sitemap indexes", "type": "boolean", "description": "If a URL points to a sitemap index (a list of nested sitemaps), automatically follow and process each of them. Turn off to only list the nested sitemap URLs.", "default": true }, "maxUrls": { "title": "Maximum URLs", "type": "integer", "description": "Cap on the total number of URLs to return across all sitemaps.", "default": 50000, "minimum": 1, "maximum": 1000000 }, "maxNestedSitemaps": { "title": "Maximum nested sitemaps", "type": "integer", "description": "Cap on how many sitemap files to fetch when following indexes (a guardrail for very large sites with many shards).", "default": 50, "minimum": 1, "maximum": 500 } }, "required": []}1# Sitemap to URL List Extractor 🗺️2
3Extract **every URL from any website's sitemap** as clean structured JSON, with last-modified dates, priority, and change frequency. Handles **sitemap indexes** (recursing into nested sitemaps) and **gzipped sitemaps** (`.xml.gz`) automatically — the parts most lazy sitemap scrapers get wrong.4
5Perfect for SEO audits, content inventories, building crawl lists, monitoring site changes, and feeding URL lists into other tools.6
7## Why use it8
9- 🔁 **Recursive sitemap indexes** — point it at `sitemap_index.xml` and it'll follow every nested sitemap automatically10- 📦 **Gzipped sitemaps handled** — many large sites ship `.xml.gz`, decompressed transparently11- 🗓️ **Full metadata** — `lastmod`, `priority`, `changefreq`, plus image URLs from the Google image sitemap extension12- 🛡️ **Guardrails** — configurable caps on total URLs and nested sitemaps so a giant site can't run away13- 🌐 **Works on any website** — news sites, e-commerce stores, blogs, documentation, public web apps14
15## Use cases16
17- **SEO audits** — list every indexed URL on a site, sort by lastmod, find stale content18- **Crawl seed lists** — generate URL lists for downstream scrapers or archival tools19- **Content inventories** — see what a competitor or partner site is actually publishing20- **Change monitoring** — schedule it and detect newly added pages21- **Site migrations** — get the full URL set before redirect mapping22
23## Input24
25| Field | Description |26|-------|-------------|27| **Sitemap URLs** | One or more sitemap URLs (`sitemap.xml`, `sitemap_index.xml`, or `.xml.gz`). |28| **Follow indexes** | If a URL points to an index, recurse and process each nested sitemap. |29| **Maximum URLs** | Total URL cap across all sitemaps. |30| **Maximum nested sitemaps** | Cap on number of sitemap files fetched when following indexes. |3132## Output33
34```json35{36 "url": "https://example.com/page1",37 "lastModified": "2025-05-10",38 "changeFrequency": "weekly",39 "priority": "0.8",40 "images": ["https://example.com/img1.jpg"],41 "sourceSitemap": "https://example.com/sitemap-products.xml",42 "sourceRoot": "https://example.com/sitemap_index.xml"43}44```45
46Export to JSON, CSV, or Excel, or pull via the Apify API.47
48## Notes49
50- Supports the standard [sitemaps.org protocol](https://www.sitemaps.org/protocol.html), sitemap index files, and the Google image-sitemap extension.51- Always respects each site's `robots.txt` policy on access — please use responsibly.52- Independent tool; sitemaps remain the property of their publishers.1"""Sitemap to URL List — Apify Actor.2
3Give it a sitemap URL (or several) and it returns a clean list of every URL4inside, with lastmod / priority / changefreq metadata. Handles the three5flavors of real-world sitemaps:6
7 1. Regular sitemap (<urlset> with <url> entries)8 2. Sitemap index (<sitemapindex> -> list of nested sitemap URLs, recursed)9 3. Gzipped sitemaps (.xml.gz, automatically decompressed)10
11Also extracts the image:loc URLs when present (Google image sitemap extension).12"""13
14from __future__ import annotations15
16import asyncio17import gzip18import io19from xml.etree import ElementTree as ET20
21import httpx22from apify import Actor23
24# Sitemap protocol namespaces. We strip them when reading by local-name lookup.25SITEMAP_NS = "http://www.sitemaps.org/schemas/sitemap/0.9"26IMAGE_NS = "http://www.google.com/schemas/sitemap-image/1.1"27
28
29def _localname(tag: str) -> str:30 return tag.rsplit("}", 1)[-1]31
32
33def _text(el, name: str) -> str | None:34 for child in el:35 if _localname(child.tag) == name and child.text:36 t = child.text.strip()37 if t:38 return t39 return None40
41
42def parse_sitemap(xml_text: str) -> tuple[str, list[dict]]:43 """Parse a sitemap XML. Returns ('urlset' | 'sitemapindex', items)."""44 root = ET.fromstring(xml_text)45 root_name = _localname(root.tag)46
47 if root_name == "urlset":48 urls = []49 for url_el in root:50 if _localname(url_el.tag) != "url":51 continue52 loc = _text(url_el, "loc")53 if not loc:54 continue55 # Collect image:loc entries if present.56 images = []57 for child in url_el:58 if _localname(child.tag) == "image":59 for grand in child:60 if _localname(grand.tag) == "loc" and grand.text:61 images.append(grand.text.strip())62 urls.append({63 "url": loc,64 "lastModified": _text(url_el, "lastmod"),65 "changeFrequency": _text(url_el, "changefreq"),66 "priority": _text(url_el, "priority"),67 "images": images,68 })69 return "urlset", urls70
71 if root_name == "sitemapindex":72 sub = []73 for sm_el in root:74 if _localname(sm_el.tag) != "sitemap":75 continue76 loc = _text(sm_el, "loc")77 if loc:78 sub.append({"sitemap": loc, "lastModified": _text(sm_el, "lastmod")})79 return "sitemapindex", sub80
81 # Unknown root — return as empty rather than crashing.82 return root_name, []83
84
85async def fetch_sitemap(client: httpx.AsyncClient, url: str, log) -> str | None:86 """Fetch a sitemap. Auto-decompresses .gz responses."""87 for attempt in range(1, 4):88 try:89 resp = await client.get(url)90 resp.raise_for_status()91 content = resp.content92 # Detect gzip: by URL suffix OR by magic bytes.93 is_gz = url.endswith(".gz") or content[:2] == b"\x1f\x8b"94 if is_gz:95 try:96 content = gzip.decompress(content)97 except (OSError, gzip.BadGzipFile) as exc:98 log.warning(f"Failed to gunzip {url}: {exc}")99 return None100 return content.decode("utf-8", errors="replace")101 except httpx.HTTPError as exc:102 log.warning(f"Fetch attempt {attempt} failed for {url}: {exc}")103 if attempt < 3:104 await asyncio.sleep(attempt * 2)105 return None106
107
108async def main() -> None:109 async with Actor:110 actor_input = await Actor.get_input() or {}111 seed_urls = [u.strip() for u in (actor_input.get("sitemapUrls", []) or []) if u and u.strip()]112 follow_indexes = bool(actor_input.get("followIndexes", True))113 max_urls = int(actor_input.get("maxUrls", 50000))114 max_nested_sitemaps = int(actor_input.get("maxNestedSitemaps", 50))115
116 if not seed_urls:117 Actor.log.warning("No sitemap URLs provided.")118 await Actor.push_data([])119 return120
121 Actor.log.info(f"Processing {len(seed_urls)} sitemap(s). Follow indexes: {follow_indexes}.")122
123 # BFS queue of sitemaps to process.124 to_process: list[tuple[str, str]] = [(u, u) for u in seed_urls] # (url, source_root)125 processed_sitemaps = 0126 pushed_urls = 0127 seen_sitemaps: set[str] = set()128
129 async with httpx.AsyncClient(130 timeout=40.0, follow_redirects=True,131 headers={"User-Agent": "scrapeworks-sitemap-to-urls/0.1", "Accept": "application/xml, text/xml, */*"},132 ) as client:133 while to_process and pushed_urls < max_urls and processed_sitemaps < max_nested_sitemaps:134 sm_url, source_root = to_process.pop(0)135 if sm_url in seen_sitemaps:136 continue137 seen_sitemaps.add(sm_url)138
139 xml = await fetch_sitemap(client, sm_url, Actor.log)140 if xml is None:141 Actor.log.warning(f" Skipping unreachable {sm_url}")142 continue143
144 try:145 kind, items = parse_sitemap(xml)146 except ET.ParseError as exc:147 Actor.log.warning(f" Could not parse {sm_url}: {exc}")148 continue149
150 processed_sitemaps += 1151 Actor.log.info(f" {sm_url}: {kind} with {len(items)} entries")152
153 if kind == "urlset":154 remaining = max_urls - pushed_urls155 batch = items[:remaining]156 for rec in batch:157 rec["sourceSitemap"] = sm_url158 rec["sourceRoot"] = source_root159 if batch:160 await Actor.push_data(batch)161 pushed_urls += len(batch)162 elif kind == "sitemapindex":163 if follow_indexes:164 for entry in items:165 sub_url = entry.get("sitemap")166 if sub_url:167 to_process.append((sub_url, source_root))168
169 Actor.log.info(f"Done. Returned {pushed_urls} URLs from {processed_sitemaps} sitemap(s).")1import asyncio2from .main import main3
4asyncio.run(main())