Sitemap URL Auditor & Site Structure Mapper

Extract every URL from any XML sitemap and get back a structured, filtered, analysis-ready inventory of a website — not just a flat list of links.

Point it at a sitemap URL or just a domain name, and the Actor discovers the sitemaps itself, unpacks gzipped files, walks sitemap indexes recursively, removes duplicates, and enriches every URL with the structural and freshness data you need for a technical SEO audit, a content refresh plan, or a crawl seed list.

---

What this Actor does

• Sitemap URL extraction — pulls <loc>, <lastmod>, <changefreq> and <priority> from every <url> entry.
• Sitemap index crawling — follows nested sitemap indexes to any depth, with loop protection.
• Gzip support — reads .xml.gz sitemaps transparently, which most large sites publish.
• robots.txt sitemap discovery — give it example.com and it finds the sitemaps for you.
• Content freshness analytics — how old is each page, and how much of the site is stale?
• Site structure mapping — every URL classified by section, depth and slug.

---

Key features

1. Auto-discovery from a bare domain

You do not need to know where the sitemap lives. Enter example.com and the Actor:

1. reads https://example.com/robots.txt and follows every Sitemap: directive it declares (relative paths included), then
2. falls back to well-known locations — /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap/sitemap.xml, /wp-sitemap.xml, /sitemap.xml.gz.

Gzipped sitemaps are detected from the file's own bytes rather than its Content-Encoding header, so .xml.gz files that servers hand over as raw binary are still parsed correctly.

2. Precision filtering — export only the URLs you need

Large sitemaps are noisy. Four filters narrow the export before it ever reaches your dataset:

Input	Effect
includeUrlPatterns	Keep only URLs matching at least one regular expression, e.g. /blog/
excludeUrlPatterns	Drop URLs matching a regular expression, e.g. /tag/ or \.pdf$
modifiedAfter	Keep only pages changed on or after a date — find what is new
modifiedBefore	Keep only pages changed on or before a date — find what is stale

An invalid regular expression is skipped rather than failing the run, so one typo never costs you the whole job.

3. Structure and freshness enrichment on every row

Each URL is returned already analysed, so you can pivot the dataset without writing any post-processing:

• siteSection, pathDepth, pageSlug — the shape of the site
• daysSinceModified and freshnessBucket (last_7_days, last_30_days, last_90_days, last_year, over_a_year, unknown)
• alternateLocales — hreflang codes declared for the page
• imageCount / videoCount — image and video sitemap extension entries
• isHttps, hasQueryString — quick technical-hygiene flags

4. A summary report row

Set emitSummaryReport and the final dataset item (recordType: "summary") gives you the whole run at a glance: how many sitemaps were read, how many URLs were exported, how many duplicates were removed, how many were filtered out, a per-section count breakdown, a content-freshness breakdown, the oldest and newest lastmod dates, and every error encountered.

---

Input

Field	Type	Default	Description
sitemapSource (required)	string	—	Sitemap URL, .xml.gz URL, or a bare domain
includeUrlPatterns	array of strings	—	Regex allowlist
excludeUrlPatterns	array of strings	—	Regex blocklist
modifiedAfter	string	—	YYYY-MM-DD lower bound on <lastmod>
modifiedBefore	string	—	YYYY-MM-DD upper bound on <lastmod>
maxUrls	integer	0	Stop after N URLs; 0 means no limit
maxSitemaps	integer	200	Cap on sitemap files fetched
emitSummaryReport	boolean	true	Append the summary row
requestTimeoutSec	integer	45	Per-request timeout
proxyConfig	object	disabled	Optional Apify Proxy configuration

Note on proxies: sitemaps are static XML files and are rarely blocked, so no proxy is used by default. Turn proxyConfig on only if a target restricts direct access — this keeps proxy traffic for the jobs that actually need it.

Example input

Find every blog post on a site that has not been touched since 2023:

{
  "sitemapSource": "example.com",
  "includeUrlPatterns": ["/blog/"],
  "excludeUrlPatterns": ["/tag/", "/author/"],
  "modifiedBefore": "2023-12-31",
  "emitSummaryReport": true
}

---

Output

Page record

{
  "recordType": "page",
  "pageUrl": "https://example.com/blog/post-one",
  "sourceSitemapUrl": "https://example.com/sitemap-posts.xml",
  "lastModified": "2024-05-01T00:00:00+00:00",
  "lastModifiedRaw": "2024-05-01",
  "daysSinceModified": 9,
  "freshnessBucket": "last_30_days",
  "updateFrequency": "weekly",
  "crawlPriority": 0.8,
  "alternateLocales": ["de"],
  "imageCount": 1,
  "videoCount": 0,
  "siteSection": "blog",
  "pathDepth": 2,
  "pageSlug": "post-one",
  "isHttps": true,
  "hasQueryString": false,
  "discoveredAt": "2026-08-11T09:14:02.117000+00:00"
}

Summary record

{
  "recordType": "summary",
  "sitemapSource": "example.com",
  "sitemapsProcessed": 12,
  "sitemapUrls": ["https://example.com/sitemap-posts.xml"],
  "totalEntriesSeen": 8451,
  "uniqueUrlsExported": 8102,
  "duplicatesSkipped": 349,
  "filteredOutCount": 0,
  "sectionBreakdown": { "blog": 5100, "products": 2800, "(root)": 202 },
  "freshnessBreakdown": { "last_30_days": 410, "over_a_year": 3900 },
  "oldestLastModified": "2016-03-02T00:00:00+00:00",
  "newestLastModified": "2026-08-10T00:00:00+00:00",
  "errors": [],
  "finishedAt": "2026-08-11T09:15:40.882000+00:00"
}

Filter the dataset on recordType to separate page rows from the summary row.

---

Common use cases

• Technical SEO audits — measure index bloat, find pages buried at excessive pathDepth, and spot URLs still on HTTP.
• Content refresh planning — combine modifiedBefore with freshnessBucket to build a prioritised list of stale articles.
• Crawl seed lists — export a filtered, deduplicated URL set to feed into a scraper or a site-wide crawler.
• Competitor site mapping — see how a competitor's catalogue or blog is structured and sized via sectionBreakdown.
• Migration QA — snapshot a sitemap before and after a replatform and diff the URL inventory.
• International SEO — audit hreflang coverage using alternateLocales.

---

Notes and limits

• Only URLs declared in the sitemap are returned; the Actor does not crawl pages or follow on-page links.
• <lastmod> values are self-reported by the target site and are not always accurate.
• When a date window is set, URLs that declare no <lastmod> are excluded, because their age cannot be established.
• Malformed XML is parsed in recovery mode, so partially broken sitemaps still yield the entries that are readable.
• maxSitemaps (default 200) bounds very large sitemap trees; if it is reached, the run stops early and records the fact in the summary's errors.

---

Local development

pip install -r requirements.txt
python -m tests.test_core      # 53 checks: parsing, gzip, filters, recursion, loop guard, live run
apify run