Sitemap Extractor & SEO Index Audit

Extract every URL from a website's XML sitemaps and find the pages Google can't index. Give it a domain or a sitemap URL. The actor reads robots.txt to find the sitemaps, follows nested sitemap indexes (including .xml.gz), and lists every URL with lastmod, changefreq, priority, hreflang alternates and image/video counts. It can also fetch each URL and flag 404s, redirects, noindex, canonical mismatches, robots.txt blocks, stale lastmod dates and duplicates, which are the usual reasons for "Discovered / Crawled – currently not indexed" in Google Search Console.

What you get

• A full sitemap URL list (CSV, JSON or Excel export) across all sitemap files, de-duplicated.
• An indexability audit per URL: HTTP status, redirect chain, final URL, meta robots, X-Robots-Tag, canonical, page title and response time.
• Issue flags for every URL (see the table below).
• A site health score plus issue counts and sitemap-file errors in the SUMMARY record.
• An ISSUES CSV of only the problem URLs, ready to hand to a developer or import into a spreadsheet.

Use cases

• Technical SEO audits: find sitemap URLs that return 404, redirect, are noindexed or point their canonical elsewhere. Google treats these as sitemap errors.
• Migrations and redesigns: check that the new sitemap only lists live, indexable URLs.
• Competitor research: extract a competitor's full URL inventory, or find their newest pages by lastmod.
• Content inventory: get a full page list with last-modified dates for content pruning or refresh planning.
• Monitoring: schedule it weekly and alert when new 404s or noindex pages appear in the sitemap.
• Feeding other tools: pass the URL list to crawlers, scrapers, RAG/LLM pipelines or the Wayback Machine URL Extractor.

Input

Field	Description	Default
startUrls	Domains (example.com), homepages, or direct sitemap URLs	required
auditPages	Fetch and audit each URL. Turn it off to only extract URLs	true
maxUrlsPerSite	Maximum unique sitemap URLs per site	100
maxAuditsPerSite	Maximum URLs to fetch and audit per site	50
outputOnlyIssues	Save only URLs that have issues (you pay only for saved URLs)	false
staleLastmodDays	Flag lastmod dates older than N days (0 = off)	365
includeUrlPatterns / excludeUrlPatterns	Regex filters, e.g. /blog/	–
maxSitemapsPerSite	Maximum sitemap files read (including nested indexes)	200
concurrency	Parallel requests per site (keep it polite)	5

Example:

{
  "startUrls": ["example.com", "https://shop.example.com/sitemap_index.xml"],
  "auditPages": true,
  "maxUrlsPerSite": 5000,
  "maxAuditsPerSite": 1000,
  "outputOnlyIssues": false
}

Output

Each dataset item is one sitemap URL:

{
  "site": "wordpress.org",
  "url": "https://wordpress.org/enterprise/media/",
  "sitemapUrl": "https://wordpress.org/sitemap-1.xml",
  "lastmod": "2022-03-28T20:20:42Z",
  "priority": null,
  "hreflangAlternates": null,
  "statusCode": 200,
  "finalUrl": "https://wordpress.org/enterprise/media/",
  "redirectCount": 0,
  "metaRobots": "max-image-preview:large, noindex,follow",
  "canonical": "https://wordpress.org/enterprise/media/",
  "title": "Media | WordPress.org",
  "responseTimeMs": 286,
  "issues": ["stale-lastmod", "noindex"],
  "issueCount": 2
}

Key-value store records:

• SUMMARY: per-site health score, issue counts, sitemaps read, sitemap errors and site-level problems (missing robots.txt, no sitemap, and so on).
• ISSUES: a CSV of the problem URLs.

Issues detected

Issue	Meaning
http-404, http-410, http-5xx…	The sitemap lists a URL that returns an error
redirected	The sitemap URL redirects. List the final URL instead
noindex	meta robots or X-Robots-Tag says noindex, which contradicts the sitemap
canonical-mismatch	The canonical tag points to a different URL
blocked-by-robots-txt	robots.txt disallows the URL for Googlebot
unreachable	Timeout or connection error
non-html-content	Returns a non-HTML content type unexpectedly
stale-lastmod / future-lastmod / invalid-lastmod	Problems with the lastmod date
duplicate-in-sitemaps	The same URL is listed more than once
http-not-https, off-domain-url, invalid-url	URL hygiene problems

Health score is the percentage of audited URLs that are indexable: no error status, redirect, noindex, canonical mismatch or robots block.

Pricing (pay per event)

Event	Price
Actor start (per run)	$0.005
Sitemap URL saved	$0.0003 (= $0.30 per 1,000 URLs)
Page audited	$0.0012 (= $1.20 per 1,000 audited pages)

A full audit of 1,000 pages costs about $1.50. Extracting 10,000 URLs without auditing costs about $3. Set a maximum cost per run, and the actor stops cleanly at your limit while keeping the results collected so far.

FAQ

Is this legal? The actor reads public sitemaps and robots.txt (which exist so crawlers can read them) and requests public pages at a polite rate with a clear user agent. It doesn't log in or bypass anything.

Can it find sitemaps automatically? Yes. It reads Sitemap: lines in robots.txt, then falls back to /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and other common paths.

Does it support gzip, sitemap indexes, hreflang, image and news sitemaps? Yes.

Does it execute JavaScript? No. It reads the HTML head as served, which is what matters for meta robots and canonical in most cases.

Related actors by pandabear

• App Store Scraper - Search, Top Charts & Keyword Ranks: iOS apps by keyword, top chart, category or developer, with full details and ASO keyword rank tracking.
• Wayback Machine URL Extractor & Lost Page Finder: finds old URLs that dropped out of your sitemap and now 404, and suggests 301 redirects.
• Bulk WHOIS & Domain Availability Checker: bulk RDAP/WHOIS lookup with registrar, creation and expiry dates, domain age, availability and DNS.
• App Store Reviews Scraper - All Countries + Analytics: Apple App Store reviews from 76 countries, with app details, rating trends by version and complaint keywords.
• Website Contact Scraper - Emails, Phones, Socials & Tech Stack: emails, phones, social links, company info and tech stack from any list of websites.
• Trending Topics Scraper - Google Trends, HN, Wikipedia & More: trending topics from Google Trends, Hacker News, Wikipedia, GitHub, Bluesky, Baidu and more, with heat scores and cross-platform clusters.

---

中文说明

这个工具做什么

输入一个域名或 sitemap 地址，工具会自动从 robots.txt 和常见路径找到网站的全部 XML sitemap，递归读取 sitemap 索引（支持 .xml.gz），提取所有 URL，并附带 lastmod、changefreq、priority、hreflang 多语言链接和图片/视频数量。开启审计后，工具会逐个访问这些 URL，检查谷歌无法收录的原因：404/5xx 错误、重定向、noindex、canonical 指向别处、被 robots.txt 屏蔽、lastmod 过旧、重复 URL 等。这些正是 Google Search Console 里"已发现/已抓取 – 尚未编入索引"的常见原因。

适用场景

• 技术 SEO 审计：找出 sitemap 中的死链、重定向、noindex 和 canonical 冲突
• 网站改版或迁移后检查新 sitemap 的质量
• 竞品分析：导出竞争对手的完整 URL 列表，按 lastmod 找出他们的最新页面
• 内容盘点：获取全站页面和最后修改时间
• 定期监控：每周运行一次，sitemap 出现新的 404 或 noindex 时提醒你
• 为爬虫、抓取工具、RAG/大模型流程提供 URL 列表

输入参数

• startUrls：域名、首页或 sitemap 地址（必填）
• auditPages：是否逐个访问并审计 URL（默认开启；关闭后只提取 URL，更便宜）
• maxUrlsPerSite：每个网站最多提取多少个 URL（默认 100）
• maxAuditsPerSite：每个网站最多审计多少个 URL（默认 50）
• outputOnlyIssues：只保存有问题的 URL（只对保存的 URL 收费）
• staleLastmodDays：lastmod 早于多少天就标记为过旧（默认 365，填 0 关闭）
• includeUrlPatterns / excludeUrlPatterns：正则过滤
• maxSitemapsPerSite、concurrency：高级设置

输出

• 数据集：每条记录对应一个 URL，包含状态码、最终 URL、重定向链、meta robots、X-Robots-Tag、canonical、标题、响应时间，以及问题列表 issues
• SUMMARY：每个网站的健康分（可收录 URL 所占比例）、各类问题数量、sitemap 文件错误
• ISSUES：只含问题 URL 的 CSV，可以直接发给开发人员

价格（按事件计费）

• 每次运行启动：$0.005
• 每保存一个 URL：$0.0003（每 1000 个 $0.30）
• 每审计一个页面：$0.0012（每 1000 个 $1.20）

完整审计 1000 个页面约 $1.50；只提取 1 万个 URL 约 $3。可以设置单次运行的费用上限，达到上限后工具会正常停止并保留已有结果。

常见问题

• 合法吗？ 只读取公开的 sitemap、robots.txt 和公开网页，请求频率克制，User-Agent 清晰标识，不登录、不绕过任何限制。
• 能自动找到 sitemap 吗？ 能。优先读取 robots.txt 里的 Sitemap: 行，找不到时再尝试 /sitemap.xml、/sitemap_index.xml、/wp-sitemap.xml 等常见路径。
• 支持 gzip、sitemap 索引、hreflang、图片和新闻 sitemap 吗？ 都支持。
• 会执行 JavaScript 吗？ 不会。工具读取服务器返回的原始 HTML，大多数情况下这已足够判断 meta robots 和 canonical。

pandabear 的其他工具

• App Store Scraper - Search, Top Charts & Keyword Ranks：按关键词、排行榜、分类或开发者抓取苹果应用数据，并跟踪 ASO 关键词排名
• Wayback Machine URL Extractor & Lost Page Finder：找回从 sitemap 中消失、现在返回 404 的旧网址，并给出 301 重定向建议
• Bulk WHOIS & Domain Availability Checker：批量查询 RDAP/WHOIS，包括注册商、注册和到期日期、域名年龄、是否可注册以及 DNS
• App Store Reviews Scraper：抓取苹果 App Store 76 个国家的应用评论，附带应用详情、按版本的评分趋势和差评关键词
• Website Contact Scraper：从网站列表中提取邮箱、电话、社交媒体链接、公司信息和技术栈
• Trending Topics Scraper：汇总 Google Trends、Hacker News、维基百科、GitHub、Bluesky、百度热搜等平台的热门话题，附热度分和跨平台聚类