XML Sitemap Checker & Extractor: 404, noindex & Canonical Audit
Pricing
from $0.30 / 1,000 sitemap url saveds
XML Sitemap Checker & Extractor: 404, noindex & Canonical Audit
Extract every URL from a site's XML sitemaps (robots.txt, sitemap index, .xml.gz) and audit the ones that block indexing: 404s, redirects, noindex, canonical mismatch, robots.txt blocks, stale lastmod. Health score + issues CSV. $0.30/1k URLs extracted, $1.20/1k pages audited.
Pricing
from $0.30 / 1,000 sitemap url saveds
Rating
0.0
(0)
Developer
pdbear
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
18 hours ago
Last modified
Categories
Features
Share
XML Sitemap Checker & Extractor: find sitemap URLs Google will not index
Best for: SEO work when pages sit in “Discovered – currently not indexed” or “Crawled – currently not indexed,” or when you need every URL a site declares in its XML sitemaps, as a CSV.
Not for: a full-site crawler that discovers URLs not listed in a sitemap, or a Google Search Console replacement. This Actor reads sitemaps and, optionally, fetches those URLs.
A common cause of “not indexed” is a dirty sitemap: it lists URLs that 404, redirect, carry noindex, point their canonical somewhere else, or are blocked by robots.txt. Those URLs waste crawl budget. Give this Actor a domain or a sitemap URL. It discovers sitemaps from robots.txt, follows nested sitemap indexes (including .xml.gz), extracts every URL, then can fetch each one and flag what is wrong. You get a health score and an ISSUES CSV you can hand to a developer.
Why this one
- Sitemap discovery is included. robots.txt, common sitemap paths, nested indexes, and
.xml.gz. - The audit is about indexability, not only HTTP status. 404s and redirect chains, plus
noindex, canonical mismatch, robots.txt blocks, stalelastmod, and title. - You can extract without auditing. Turn
auditPagesoff and you only pay $0.30 per 1,000 URLs saved. - Honest price. $0.30 per 1,000 URLs extracted, $1.20 per 1,000 pages audited, $0.005 to start. A status-only Store checker may be cheaper per URL; you are paying for the extra indexability fields and the issues export.
Quick start
{"startUrls": ["https://www.djangoproject.com"],"auditPages": true,"maxUrlsPerSite": 100,"maxAuditsPerSite": 50,"outputOnlyIssues": false}
Input
| Field | Description | Default |
|---|---|---|
startUrls | Site home pages or direct sitemap URLs | sample site |
auditPages | Fetch each URL and audit indexability | true |
maxUrlsPerSite | Cap on URLs extracted | — |
maxAuditsPerSite | Cap on pages fetched | — |
outputOnlyIssues | Keep only URLs with a problem | false |
staleLastmodDays | Flag lastmod older than this many days | — |
includeUrlPatterns / excludeUrlPatterns | Optional filters | — |
maxSitemapsPerSite | Cap on sitemap files followed | — |
concurrency | Parallel fetches | — |
Output example
Each dataset row is one sitemap URL, with audit fields when auditPages is on (status, final URL, noindex, canonical, title, issue flags). The run also writes:
ISSUES— only the rows that failed a check, as CSVSUMMARY— counts, health score, sitemap files found
Pricing (pay per event)
| Event | Price |
|---|---|
| Actor start (per run) | $0.005 |
| Sitemap URL saved | $0.0003 ($0.30 per 1,000) |
| Page audited | $0.0012 ($1.20 per 1,000) |
Example: extract 1,000 URLs and audit 200 of them ≈ $0.005 + $0.30 + $0.24 = $0.545. Set a maximum cost per run; the Actor stops cleanly and keeps what it already saved.
Limitations
- Only URLs that appear in sitemaps are considered. Orphan pages are out of scope.
- A live fetch can be blocked or throttled by the target site. Those rows are marked, not invented.
- This does not query Google Search Console, so it cannot prove why Google chose not to index a page.
FAQ
Will this get my pages indexed? No. It tells you which sitemap URLs are unhealthy, so you or a developer can fix them.
Can I only extract URLs? Yes. Set auditPages to false.
Does it follow sitemap indexes and .xml.gz? Yes.
Related actors by pandabear
- Wayback Machine Scraper - Find Lost 404 Pages & Redirect Map — old URLs that dropped out of the sitemap and now 404, plus a 301 map.
- Bulk WHOIS Lookup - Domain Expiry & Availability Checker
- Website Email Extractor & Contact Scraper - Phones & Socials
- App Store Scraper - Search, Top Charts & ASO Keyword Ranks
- App Store Reviews Scraper - Export iOS Reviews, All Countries
- Trending Topics Scraper - Google Trends, HN, Bluesky & More
Keywords
xml sitemap checker, sitemap extractor, sitemap URL extractor, sitemap audit, 404 in sitemap, noindex, canonical mismatch, discovered not indexed, crawled not indexed, robots.txt sitemap, seo index audit, sitemap health score
中文说明
这个工具做什么
读取网站的 XML 站点地图(从 robots.txt、sitemap index、.xml.gz 发现),抽出全部 URL,并可逐个检查影响收录的问题:404、重定向、noindex、canonical 不一致、被 robots.txt 拦截、过期 lastmod。输出健康分和问题 CSV。它不是整站爬虫,也不替代 Google Search Console。
价格
每次启动 $0.005;每抽出 1,000 条 URL $0.30;每审计 1,000 个页面 $1.20。只抽取、不审计时只付第一项。
More tools by pdbear: https://apify.com/pandabear · https://pdbear.com