XML Sitemap Checker & Extractor: 404, noindex & Canonical Audit avatar

XML Sitemap Checker & Extractor: 404, noindex & Canonical Audit

Pricing

from $0.30 / 1,000 sitemap url saveds

Go to Apify Store
XML Sitemap Checker & Extractor: 404, noindex & Canonical Audit

XML Sitemap Checker & Extractor: 404, noindex & Canonical Audit

Extract every URL from a site's XML sitemaps (robots.txt, sitemap index, .xml.gz) and audit the ones that block indexing: 404s, redirects, noindex, canonical mismatch, robots.txt blocks, stale lastmod. Health score + issues CSV. $0.30/1k URLs extracted, $1.20/1k pages audited.

Pricing

from $0.30 / 1,000 sitemap url saveds

Rating

0.0

(0)

Developer

pdbear

pdbear

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

18 hours ago

Last modified

Features

Limited permissions

Share

XML Sitemap Checker & Extractor: find sitemap URLs Google will not index

Best for: SEO work when pages sit in “Discovered – currently not indexed” or “Crawled – currently not indexed,” or when you need every URL a site declares in its XML sitemaps, as a CSV.

Not for: a full-site crawler that discovers URLs not listed in a sitemap, or a Google Search Console replacement. This Actor reads sitemaps and, optionally, fetches those URLs.

A common cause of “not indexed” is a dirty sitemap: it lists URLs that 404, redirect, carry noindex, point their canonical somewhere else, or are blocked by robots.txt. Those URLs waste crawl budget. Give this Actor a domain or a sitemap URL. It discovers sitemaps from robots.txt, follows nested sitemap indexes (including .xml.gz), extracts every URL, then can fetch each one and flag what is wrong. You get a health score and an ISSUES CSV you can hand to a developer.

Why this one

  • Sitemap discovery is included. robots.txt, common sitemap paths, nested indexes, and .xml.gz.
  • The audit is about indexability, not only HTTP status. 404s and redirect chains, plus noindex, canonical mismatch, robots.txt blocks, stale lastmod, and title.
  • You can extract without auditing. Turn auditPages off and you only pay $0.30 per 1,000 URLs saved.
  • Honest price. $0.30 per 1,000 URLs extracted, $1.20 per 1,000 pages audited, $0.005 to start. A status-only Store checker may be cheaper per URL; you are paying for the extra indexability fields and the issues export.

Quick start

{
"startUrls": ["https://www.djangoproject.com"],
"auditPages": true,
"maxUrlsPerSite": 100,
"maxAuditsPerSite": 50,
"outputOnlyIssues": false
}

Input

FieldDescriptionDefault
startUrlsSite home pages or direct sitemap URLssample site
auditPagesFetch each URL and audit indexabilitytrue
maxUrlsPerSiteCap on URLs extracted—
maxAuditsPerSiteCap on pages fetched—
outputOnlyIssuesKeep only URLs with a problemfalse
staleLastmodDaysFlag lastmod older than this many days—
includeUrlPatterns / excludeUrlPatternsOptional filters—
maxSitemapsPerSiteCap on sitemap files followed—
concurrencyParallel fetches—

Output example

Each dataset row is one sitemap URL, with audit fields when auditPages is on (status, final URL, noindex, canonical, title, issue flags). The run also writes:

  • ISSUES — only the rows that failed a check, as CSV
  • SUMMARY — counts, health score, sitemap files found

Pricing (pay per event)

EventPrice
Actor start (per run)$0.005
Sitemap URL saved$0.0003 ($0.30 per 1,000)
Page audited$0.0012 ($1.20 per 1,000)

Example: extract 1,000 URLs and audit 200 of them ≈ $0.005 + $0.30 + $0.24 = $0.545. Set a maximum cost per run; the Actor stops cleanly and keeps what it already saved.

Limitations

  • Only URLs that appear in sitemaps are considered. Orphan pages are out of scope.
  • A live fetch can be blocked or throttled by the target site. Those rows are marked, not invented.
  • This does not query Google Search Console, so it cannot prove why Google chose not to index a page.

FAQ

Will this get my pages indexed? No. It tells you which sitemap URLs are unhealthy, so you or a developer can fix them.

Can I only extract URLs? Yes. Set auditPages to false.

Does it follow sitemap indexes and .xml.gz? Yes.

Keywords

xml sitemap checker, sitemap extractor, sitemap URL extractor, sitemap audit, 404 in sitemap, noindex, canonical mismatch, discovered not indexed, crawled not indexed, robots.txt sitemap, seo index audit, sitemap health score


中文说明

这个工具做什么

读取网站的 XML 站点地图(从 robots.txt、sitemap index、.xml.gz 发现),抽出全部 URL,并可逐个检查影响收录的问题:404、重定向、noindex、canonical 不一致、被 robots.txt 拦截、过期 lastmod。输出健康分和问题 CSV。它不是整站爬虫,也不替代 Google Search Console。

价格

每次启动 $0.005;每抽出 1,000 条 URL $0.30;每审计 1,000 个页面 $1.20。只抽取、不审计时只付第一项。

More tools by pdbear: https://apify.com/pandabear · https://pdbear.com