# Sitemap Extractor & Sitemap URL Scraper + SEO Index Audit (`pandabear/sitemap-seo-auditor`) Actor

Sitemap extractor: get every URL from a website's XML sitemaps (robots.txt, sitemap index, .xml.gz) and audit them for SEO indexing problems: 404s, redirects, noindex, canonical mismatch, robots.txt blocks and stale lastmod.

- **URL**: https://apify.com/pandabear/sitemap-seo-auditor.md
- **Developed by:** [Andy zeng](https://apify.com/pandabear) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 sitemap url saveds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & SEO Index Audit

**Extract every URL from a website's XML sitemaps and find the pages Google can't index.** Give it a domain or a sitemap URL. The actor reads `robots.txt` to find the sitemaps, follows nested sitemap indexes (including `.xml.gz`), and lists every URL with `lastmod`, `changefreq`, `priority`, hreflang alternates and image/video counts. It can also fetch each URL and flag **404s, redirects, noindex, canonical mismatches, robots.txt blocks, stale lastmod dates and duplicates**, which are the usual reasons for "Discovered / Crawled – currently not indexed" in Google Search Console.

### What you get

- **A full sitemap URL list** (CSV, JSON or Excel export) across all sitemap files, de-duplicated.
- **An indexability audit per URL:** HTTP status, redirect chain, final URL, `meta robots`, `X-Robots-Tag`, canonical, page title and response time.
- **Issue flags** for every URL (see the table below).
- **A site health score** plus issue counts and sitemap-file errors in the `SUMMARY` record.
- **An `ISSUES` CSV** of only the problem URLs, ready to hand to a developer or import into a spreadsheet.

### Use cases

- **Technical SEO audits:** find sitemap URLs that return 404, redirect, are noindexed or point their canonical elsewhere. Google treats these as sitemap errors.
- **Migrations and redesigns:** check that the new sitemap only lists live, indexable URLs.
- **Competitor research:** extract a competitor's full URL inventory, or find their newest pages by `lastmod`.
- **Content inventory:** get a full page list with last-modified dates for content pruning or refresh planning.
- **Monitoring:** schedule it weekly and alert when new 404s or noindex pages appear in the sitemap.
- **Feeding other tools:** pass the URL list to crawlers, scrapers, RAG/LLM pipelines or the Wayback Machine URL Extractor.

### Input

| Field | Description | Default |
|---|---|---|
| `startUrls` | Domains (`example.com`), homepages, or direct sitemap URLs | required |
| `auditPages` | Fetch and audit each URL. Turn it off to only extract URLs | `true` |
| `maxUrlsPerSite` | Maximum unique sitemap URLs per site | 100 |
| `maxAuditsPerSite` | Maximum URLs to fetch and audit per site | 50 |
| `outputOnlyIssues` | Save only URLs that have issues (you pay only for saved URLs) | `false` |
| `staleLastmodDays` | Flag `lastmod` dates older than N days (0 = off) | 365 |
| `includeUrlPatterns` / `excludeUrlPatterns` | Regex filters, e.g. `/blog/` | – |
| `maxSitemapsPerSite` | Maximum sitemap files read (including nested indexes) | 200 |
| `concurrency` | Parallel requests per site (keep it polite) | 5 |

Example:

```json
{
  "startUrls": ["example.com", "https://shop.example.com/sitemap_index.xml"],
  "auditPages": true,
  "maxUrlsPerSite": 5000,
  "maxAuditsPerSite": 1000,
  "outputOnlyIssues": false
}
```

### Output

Each dataset item is one sitemap URL:

```json
{
  "site": "wordpress.org",
  "url": "https://wordpress.org/enterprise/media/",
  "sitemapUrl": "https://wordpress.org/sitemap-1.xml",
  "lastmod": "2022-03-28T20:20:42Z",
  "priority": null,
  "hreflangAlternates": null,
  "statusCode": 200,
  "finalUrl": "https://wordpress.org/enterprise/media/",
  "redirectCount": 0,
  "metaRobots": "max-image-preview:large, noindex,follow",
  "canonical": "https://wordpress.org/enterprise/media/",
  "title": "Media | WordPress.org",
  "responseTimeMs": 286,
  "issues": ["stale-lastmod", "noindex"],
  "issueCount": 2
}
```

Key-value store records:

- `SUMMARY`: per-site health score, issue counts, sitemaps read, sitemap errors and site-level problems (missing robots.txt, no sitemap, and so on).
- `ISSUES`: a CSV of the problem URLs.

#### Issues detected

| Issue | Meaning |
|---|---|
| `http-404`, `http-410`, `http-5xx`… | The sitemap lists a URL that returns an error |
| `redirected` | The sitemap URL redirects. List the final URL instead |
| `noindex` | `meta robots` or `X-Robots-Tag` says noindex, which contradicts the sitemap |
| `canonical-mismatch` | The canonical tag points to a different URL |
| `blocked-by-robots-txt` | robots.txt disallows the URL for Googlebot |
| `unreachable` | Timeout or connection error |
| `non-html-content` | Returns a non-HTML content type unexpectedly |
| `stale-lastmod` / `future-lastmod` / `invalid-lastmod` | Problems with the `lastmod` date |
| `duplicate-in-sitemaps` | The same URL is listed more than once |
| `http-not-https`, `off-domain-url`, `invalid-url` | URL hygiene problems |

**Health score** is the percentage of audited URLs that are indexable: no error status, redirect, noindex, canonical mismatch or robots block.

### Pricing (pay per event)

| Event | Price |
|---|---|
| Actor start (per run) | $0.005 |
| Sitemap URL saved | $0.0003 (= $0.30 per 1,000 URLs) |
| Page audited | $0.0012 (= $1.20 per 1,000 audited pages) |

A full audit of 1,000 pages costs about **$1.50**. Extracting 10,000 URLs without auditing costs about **$3**. Set a maximum cost per run, and the actor stops cleanly at your limit while keeping the results collected so far.

### FAQ

**Is this legal?** The actor reads public sitemaps and robots.txt (which exist so crawlers can read them) and requests public pages at a polite rate with a clear user agent. It doesn't log in or bypass anything.

**Can it find sitemaps automatically?** Yes. It reads `Sitemap:` lines in robots.txt, then falls back to `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and other common paths.

**Does it support gzip, sitemap indexes, hreflang, image and news sitemaps?** Yes.

**Does it execute JavaScript?** No. It reads the HTML head as served, which is what matters for meta robots and canonical in most cases.

### Related actors by pandabear

- [App Store Scraper - Search, Top Charts & Keyword Ranks](https://apify.com/pandabear/app-store-search-charts-scraper): iOS apps by keyword, top chart, category or developer, with full details and ASO keyword rank tracking.
- [Wayback Machine URL Extractor & Lost Page Finder](https://apify.com/pandabear/wayback-lost-url-finder): finds old URLs that dropped out of your sitemap and now 404, and suggests 301 redirects.
- [Bulk WHOIS & Domain Availability Checker](https://apify.com/pandabear/bulk-whois-domain-checker): bulk RDAP/WHOIS lookup with registrar, creation and expiry dates, domain age, availability and DNS.
- [App Store Reviews Scraper - All Countries + Analytics](https://apify.com/pandabear/app-store-reviews-scraper): Apple App Store reviews from 76 countries, with app details, rating trends by version and complaint keywords.
- [Website Contact Scraper - Emails, Phones, Socials & Tech Stack](https://apify.com/pandabear/website-contact-enrichment): emails, phones, social links, company info and tech stack from any list of websites.
- [Trending Topics Scraper - Google Trends, HN, Wikipedia & More](https://apify.com/pandabear/trending-topics-aggregator): trending topics from Google Trends, Hacker News, Wikipedia, GitHub, Bluesky, Baidu and more, with heat scores and cross-platform clusters.

***

### 中文说明

#### 这个工具做什么

输入一个域名或 sitemap 地址，工具会自动从 `robots.txt` 和常见路径找到网站的全部 XML sitemap，递归读取 sitemap 索引（支持 `.xml.gz`），提取所有 URL，并附带 `lastmod`、`changefreq`、`priority`、hreflang 多语言链接和图片/视频数量。开启审计后，工具会逐个访问这些 URL，检查谷歌无法收录的原因：404/5xx 错误、重定向、noindex、canonical 指向别处、被 robots.txt 屏蔽、lastmod 过旧、重复 URL 等。这些正是 Google Search Console 里"已发现/已抓取 – 尚未编入索引"的常见原因。

#### 适用场景

- 技术 SEO 审计：找出 sitemap 中的死链、重定向、noindex 和 canonical 冲突
- 网站改版或迁移后检查新 sitemap 的质量
- 竞品分析：导出竞争对手的完整 URL 列表，按 lastmod 找出他们的最新页面
- 内容盘点：获取全站页面和最后修改时间
- 定期监控：每周运行一次，sitemap 出现新的 404 或 noindex 时提醒你
- 为爬虫、抓取工具、RAG/大模型流程提供 URL 列表

#### 输入参数

- `startUrls`：域名、首页或 sitemap 地址（必填）
- `auditPages`：是否逐个访问并审计 URL（默认开启；关闭后只提取 URL，更便宜）
- `maxUrlsPerSite`：每个网站最多提取多少个 URL（默认 100）
- `maxAuditsPerSite`：每个网站最多审计多少个 URL（默认 50）
- `outputOnlyIssues`：只保存有问题的 URL（只对保存的 URL 收费）
- `staleLastmodDays`：lastmod 早于多少天就标记为过旧（默认 365，填 0 关闭）
- `includeUrlPatterns` / `excludeUrlPatterns`：正则过滤
- `maxSitemapsPerSite`、`concurrency`：高级设置

#### 输出

- 数据集：每条记录对应一个 URL，包含状态码、最终 URL、重定向链、meta robots、X-Robots-Tag、canonical、标题、响应时间，以及问题列表 `issues`
- `SUMMARY`：每个网站的健康分（可收录 URL 所占比例）、各类问题数量、sitemap 文件错误
- `ISSUES`：只含问题 URL 的 CSV，可以直接发给开发人员

#### 价格（按事件计费）

- 每次运行启动：$0.005
- 每保存一个 URL：$0.0003（每 1000 个 $0.30）
- 每审计一个页面：$0.0012（每 1000 个 $1.20）

完整审计 1000 个页面约 $1.50；只提取 1 万个 URL 约 $3。可以设置单次运行的费用上限，达到上限后工具会正常停止并保留已有结果。

#### 常见问题

- **合法吗？** 只读取公开的 sitemap、robots.txt 和公开网页，请求频率克制，User-Agent 清晰标识，不登录、不绕过任何限制。
- **能自动找到 sitemap 吗？** 能。优先读取 robots.txt 里的 `Sitemap:` 行，找不到时再尝试 `/sitemap.xml`、`/sitemap_index.xml`、`/wp-sitemap.xml` 等常见路径。
- **支持 gzip、sitemap 索引、hreflang、图片和新闻 sitemap 吗？** 都支持。
- **会执行 JavaScript 吗？** 不会。工具读取服务器返回的原始 HTML，大多数情况下这已足够判断 meta robots 和 canonical。

#### pandabear 的其他工具

- [App Store Scraper - Search, Top Charts & Keyword Ranks](https://apify.com/pandabear/app-store-search-charts-scraper)：按关键词、排行榜、分类或开发者抓取苹果应用数据，并跟踪 ASO 关键词排名
- [Wayback Machine URL Extractor & Lost Page Finder](https://apify.com/pandabear/wayback-lost-url-finder)：找回从 sitemap 中消失、现在返回 404 的旧网址，并给出 301 重定向建议
- [Bulk WHOIS & Domain Availability Checker](https://apify.com/pandabear/bulk-whois-domain-checker)：批量查询 RDAP/WHOIS，包括注册商、注册和到期日期、域名年龄、是否可注册以及 DNS
- [App Store Reviews Scraper](https://apify.com/pandabear/app-store-reviews-scraper)：抓取苹果 App Store 76 个国家的应用评论，附带应用详情、按版本的评分趋势和差评关键词
- [Website Contact Scraper](https://apify.com/pandabear/website-contact-enrichment)：从网站列表中提取邮箱、电话、社交媒体链接、公司信息和技术栈
- [Trending Topics Scraper](https://apify.com/pandabear/trending-topics-aggregator)：汇总 Google Trends、Hacker News、维基百科、GitHub、Bluesky、百度热搜等平台的热门话题，附热度分和跨平台聚类

# Actor input Schema

## `startUrls` (type: `array`):

Domains (example.com), homepages, or direct sitemap URLs (https://example.com/sitemap.xml). For a domain, sitemaps are discovered from robots.txt and common locations.

## `auditPages` (type: `boolean`):

Fetch every sitemap URL and check HTTP status, redirects, meta robots / X-Robots-Tag noindex, canonical tag and title. Turn off to only extract sitemap URLs (cheaper and faster).

## `maxUrlsPerSite` (type: `integer`):

Stop reading sitemaps after this many unique URLs per site.

## `maxAuditsPerSite` (type: `integer`):

Only the first N extracted URLs are fetched and audited. The rest are still saved with sitemap-level checks.

## `outputOnlyIssues` (type: `boolean`):

Save only URLs that have at least one problem (you are charged only for saved URLs).

## `staleLastmodDays` (type: `integer`):

Flag URLs whose <lastmod> is older than this many days. 0 disables the check.

## `includeUrlPatterns` (type: `array`):

Only keep URLs matching at least one of these regular expressions, e.g. /blog/

## `excludeUrlPatterns` (type: `array`):

Skip URLs matching any of these regular expressions, e.g. /tag/ or ?page=

## `maxSitemapsPerSite` (type: `integer`):

Upper limit of sitemap files (including nested sitemap indexes) read per site.

## `concurrency` (type: `integer`):

Parallel page requests per site. Keep it low (3-8) to be polite to the audited server.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.djangoproject.com"
  ],
  "auditPages": true,
  "maxUrlsPerSite": 100,
  "maxAuditsPerSite": 50,
  "outputOnlyIssues": false,
  "staleLastmodDays": 365,
  "maxSitemapsPerSite": 200,
  "concurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

Every extracted sitemap URL with status, canonical, meta robots, lastmod and detected issues.

## `issues` (type: `string`):

CSV of URLs with at least one SEO issue (site,url,issues,statusCode,finalUrl,canonical,lastmod,sitemapUrl).

## `summary` (type: `string`):

Per-site health score, issue counts, sitemap files read and sitemap-level errors.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.djangoproject.com"
    ],
    "maxUrlsPerSite": 100,
    "maxAuditsPerSite": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("pandabear/sitemap-seo-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.djangoproject.com"],
    "maxUrlsPerSite": 100,
    "maxAuditsPerSite": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("pandabear/sitemap-seo-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.djangoproject.com"
  ],
  "maxUrlsPerSite": 100,
  "maxAuditsPerSite": 50
}' |
apify call pandabear/sitemap-seo-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pandabear/sitemap-seo-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oqcQicho33areMtcL/builds/FIPscQp1aDWZnv2Ub/openapi.json