# XML Sitemap Auditor, URL Checker & Inventory (`xjenn/sitemap-url-qa-inventory`) Actor

Discover nested XML sitemaps, deduplicate and validate URLs, flag broken pages and redirects, and export a clean crawl inventory to JSON, CSV, Excel, XML, or RSS. Built for SEO audits, site migrations, QA, monitoring, and automation with conservative limits and transparent error records.

- **URL**: https://apify.com/xjenn/sitemap-url-qa-inventory.md
- **Developed by:** [Xjenn tools](https://apify.com/xjenn) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL QA & Crawl Inventory

Turn a website root or XML sitemap into a clean, auditable URL inventory. The Actor recursively discovers sitemap indexes, reads `.xml.gz` files, deduplicates entries, and optionally checks every public URL for HTTP status, redirects, response time, content type, and Last-Modified metadata.

It uses plain HTTP requests—no browser, proxy, external API, or LLM—so runs stay predictable and the output works well in JSON/CSV exports, Apify API integrations, schedules, and MCP/agent workflows.

### What you get

- Recursive `sitemapindex` and `urlset` traversal
- `robots.txt` sitemap discovery with `/sitemap.xml` fallback
- Gzip-compressed sitemap support
- URL deduplication across sitemap files
- HEAD checks with safe GET fallback for servers that reject HEAD
- Redirect chain, final URL, HTTP status, response time, content type, and HTTP Last-Modified
- Sitemap `lastmod`, `changefreq`, and `priority`
- Wildcard include/exclude filters and hard URL/sitemap limits
- Explicit `sitemap_error` records instead of silent partial results
- Public-network-only fetching to reduce SSRF risk
- A run summary in the `OUTPUT` key-value-store record

### Quick start

```json
{
  "startUrls": [{ "url": "https://example.com" }],
  "discoverFromRobots": true,
  "checkStatus": true,
  "sameOriginOnly": true,
  "maxUrls": 10000,
  "maxSitemaps": 100,
  "maxConcurrency": 15,
  "requestTimeoutSecs": 20,
  "maxRedirects": 5,
  "includePatterns": [],
  "excludePatterns": []
}
```

You can also enter a sitemap directly:

```json
{
  "startUrls": [{ "url": "https://example.com/sitemap-index.xml" }],
  "checkStatus": false
}
```

### Input

| Field | Default | Description |
|---|---:|---|
| `startUrls` | required | Website roots, sitemap files, sitemap indexes, or `.xml.gz` URLs. |
| `discoverFromRobots` | `true` | Reads `Sitemap:` directives from `robots.txt` for website-root inputs. |
| `checkStatus` | `true` | Audits every discovered page URL. Disable for inventory-only runs. |
| `sameOriginOnly` | `true` | Keeps sitemap files and page URLs on the hostname of the matching start URL. |
| `maxUrls` | `10000` | Hard output-work limit, from 1 to 100,000. |
| `maxSitemaps` | `100` | Maximum recursively processed sitemap files, from 1 to 1,000. |
| `maxConcurrency` | `15` | Parallel page checks, from 1 to 50. |
| `requestTimeoutSecs` | `20` | Per-request timeout, from 3 to 120 seconds. |
| `maxRedirects` | `5` | Maximum redirects per request, from 0 to 10. |
| `includePatterns` | `[]` | Optional wildcard patterns. At least one must match when supplied. |
| `excludePatterns` | `[]` | Wildcard patterns that remove matching URLs. |

Wildcard examples: `*/blog/*`, `*.pdf`, `*?preview=*`.

### Dataset output

Each URL produces one visible dataset item. A typical item looks like this:

```json
{
  "recordType": "url",
  "url": "https://example.com/products/blue-widget",
  "sourceSitemap": "https://example.com/product-sitemap.xml.gz",
  "lastmod": "2026-08-20",
  "changefreq": "weekly",
  "priority": 0.8,
  "statusCode": 301,
  "finalUrl": "https://example.com/product/blue-widget",
  "isBroken": false,
  "isRedirect": true,
  "redirectCount": 1,
  "redirectChain": ["https://example.com/products/blue-widget"],
  "responseTimeMs": 182,
  "contentType": "text/html; charset=utf-8",
  "httpLastModified": "Thu, 20 Aug 2026 08:00:00 GMT",
  "issueCodes": ["REDIRECT"],
  "checkedAt": "2026-08-21T12:00:00.000Z",
  "errorCode": null,
  "errorMessage": null
}
```

Sitemap-level failures are returned as free, explicit `sitemap_error` records with an `errorCode` and `errorMessage`. Common issue/error codes include `HTTP_404`, `REDIRECT`, `MISSING_LASTMOD`, `MALFORMED_XML`, `UNSUPPORTED_SITEMAP`, `REQUEST_TIMEOUT`, `DNS_ERROR`, `RESPONSE_TOO_LARGE`, and `PRIVATE_NETWORK_BLOCKED`.

The `OUTPUT` record contains aggregate counts such as discovered/processed URLs, broken URLs, redirects, missing `lastmod`, sitemap errors, and whether a run stopped at a configured or spending limit.

### Pricing

Recommended Store setup: **$0.0004 per `url-audited` event ($0.40 per 1,000 audited URL records)** plus Apify's default `apify-actor-start` event. Remove the `apify-default-dataset-item` synthetic event to prevent double charging. Sitemap-level error records are not charged by the custom event.

The Actor checks the run spending limit after each charged result and stops cleanly when the limit is reached. Actual Store pricing is displayed by Apify before a run starts and controls over this README if they differ.

### Responsible use and privacy

- Use the Actor only for websites you own, administer, or are authorized to audit, or for public resources where automated access is allowed.
- Respect website terms, applicable law, rate limits, and server capacity. Lower `maxConcurrency` when necessary.
- The Actor does not bypass authentication, CAPTCHAs, access controls, or anti-bot systems.
- It does not request residential proxies and does not collect account credentials.
- URL credentials and targets resolving to loopback, link-local, private, multicast, or otherwise unsafe network addresses are blocked.
- Fetched response bodies are not stored. Sitemap XML is held in memory only while parsing; page checks normally use HEAD.

### Limitations

- A successful HEAD response does not prove that a JavaScript application renders correctly.
- Some servers intentionally block HEAD or automated clients. The Actor retries 403, 405, and 501 responses with a small ranged GET, but a block may remain.
- `sameOriginOnly` compares hostnames exactly; `www.example.com` and `example.com` are different hostnames.
- The Actor validates sitemap structure and URL availability, not full search-engine indexing eligibility.
- Very slow sites can make large status-check runs expensive. Start with a small `maxUrls` and use an Apify run spending limit.

### Local development

```bash
npm install
node --test test/*.test.mjs
node --check src/core.mjs
node --check src/main.mjs
```

For local pay-per-event behavior, Apify supports `ACTOR_TEST_PAY_PER_EVENT=true`. Final pricing and cloud usage must be verified in an unpublished Apify test Actor before Store publication.

### Support scope

Bug reports should include the run ID, a public reproducible sitemap URL, expected result, and actual result. Private-site access, custom scraping, account login, and site-specific bypasses are outside the support scope.

# Actor input Schema

## `startUrls` (type: `array`):

Website roots, sitemap.xml URLs, sitemap indexes, or .xml.gz sitemap files.

## `discoverFromRobots` (type: `boolean`):

For website-root inputs, read Sitemap: directives from robots.txt before trying /sitemap.xml.

## `checkStatus` (type: `boolean`):

Check HTTP status, redirects, response time, content type, and Last-Modified for every discovered URL.

## `sameOriginOnly` (type: `boolean`):

Ignore sitemap entries on other hostnames. Recommended for predictable scope and cost.

## `maxUrls` (type: `integer`):

Hard limit across all inputs. Processing stops cleanly when reached.

## `maxSitemaps` (type: `integer`):

Limits recursive sitemap-index traversal.

## `maxConcurrency` (type: `integer`):

Number of page-status checks performed in parallel.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each sitemap or page request.

## `maxRedirects` (type: `integer`):

Maximum number of HTTP redirects followed while checking each discovered URL.

## `includePatterns` (type: `array`):

Optional wildcard patterns such as */blog/* or \*.pdf. A URL must match at least one when provided.

## `excludePatterns` (type: `array`):

Wildcard patterns to omit, such as */tag/* or *?preview=*.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "discoverFromRobots": true,
  "checkStatus": true,
  "sameOriginOnly": true,
  "maxUrls": 10000,
  "maxSitemaps": 100,
  "maxConcurrency": 15,
  "requestTimeoutSecs": 20,
  "maxRedirects": 5,
  "includePatterns": [],
  "excludePatterns": []
}
```

# Actor output Schema

## `urlRecords` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("xjenn/sitemap-url-qa-inventory").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("xjenn/sitemap-url-qa-inventory").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call xjenn/sitemap-url-qa-inventory --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,xjenn/sitemap-url-qa-inventory"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Nj00Mh8jteYycwdfl/builds/cUZxk5OefWIVcGZ30/openapi.json