# Sitemap and URL Status Auditor (`vizellie/sitemap-url-auditor`) Actor

Checks every URL in a sitemap for broken links, redirect chains, noindex conflicts and canonical problems. You get a health score and a list of what to fix first. Turn on change tracking to see what changed since the last run. $0.001 per URL, and $0.03 for an optional client report.

- **URL**: https://apify.com/vizellie/sitemap-url-auditor.md
- **Developed by:** [Vi Zellie](https://apify.com/vizellie) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap and URL Status Auditor

Point it at a website or a sitemap. It finds the sitemap, checks each URL, and tells you what is broken, what redirects, and which pages should not be in the sitemap. Every run also produces a health score, a short fix-these-first list, and a shareable HTML report.

Not legal advice; general information only.

### Who it is for

Site owners, SEO consultants and agencies who want a repeatable check after a migration, a content update, or as a weekly health job.

### What you get

- Sitemap discovery from `robots.txt`, sitemap indexes, `.xml.gz` files and plain-text sitemaps.
- For each URL: HTTP status, final URL, redirect hop count, response time, content type, title, canonical and index instructions.
- Flags for broken links, long redirect chains, redirect loops, noindex pages in the sitemap, canonical mismatches and stale sitemap dates.
- A health score from 0 to 100, plus a prioritised list of what to fix first.
- Optional change tracking: URLs that newly broke, were added, or disappeared since the last run.
- An HTML report saved with the run, with no page bodies stored.
- Polite crawling that honours `robots.txt`. Skipped URLs are not charged.

The auditor only requests public websites. Private, local and metadata addresses are skipped and are not charged.

### Tutorial

1. Paste one or more websites or sitemap URLs into **Website or sitemap URLs**.
2. Set **Max URLs** to the budget you want for this run. Several sites share that budget evenly.
3. Leave **Respect robots.txt**, **Same domain only** and the SEO checks on unless you have a reason to change them.
4. To compare with last time, turn on **Track changes since the last run** and keep the same changes key for that site.
5. Run the Actor.
6. Open the dataset for every URL, the `OUTPUT` record for the score and the change list, and the `REPORT` record for the HTML summary.
7. Schedule the task weekly if you want a standing check.

### Client-ready report

Use this when you want one file to send to a client. It is off by default. Off does not add a file and does not add a fee.

1. Paste a website or a sitemap URL.
2. Set **Client-ready report** to **HTML**.
3. Run the Actor.
4. Open the Output tab. **Client report (HTML)** is `REPORT.html` in the key-value store. **Client report summary** lists that key.

The page has a cover with the site, the date and the health grade, a short summary in plain language, and a table of findings. Each finding says what it means and how to fix it. A run with more than one site has a section per site. Pass, warning and fail stay readable in black and white. The footer says Generated with Sitemap and URL Status Auditor.

The report fee is charged once per run, and only after the file is saved. A failed save is not charged. The dataset and the existing `REPORT` summary stay included in the URL fee. This Actor saves HTML. It does not save a PDF.

### Example input

```json
{
  "startUrls": ["https://example.com"],
  "maxUrls": 100,
  "maxConcurrency": 5,
  "sameDomainOnly": true,
  "trackChanges": true,
  "changesKey": "example.com",
  "staleLastmodDays": 365
}
```

### Example output

One dataset row per URL that was actually checked:

```json
{
  "url": "https://example.com/pricing",
  "finalStatus": 200,
  "finalUrl": "https://example.com/pricing",
  "redirectHopCount": 0,
  "title": "Pricing",
  "titleLength": 7,
  "canonical": "https://example.com/pricing",
  "canonicalMismatch": false,
  "noindex": false,
  "robotsConflict": false,
  "lastmodStale": false,
  "isBroken": false,
  "issues": ["title-too-short"]
}
```

The run summary includes `health.score`, `health.fixFirst`, status counts, and `changes` when tracking is on. `changes` lists URLs that newly broke, were added, or disappeared. If the run stopped early because of a limit, disappeared URLs are left out so a shorter run is not mistaken for a wave of deletions.

### Pricing

You pay a small fee when the run starts, plus a fee for each URL that is checked.

| What | Price |
|---|---|
| Run start | $0.00005 |
| URL checked | $0.001 |
| Client-ready report | $0.03 |

URLs skipped because of `robots.txt`, because they are outside the domain filter, or because they are not on the public internet, are not charged. The health score, the change list and the HTML summary in `REPORT` are included. They do not add a fee. The client-ready report is a separate file and is charged once per run, only when that file is saved. Leave **Client-ready report** on Off and that fee is not charged. Add $0.03 to any example below when the report is saved. The start fee is Apify's default, so a scheduled run that checks nothing stays cheap.

**Examples**

- 100 URLs: $0.00005 + (100 × $0.001) = **$0.10005**.
- 500 URLs: $0.00005 + (500 × $0.001) = **$0.50005**.
- A weekly check of 500 URLs: about **$2.00** a month.
- A run that checks nothing: **$0.00005**.

Set **Max URLs**, and Apify's maximum cost per run, before you point this at a large site. The pricing tab on the store is the price you are charged if it ever differs.

### Scheduling and integrations

Save a task with **Track changes** on and schedule it. Use one changes key per site. Send the dataset or the HTML report to Google Sheets, Slack, email, Make or Zapier, or read it from an AI agent over MCP. The report is a single HTML file in the run's key-value store under `REPORT`.

### No personal data (AVG/GDPR)

The audit stores technical results, not page content.

- Page bodies are not stored. Only status, timing, and the meta fields needed for the SEO checks (title, canonical, robots, headings count, language).
- Email addresses, phone numbers, street addresses and secret-like query values are removed from stored URLs, titles and the report.
- Cookies and request headers are never logged or saved.
- The change history stores URL and status only.
- A person's name that is the public title of a page can remain, because the title itself is what the length check measures. Contact details inside that title are removed.

Only audit sites you are allowed to check, and keep concurrency low on sites you do not operate.

### FAQ

**The score is not 100. What should I open first?**
Open `health.fixFirst` in the summary, or the same list in the HTML report. It is ordered so the most serious items are first.

**Why were some URLs free?**
Skipped URLs are not checked and not charged. That includes `robots.txt` exclusions and addresses that are not public.

**Why is a sitemap URL marked noindex?**
The sitemap lists it for indexing, but the page tells Google not to index it. Remove it from the sitemap or allow indexing.

**A redirect looks fine. Why is it flagged?**
A single hop is reported, not treated as a failure. Chains longer than two hops are flagged when that check is on. You can turn the flag off and still see the hop count.

**What is a stale lastmod?**
The date in the sitemap is older than the number of days you set (365 by default). Set **Stale lastmod** to 0 to turn the check off.

### Good to know

Some sites answer cloud servers with a challenge page. The result shows that status rather than hiding it. JavaScript is not executed, so tags that only appear in the browser are not seen. With several input sites, **Max URLs** is split evenly.

### Next additions

- Agency white-label logo
- Dutch report

# Actor input Schema

## `startUrls` (type: `array`):

Websites (for example https://example.com) or sitemap URLs (for example https://example.com/sitemap.xml). For a website, sitemaps are discovered from robots.txt, with /sitemap.xml and /sitemap_index.xml as a fallback. Private and non-public addresses are skipped and are not charged.

## `maxUrls` (type: `integer`):

Maximum number of URLs to check per run (across all inputs). Keeps cost predictable.

## `maxConcurrency` (type: `integer`):

Parallel requests. Kept low by default to be polite to the sites you audit. Requests to the same host are also spaced by "Min delay per host" and any robots.txt Crawl-delay (max 10 s).

## `sameDomainOnly` (type: `boolean`):

Only check URLs on the same domain as the input (subdomains included). Off-domain URLs listed in a sitemap are skipped and counted in the summary.

## `useHeadRequests` (type: `boolean`):

Send a light HEAD request first. If the server rejects HEAD or returns an error, the URL is re-checked with GET so you get no false "broken" results. HTML pages are then fetched with GET to read title, canonical and meta robots (if "Check page content" is on).

## `checkPageContent` (type: `boolean`):

For HTML pages that return 200, download the first 1 MB and extract title, title length, canonical, meta robots, H1 count and lang. Turn off for a status-only check (faster, less traffic).

## `respectRobotsTxt` (type: `boolean`):

Skip URLs that robots.txt disallows for this crawler (they are listed in the summary and not charged). Strongly recommended; only turn off for sites you own.

## `includeInputUrls` (type: `boolean`):

Check the website URLs you entered (e.g. the homepage) in addition to the sitemap URLs.

## `checkRedirectChains` (type: `boolean`):

Save the redirect hop count, and flag chains longer than 2 hops.

## `checkRobotsConflicts` (type: `boolean`):

Flag pages whose index instructions disagree, including sitemap URLs marked noindex.

## `checkCanonicalMismatch` (type: `boolean`):

Flag HTML pages whose canonical URL is not the final URL after redirects.

## `staleLastmodDays` (type: `integer`):

Flag sitemap URLs whose lastmod is older than this many days. Use 0 to turn the check off.

## `trackChanges` (type: `boolean`):

Remember URL statuses and, on the next run, list URLs that newly broke, were added, or disappeared. This is included in the run price.

## `changesStoreName` (type: `string`):

Named store for the previous audit. Use a different store or key for each site.

## `changesKey` (type: `string`):

Key inside the changes store. Use one key per site so snapshots stay separate.

## `maxSitemaps` (type: `integer`):

Maximum number of sitemap files to download (sitemap indexes can contain hundreds).

## `maxRedirects` (type: `integer`):

Stop following a redirect chain after this many hops. The hop count is saved on every URL.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout.

## `minDelayPerHostMs` (type: `integer`):

Minimum time between two requests to the same host.

## `report` (type: `string`):

Save an HTML report you can hand to a client. Off is the default: the run, the dataset and the price stay as they are today. The report fee is charged once per run, and only after the file is saved.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com",
    "https://example.com/sitemap_index.xml"
  ],
  "maxUrls": 100,
  "maxConcurrency": 5,
  "sameDomainOnly": true,
  "useHeadRequests": true,
  "checkPageContent": true,
  "respectRobotsTxt": true,
  "includeInputUrls": true,
  "checkRedirectChains": true,
  "checkRobotsConflicts": true,
  "checkCanonicalMismatch": true,
  "staleLastmodDays": 365,
  "trackChanges": false,
  "changesStoreName": "sitemap-audit-state",
  "changesKey": "default",
  "maxSitemaps": 50,
  "maxRedirects": 10,
  "requestTimeoutSecs": 20,
  "minDelayPerHostMs": 250,
  "report": "off"
}
```

# Actor output Schema

## `urls` (type: `string`):

The overview dataset view, one row per checked URL.

## `seo` (type: `string`):

The seo dataset view.

## `redirects` (type: `string`):

The redirects dataset view.

## `summary` (type: `string`):

The OUTPUT record in the default key-value store.

## `report` (type: `string`):

The REPORT record in the default key-value store.

## `clientReport` (type: `string`):

The REPORT.html record. Saved when Client-ready report is HTML.

## `clientReportSummary` (type: `string`):

The CLIENT_REPORT record, with the report key.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("vizellie/sitemap-url-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("vizellie/sitemap-url-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrls": 100
}' |
apify call vizellie/sitemap-url-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vizellie/sitemap-url-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4YrMc95c0rOHZxMqg/builds/TBQR1A6k73xcCtg4a/openapi.json