# Sitemap Link Auditor: Broken Links, Redirects & Slow Pages (`devsolal/sitemap-link-auditor`) Actor

- **URL**: https://apify.com/devsolal/sitemap-link-auditor.md
- **Developed by:** [David-Solal Chalmeau](https://apify.com/devsolal) (community)
- **Categories:** Developer tools, SEO tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Link Auditor: Broken Links, Redirects & Slow Pages

Give it a site you own. It finds the site's sitemap, lists the URLs the site published there, and checks each one: HTTP status, the full redirect chain, how long the response took, content type, and the `lastmod` the sitemap declares. You get one flat JSON row per URL with a plain list of issues, ready for a dashboard, a CI check, or an AI agent.

This Actor only audits URLs the site itself published in its own sitemap. It does not crawl, it does not follow links found in HTML, and it does not check external links.

### What you get per URL

| Field | Meaning |
|---|---|
| `site` | The site from your input |
| `url` | The URL as published in the sitemap |
| `finalUrl` | Where the request ended up |
| `statusCode` | Final HTTP status (number), `null` when the URL was not fetched |
| `redirectChain` | `[{url, status}]` for every hop, including the final response |
| `redirectChainTruncated` | `true` when more than 5 redirects were attempted; the chain stops at the cap |
| `redirectCount` | Number of redirects followed (max 5) |
| `responseTimeMs` | Time to the final response headers |
| `contentType` | `Content-Type` of the final response, or `null` |
| `lastmod` | `<lastmod>` from the sitemap, or `null` |
| `status` | `OK`, `ISSUE`, `TIMEOUT`, `ROBOTS_DISALLOWED`, `EXTERNAL_HOST_SKIPPED`, `REFUSED` or `ERROR` |
| `issues` | List of `{severity, code, message}` — see below |
| `checkedAt` | ISO timestamp |

Issue codes: `BROKEN_4XX`, `SERVER_ERROR_5XX`, `REDIRECT_CHAIN_LONG` (3+ hops, or more than 5 when `redirectChainTruncated` is set), `REDIRECT_TO_DIFFERENT_HOST`, `SLOW_RESPONSE` (>3000 ms), `TIMEOUT`, `ROBOTS_DISALLOWED`.

Error rows (free, no `issues` list): `INVALID_SITE`, `NO_SITEMAP`, `SITEMAP_UNUSABLE`, `SITEMAP_FETCH_FAILED`, `CHECK_FAILED`, `REFUSED_PRIVATE_ADDRESS`, `FETCH_FAILED`.

### Example input

```json
{ "sites": ["www.sitemaps.org"], "maxUrlsPerSite": 3 }
```

### Example output (real run, one row shown in full)

```json
{
  "site": "www.sitemaps.org",
  "url": "https://www.sitemaps.org/",
  "finalUrl": "https://www.sitemaps.org/",
  "statusCode": 200,
  "redirectChain": [{ "url": "https://www.sitemaps.org/", "status": 200 }],
  "redirectCount": 0,
  "responseTimeMs": 142,
  "contentType": "text/html",
  "lastmod": "2016-11-21",
  "issues": [],
  "checkedAt": "2026-10-28T05:31:12.744Z",
  "status": "OK"
}
```

A site without a sitemap returns one free row instead:

```json
{ "site": "example.com", "error": "NO_SITEMAP", "message": "No sitemap found via robots.txt, /sitemap.xml or /sitemap_index.xml" }
```

### How discovery works

1. `https://{host}/robots.txt` — every `Sitemap:` line is read.
2. No usable sitemap there? Then `/sitemap.xml`, then `/sitemap_index.xml`.
3. Sitemap indexes are followed up to 3 levels deep; gzip documents are inflated; `<loc>`, `<lastmod>`, CDATA and entities are parsed, including sloppy real-world XML. Parsing is a single linear pass (each tag is located by forward scanning, never by a regex retried at every position), so unclosed or truncated markup cannot stall the run. XML parsing is capped at 4 MB per document (above this the document is reported as `SITEMAP_UNUSABLE`, since this Actor checks at most 5000 URLs anyway), and only the first `maxUrlsPerSite` entries are collected.
4. Nothing found → one free `NO_SITEMAP` row for that site. A sitemap that was found but rejected (too large, bad gzip, not a sitemap, disallowed by robots.txt) produces one free `SITEMAP_UNUSABLE` row naming the URL and the reason instead.

### Pricing

Pay per event: a small start fee per run plus **$0.0018 for each URL actually fetched and delivered**. Rows we do not fetch are free: `NO_SITEMAP`, `SITEMAP_UNUSABLE`, invalid sites, fetch errors, timeouts (status `TIMEOUT`, no HTTP response received), URLs robots.txt disallows (status `ROBOTS_DISALLOWED`), URLs on another host (status `EXTERNAL_HOST_SKIPPED`) and refused internal addresses.

`includeOk: false` hides healthy results: a URL is delivered unless it came back as a clean `OK` (no `issues`, no `error`). You still receive every problem row **and** every free row — robots-disallowed, external-host-skipped, refused, and every error row (including `CHECK_FAILED` and `NO_STATUS`, which carry no `issues` list). Every URL is still checked, but **you are only charged for fetched URLs delivered with issues** — free rows never cost anything. If your spending limit is reached the run stops cleanly instead of doing unpaid work.

### Good to know

- **Own or authorised sites only.** Public domain names only: IP addresses, `localhost`, private and internal hostnames are rejected, and a sitemap entry pointing at `169.254.169.254`, `127.0.0.1`, `::1` or `*.internal` is refused without a single request. Every connection, including each redirect hop, is made to an address validated as public at connect time; a redirect to a private or metadata address is refused rather than followed.
- **robots.txt is honoured** for the URLs we check, on every redirect hop: a disallowed URL (or a redirect target on the same host) is reported with status `ROBOTS_DISALLOWED` and never requested. Disallowed sitemap documents are not downloaded either, and that includes a declared sitemap whose redirect hop lands in a disallowed path: the hop is refused before it is dialled and the site gets one free `SITEMAP_UNUSABLE` row.
- **Same host only.** `EXTERNAL_HOST_SKIPPED` means the URL in the sitemap is on a different host than the one you gave. This is literal: if your sitemap uses `www.example.com`, pass `www.example.com` (an apex input `example.com` will report its `www` URLs as skipped).
- **Polite by default:** 4 concurrent requests per host, at least 100 ms between request starts, 10 s timeout, identifying `apify-actor-fleet/1.0` User-Agent. Every individual request is paced — each redirect hop and each `HEAD`/`GET` attempt counts as a request, not just each URL check. Pacing is a promise about request starts (each start is reserved before any wait, so concurrent slots cannot start together); network connect time can shift the arrival time a server observes by a few milliseconds. `HEAD` first, with a `GET` (at most 8 KB read) when the server answers 403/405/501 or errors.
- Limits: `maxUrlsPerSite` default 200 (hard cap 5000), `maxSites` default 5 (hard cap 50), redirects followed 5. URLs are taken in sitemap order, so the cap truncates the tail of a very large sitemap.
- `checkExternalLinks` is not supported — this Actor never leaves the sitemap.
- A bot-protection wall may answer `403` to our requests; that is reported as a broken link with the real status, because it is what a crawler would see.

### Implementation note (shared network guard)

`src/lib/http.js` is a deliberate superset of the fleet's shared network kit: it adds `HEAD` requests, a bounded body read (8 KB per link check, 20 MB per sitemap) and a redirect hop chain. It keeps every safety check from the shared kit unchanged: IP literals are validated on every hop, and each connection resolves through the address-validating lookup. Its loopback test escape (`allowPrivateHost`) accepts only `127.x` / `::1`, applies to the first hop's origin only (redirect hops to any other origin are refused), and production code never sets it. The fleet audit gate tracks this file as a declared fork and fails if it ever loses those guard calls.

# Actor input Schema

## `sites` (type: `array`):

Domain names or URLs of sites you own or are authorised to audit, for example example.com. Their sitemap is discovered automatically (robots.txt Sitemap lines, then /sitemap.xml, then /sitemap_index.xml). IP addresses, localhost and private hosts are rejected.

## `maxUrlsPerSite` (type: `integer`):

Upper bound on URLs checked per site, taken in sitemap order (hard limit 5000).

## `maxSites` (type: `integer`):

Upper bound on sites audited in this run (hard limit 50).

## `includeOk` (type: `boolean`):

true (default): one row per checked URL. false: healthy OK rows are hidden. Every URL is still checked and you still receive every problem row and every free row (robots-disallowed, external-host-skipped, refused, invalid and error rows); you are charged only for fetched URLs delivered with issues.

## Actor input object example

```json
{
  "sites": [
    "www.sitemaps.org"
  ],
  "maxUrlsPerSite": 200,
  "maxSites": 5,
  "includeOk": true
}
```

# Actor output Schema

## `results` (type: `string`):

Every checked sitemap URL as dataset items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "www.sitemaps.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("devsolal/sitemap-link-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sites": ["www.sitemaps.org"] }

# Run the Actor and wait for it to finish
run = client.actor("devsolal/sitemap-link-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "www.sitemaps.org"
  ]
}' |
apify call devsolal/sitemap-link-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devsolal/sitemap-link-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1Ydevfa0D34yJXrBz/builds/Z3I3NxkHUw6MguPIC/openapi.json