# Broken Link & Sitemap Health Checker (`jeremydonaldsonventure/broken-link-sitemap-health-check`) Actor

Finds broken links, redirect chains and loops, sitemap entries that fail, and pages missing from the sitemap. Crawls your site or reads its sitemap. Plain HTTP, obeys robots.txt. Pay per link checked.

- **URL**: https://apify.com/jeremydonaldsonventure/broken-link-sitemap-health-check.md
- **Developed by:** [Jeremy DONALDSON](https://apify.com/jeremydonaldsonventure) (community)
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 link checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Broken Link & Sitemap Health Checker do?

**Broken Link & Sitemap Health Checker** finds every **broken link** on a website (404s, server errors, dead domains, timeouts), every **redirect chain and redirect loop**, every **sitemap entry that fails**, and every **page missing from the sitemap**. Give it a home page. It reads the site's `robots.txt` and `sitemap.xml`, crawls the site's own pages, and checks each link once, **including links to other websites**.

It uses plain HTTP requests, no browser, so it is **fast and cheap**. You **pay per address checked**. Addresses the site's `robots.txt` blocks are listed free and never requested.

To try it, paste your home page into **Website, sitemap or links** and press **Start**.

### Why use it?

- 🔗 **Broken links with where they are.** Each broken link lists the pages that link to it and the link text, so you know exactly what to fix.
- 🗺️ **Sitemap health, not just links.** Flags sitemap entries that are broken, redirect, are `noindex`, point their canonical elsewhere, or are blocked by `robots.txt`. Checks each sitemap file too: missing, not XML, too big, or listing other sites.
- 🔍 **Pages missing from the sitemap.** Compares what the crawl finds with what the sitemap lists, and names the indexable pages Google may never be told about. With a full crawl it also names sitemap pages nothing links to.
- ↪️ **Every redirect hop.** Full redirect chains with each status code, loops caught, and same-site links that point at a redirect instead of the final page.
- 🤝 **Polite by default.** Obeys `robots.txt` and its `Crawl-delay`, spaces requests to each site, and never has more than two open at once on one site.
- 💸 **Pay only for checks.** Each unique address is checked and charged once, however many pages link to it.
- 🤖 **Agent-ready.** Works as a tool in the Apify MCP server, so AI assistants such as Claude or ChatGPT can run it.
- ⏱️ Everything the Apify platform gives you: **schedules** (a weekly link check), **API**, **webhooks**, and exports to **JSON, CSV, Excel** or **Google Sheets**.

### Who is it for?

- **SEO and web agencies**: technical audits, client reports, and checks after a site migration.
- **Site owners, editors and developers**: catch dead links before visitors and search engines do.
- **Anyone moving a site**: run it before and after to catch broken redirects.

### How to check a website for broken links

1. Paste the home page (or its `sitemap.xml`) into **Website, sitemap or links**.
2. Leave **What to check** on **Sitemap + crawl** for the full health check. Pick **Sitemap only** to audit the sitemap, or **Just these addresses** to check a list.
3. Set **Most pages to read** and **Most addresses to check** to the size of the site.
4. Press **Start**. A few hundred links take one to two minutes.
5. Open the **Output** tab: **All links** for the overview, **Where each link is** for the pages to fix, **Sitemap and indexing** for the sitemap view, and **Site health summary** for the roll-up.

### Modes

| Mode | Reads the sitemap | Crawls the site | Finds |
|---|---|---|---|
| **Sitemap + crawl** (default) | yes | yes | Everything: broken links, redirects, failing sitemap entries, pages missing from the sitemap, sitemap pages nothing links to |
| **Crawl only** | no | yes | Broken links and redirects on pages reachable from the start page |
| **Sitemap only** | yes | no | Sitemap entries that are broken, redirect, are noindex or canonicalised elsewhere |
| **Just these addresses** | no | no | Status and redirects for the list you give |

### How much does it cost to check links?

You pay **per link checked**: one unique address, checked once.

| | Price |
|---|---|
| Link checked | **$0.0007** each ($0.70 per 1,000), lower on Apify's Silver and Gold plans |
| Run start | Apify's standard tiny start fee |
| Addresses blocked by robots.txt, invalid addresses, the sitemap files themselves | **Free** |

Examples: a 300-link site is about **$0.21**. A 5,000-link site is about **$3.50**. Apify's free plan includes monthly credit, enough for many test runs.

**Most addresses to check** caps the charge for a run. You can also set a **maximum cost per run** when you start it: the Actor stops cleanly before it would go over, and you keep every result it finished.

### Input example

```json
{
    "startUrls": ["https://crawlee.dev"],
    "mode": "full",
    "maxPages": 60,
    "maxLinksToCheck": 600,
    "checkExternalLinks": true
}
```

### Output example

One row per address checked. A broken stylesheet found on five pages:

```json
{
    "url": "https://lipis.github.io/flag-icon-css/css/flag-icon.css",
    "httpStatus": 404,
    "result": "broken",
    "isBroken": true,
    "issues": ["BROKEN_LINK"],
    "linkType": "stylesheet",
    "internal": false,
    "linkedFromCount": 5,
    "linkedFrom": [
        { "page": "https://www.scrapethissite.com/pages/simple/", "anchorText": null }
    ],
    "charged": true
}
```

A same-site link that redirects and then fails:

```json
{
    "url": "https://www.crawlee.dev/python/docs/examples/playwright-crawler-with-fingeprint-generator",
    "finalUrl": "https://crawlee.dev/python/docs/examples/playwright-crawler-with-fingeprint-generator",
    "httpStatus": 404,
    "result": "broken",
    "redirectCount": 1,
    "redirectChain": [
        { "url": "https://www.crawlee.dev/python/docs/examples/playwright-crawler-with-fingeprint-generator", "status": 301 }
    ],
    "linkedFrom": [
        { "page": "https://crawlee.dev/blog/crawlee-for-python-v1", "anchorText": "Playwright crawler with fingerprint generator" }
    ]
}
```

An address the site's robots.txt blocks comes back free, never requested:

```json
{ "url": "https://www.scrapethissite.com/lessons/", "result": "skipped-robots", "charged": false, "errorCode": "BLOCKED_BY_ROBOTS_TXT" }
```

The **Site health summary** (key-value store record `OUTPUT`) holds counts by result and issue, the 50 broken links linked from the most pages, the longest redirect chains, each sitemap file's health, the failing sitemap entries, and the pages missing from the sitemap.

### Results and issue codes

| `result` | Meaning | Broken? |
|---|---|---|
| `ok` | 2xx, no redirect | no |
| `redirect` / `redirect-chain` | One hop / two or more hops to a working page | no |
| `broken` | 4xx (404, 410 and others) | yes |
| `blocked` | 401, 403, 429 or 999: the site refuses bots. Usually works in a browser; check by hand | no |
| `server-error`, `timeout`, `dns-error`, `connection-error`, `tls-error` | No usable answer | yes |
| `redirect-loop`, `too-many-redirects` | The redirects never end | yes |
| `skipped-robots` | Blocked by the site's robots.txt. Not requested, not charged | no |

| Issue code | Meaning |
|---|---|
| `BROKEN_LINK`, `SERVER_ERROR`, `TIMEOUT`, `DNS_ERROR`, `CONNECTION_ERROR`, `TLS_ERROR`, `REDIRECT_LOOP`, `TOO_MANY_REDIRECTS` | The link doesn't work |
| `REDIRECT_CHAIN` | Two or more redirects; link to the final address |
| `INTERNAL_LINK_TO_REDIRECT` | Your own page links to an address that redirects |
| `BLOCKED_FOR_BOTS` | The server refused an automated check |
| `SITEMAP_ENTRY_BROKEN`, `SITEMAP_ENTRY_REDIRECTS` | The sitemap lists an address that fails or redirects |
| `SITEMAP_ENTRY_NOINDEX`, `SITEMAP_ENTRY_CANONICAL_ELSEWHERE` | The sitemap lists a page that asks not to be indexed, or names another page as canonical |
| `SITEMAP_ENTRY_BLOCKED_BY_ROBOTS_TXT` | The sitemap lists an address robots.txt blocks |
| `SITEMAP_ENTRY_NOT_LINKED` | In the sitemap, but no crawled page links to it (only after a complete crawl) |
| `MISSING_FROM_SITEMAP` | An indexable same-site page the crawl found that the sitemap doesn't list (addresses with `?` query strings are left out) |

### FAQ

**Why are some links "blocked" when they work in my browser?**
Sites such as LinkedIn, Stack Overflow and some shops refuse automated requests (403, 429 or 999). The Actor reports those as `blocked`, not broken, so you can check them by hand.

**Why are some links "skipped-robots"?**
The site's `robots.txt` asks bots not to load them. The Actor respects that for every site, yours and the ones you link to. Those addresses are free.

**My site has a huge sitemap. Will it crowd out the crawl?**
No. In **Sitemap + crawl** mode the sitemap gets half of **Most addresses to check** and **Most pages to read** up front. The crawl gets the rest, and leftover sitemap entries fill any room at the end.

**Can it check pages behind a login, or pages built by JavaScript?**
No. It checks public addresses over plain HTTP. Links that only appear after JavaScript runs are not seen.

**Does it respect robots.txt?**
Yes, including `Crawl-delay` (up to 10 seconds). Requests to one site start at least 250 ms apart by default, and never more than two run at once on one site.

**Is it legal?**
It requests each public address once, as a browser would, and reads the response. Check sites you own, manage, or have permission to test.

**Can I run it on a schedule?**
Yes. Save your input as a task and add an Apify schedule, for example every Monday. Add a webhook or the Google Sheets integration to get the broken links where you work.

**Related tools**
Pair it with **Bulk Lighthouse Site Audit** (same developer) for speed, Core Web Vitals, accessibility and SEO scores on every page.

**Something went wrong.**
Open an issue on the Actor's **Issues** tab with the run link.

# Actor input Schema

## `startUrls` (type: `array`):

A home page such as https://example.com, or a sitemap such as https://example.com/sitemap.xml. In 'Just these addresses' mode, the list of links to check. https:// is added if you leave it off.

## `mode` (type: `string`):

'Sitemap + crawl' finds the most: broken links, failing sitemap entries, and pages missing from the sitemap. 'Crawl only' follows links from the start page. 'Sitemap only' checks each sitemap entry. 'Just these addresses' checks the list you give and nothing else.

## `maxPages` (type: `integer`):

How many of the site's own pages to open and read for links. Links found on them are still checked once the cap is reached; they just aren't read in turn.

## `maxLinksToCheck` (type: `integer`):

Cap on unique addresses checked per run, and so on the charge. Sitemap entries count first, then links in the order found. Your run's spending limit also stops the run.

## `checkExternalLinks` (type: `boolean`):

Checks that links to other sites still work. They are checked, never crawled.

## `checkResources` (type: `boolean`):

Checks the files each page loads as well as its links. Each file is one more address checked.

## `onlyReportProblems` (type: `boolean`):

Leaves healthy links out of the results. They are still checked and charged; the summary still counts them.

## `requestDelayMs` (type: `integer`):

Starts requests to the same site at least this far apart. A robots.txt Crawl-delay is used instead when it is longer (up to 10 s). Other sites are checked in parallel.

## `maxConcurrency` (type: `integer`):

Requests in flight at once across all sites. Never more than two to the same site.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a server before calling the link a timeout. Timeouts get one retry.

## `maxRedirects` (type: `integer`):

A link that redirects more times than this is reported as too many redirects.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "mode": "full",
  "maxPages": 10,
  "maxLinksToCheck": 50,
  "checkExternalLinks": true,
  "checkResources": false,
  "onlyReportProblems": false,
  "requestDelayMs": 250,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 15,
  "maxRedirects": 10
}
```

# Actor output Schema

## `results` (type: `string`):

One row per unique address: HTTP status, result (ok, redirect, broken, ...), redirect chain, issue codes, sitemap status and the pages that link to it. Rows with charged false were free.

## `summary` (type: `string`):

Counts by result and issue, the broken links linked from the most pages, the longest redirect chains, each sitemap file's health, failing sitemap entries and pages missing from the sitemap.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxPages": 10,
    "maxLinksToCheck": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("jeremydonaldsonventure/broken-link-sitemap-health-check").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxPages": 10,
    "maxLinksToCheck": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("jeremydonaldsonventure/broken-link-sitemap-health-check").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxPages": 10,
  "maxLinksToCheck": 50
}' |
apify call jeremydonaldsonventure/broken-link-sitemap-health-check --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jeremydonaldsonventure/broken-link-sitemap-health-check"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XwNcRuW6yyTZFAWd1/builds/OKMgZfpoL8pdzsNdS/openapi.json