# Broken Link Checker & Sitemap Extractor (SEO, robots.txt) (`chesapeake/broken-link-sitemap-checker`) Actor

Find broken links (404, 5xx, timeouts), redirect chains and broken images on any website, or extract every URL from its XML sitemaps. Includes robots.txt and AI-bot blocking summary. Fast, polite, pay per URL.

- **URL**: https://apify.com/chesapeake/broken-link-sitemap-checker.md
- **Developed by:** [Chesapeake Commerce](https://apify.com/chesapeake) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 link checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker & Sitemap Extractor (SEO, robots.txt)

Find **broken links** (404, 410, 5xx, DNS errors, timeouts), **redirect chains** and **broken images** across any website, or extract **every URL from its XML sitemaps** (sitemap indexes and `.gz` included). Each run also returns a **robots.txt summary**: declared sitemaps, disallow rules, and which **AI crawlers** (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot...) are blocked.

It uses plain HTTP, so it's fast and cheap. There's no browser and no proxy. You pay **per URL checked**, with no subscription.

### Why use it

- **Two tools in one.** A full-site broken link checker plus a sitemap extractor/validator.
- **Shows where each broken link lives.** Every result lists up to 5 pages it was **found on**, with the **anchor text**, so you can fix it in minutes.
- **Low false-alarm rate.** Each URL gets a HEAD request with GET fallback, and 429 responses are retried after `Retry-After`. Bot walls (401/403/429/999 from LinkedIn, Stack Overflow, npm...) are reported as `RESTRICTED`, not as broken.
- **Redirect audit.** You get `finalUrl`, `redirectCount` and the full `redirectChain`, which helps find redirect chains and http→https hops.
- **Sitemap health.** It lists every `<loc>` with `lastmod`, `changefreq` and `priority`. Optionally it checks the HTTP status of every sitemap URL, so you can catch 404s and redirects that waste crawl budget.
- **Polite by design.** It respects robots.txt and `Crawl-delay` and caps per-host concurrency. URLs blocked by robots.txt are skipped and **not charged**.
- **Export anywhere:** JSON, CSV, Excel, Google Sheets, API, Make, Zapier or n8n. Schedule it weekly to monitor your site.

### How to use it

1. Enter one or more websites under **Websites**, e.g. `https://example.com`.
2. Choose a **Mode**:
   - **Broken link checker** crawls up to *Max pages* internal pages and checks every unique link and image on them, both internal and external.
   - **Sitemap extractor** reads `robots.txt` `Sitemap:` lines, or `/sitemap.xml` and `/sitemap_index.xml`, follows nested indexes and lists all URLs.
3. Optional: turn on **Only output broken links**, or **Check HTTP status of sitemap URLs**.
4. Click **Start**. Use the **Broken links only** view, or filter `linkStatus = BROKEN`.

#### Input example

```json
{
  "startUrls": [{ "url": "https://crawlee.dev" }],
  "mode": "brokenLinks",
  "maxPagesPerSite": 50,
  "maxUrlsPerSite": 3000,
  "checkExternalLinks": true,
  "checkImages": true
}
```

#### Output example (broken link)

```json
{
  "type": "LINK",
  "site": "https://crawlee.dev",
  "url": "https://github.com//github.com/apify/crawlee/pull/1890/issues",
  "statusCode": 404,
  "ok": false,
  "linkStatus": "BROKEN",
  "finalUrl": "https://github.com//github.com/apify/crawlee/pull/1890/issues",
  "redirectCount": 0,
  "contentType": "text/html",
  "responseMs": 212,
  "linkType": "external",
  "isImage": false,
  "foundOn": [{ "page": "https://crawlee.dev/js/api/changelog", "text": "#1890", "kind": "link" }],
  "linkText": "#1890"
}
```

#### Output example (sitemap URL)

```json
{ "type": "SITEMAP_URL", "site": "https://crawlee.dev", "url": "https://crawlee.dev/blog", "lastmod": null, "changefreq": "weekly", "priority": "0.5", "sitemap": "https://crawlee.dev/sitemap.xml" }
```

Every site also gets one `SITE_SUMMARY` item. It covers robots.txt (`sitemapsDeclared`, `disallowRulesForAll`, `aiBotsBlocked`, `blocksAllBotsCompletely`) and totals (`pagesCrawled`, `urlsChecked`, `brokenCount`, `brokenInternal`, `brokenExternal`, `restrictedCount`, `redirectedCount`, `sitemapUrlsFound`).

`linkStatus` values: `OK`, `BROKEN` (4xx/5xx, DNS error, timeout), `RESTRICTED` (401/403/429/999: the server blocks bots, though the link usually works in a browser), and `SKIPPED_ROBOTS_TXT`.

### Pricing (pay per event)

| Event | Price |
|---|---|
| Link checked (any URL requested and status-checked, including crawled pages) | **$0.0003** ($0.30 per 1,000) |
| Sitemap URL extracted (sitemap mode without status check) | **$0.0001** ($0.10 per 1,000) |

Example: a 100-page site with 2,500 unique links costs **$0.75** (2,500 × $0.0003). Extracting a 10,000-URL sitemap costs **$1.00**. URLs skipped because of robots.txt are free. Use **Max URLs per site** to cap your spend.

### FAQ

**Does it render JavaScript?** No. It reads the HTML your server returns, which is what search engines see first. Links that are injected only by client-side JavaScript won't be found.

**Why is a working link marked RESTRICTED?** Some sites (LinkedIn, Stack Overflow, npm, Cloudflare-protected sites) refuse automated requests. We don't bypass bot protection, so those links get their own status instead of being flagged as broken.

**Does it follow nofollow links?** It checks them, but it doesn't crawl further from pages marked `<meta name="robots" content="nofollow">`.

**Can I check only my sitemap for 404s?** Yes. Use Sitemap mode with **Check HTTP status of sitemap URLs** on, and **Only output broken links** on.

**Can I run it on a schedule?** Yes. Create an Apify Schedule (for example weekly) and send results to Slack, email or Google Sheets through an integration.

# Actor input Schema

## `startUrls` (type: `array`):

Homepages or start pages of the sites to check, e.g. https://example.com.

## `mode` (type: `string`):

Both modes also return a robots.txt summary per site.

## `maxPagesPerSite` (type: `integer`):

Broken-link mode: how many internal pages are crawled for links.

## `maxUrlsPerSite` (type: `integer`):

Cap on checked links (broken-link mode) or extracted sitemap URLs (sitemap mode). Protects your budget.

## `checkExternalLinks` (type: `boolean`):

Also check links that point to other websites.

## `checkImages` (type: `boolean`):

Also check <img> sources for broken images.

## `onlyReportBroken` (type: `boolean`):

Output only BROKEN links (404/410/5xx, DNS errors, timeouts). 401/403/429 responses are reported as RESTRICTED (the server blocks bots, the link usually works in a browser). All checked URLs are still billed.

## `checkSitemapUrlStatus` (type: `boolean`):

Sitemap mode: also request every URL to find 404s and redirects in your sitemap (billed as link checks instead of sitemap URLs).

## `maxConcurrencyPerHost` (type: `integer`):

Be polite to servers. robots.txt Crawl-delay is always honored.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed for automated agents (not charged).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "mode": "brokenLinks",
  "maxPagesPerSite": 5,
  "maxUrlsPerSite": 300,
  "checkExternalLinks": true,
  "checkImages": true,
  "onlyReportBroken": false,
  "checkSitemapUrlStatus": false,
  "maxConcurrencyPerHost": 3,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `results` (type: `string`):

Every checked link (status code, redirects, pages it was found on) or sitemap URL (lastmod, priority), plus per-site summaries.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "maxPagesPerSite": 5,
    "maxUrlsPerSite": 300
};

// Run the Actor and wait for it to finish
const run = await client.actor("chesapeake/broken-link-sitemap-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "maxPagesPerSite": 5,
    "maxUrlsPerSite": 300,
}

# Run the Actor and wait for it to finish
run = client.actor("chesapeake/broken-link-sitemap-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxPagesPerSite": 5,
  "maxUrlsPerSite": 300
}' |
apify call chesapeake/broken-link-sitemap-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,chesapeake/broken-link-sitemap-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Bv1V4HfyGDZliedEY/builds/4hTLOhVGiX0Diy9Ff/openapi.json