# Bulk Broken Link Checker - Sitemap & Redirect Chains (`obligate_castle/bulk-broken-link-checker`) Actor

Find broken links on any website in bulk. Give the Actor a domain, a page URL or a sitemap.xml: it explores the pages, collects every link they publish, and tests each one. You get one row per unique link with the HTTP status, a clear category (ok, redirected, not-found, server-error, timeout,...

- **URL**: https://apify.com/obligate\_castle/bulk-broken-link-checker.md
- **Developed by:** [Yoann TUQUET](https://apify.com/obligate_castle) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Bulk Broken Link Checker – find 404s, redirect chains and dead links from a sitemap or URL

Find **broken links** on any website in bulk. Give the Actor a domain, a page URL or a **sitemap.xml**: it explores the pages, collects every link they publish, and tests each one. You get one row per unique link with the **HTTP status**, a clear **category** (ok, redirected, not-found, server-error, timeout, dns-error, ssl-error, redirect-loop...), the **full redirect chain**, the **final URL** and the **page where the link was found**, so you know exactly what to fix.

Only links that pages (or sitemaps) publish themselves are tested. The Actor never guesses paths, never scans a site for hidden files and reads no personal data: it uses plain HTTP HEAD and GET requests, exactly like a browser or a search engine would.

### Why use it

- ✅ **Starts from what you have**: a domain (the sitemap is found through robots.txt), a direct sitemap URL (indexes and .gz files are followed), or a single page URL.
- ✅ **Tells you where to fix it**: every row carries `sourcePage` (first page containing the link) and `foundOnPages` (how many explored pages contain it).
- ✅ **Full redirect chains**: each hop with its status code (`301`, `302`, `307`, `308`), redirect loops and chains that are too long are flagged.
- ✅ **Honest classification**: a 404 is broken; a 403 or 429 from a bot filter is reported as `access-restricted` with `isBroken: null` instead of a false alarm. Transient errors (timeouts, connection errors, 5xx) are retried once.
- ✅ **Cheap and fast**: HEAD requests first, GET only when a server refuses HEAD. Duplicates are removed across pages and across all sites of a run.
- ✅ **Polite by design**: respects robots.txt, at most 2 requests per second to any domain, and never requests private or local network addresses.
- ✅ **Pay only for results**: one charge per link actually tested. Links skipped because of robots.txt, invalid inputs and duplicates are not charged, and **Only return broken links** lets you pay only for the problems.

### Use cases

1. **SEO audits**: find internal 404s, long redirect chains and sitemap entries that no longer resolve before they cost you rankings.
2. **Site migrations**: after a redesign or a domain change, list every old URL that now redirects through several hops or ends in an error.
3. **Content and documentation teams**: schedule a weekly run on your docs or blog and get only the broken links (external ones included) to fix.
4. **Agencies and freelancers**: check dozens of client sites in one run and export a CSV report per site.
5. **Link building and PR**: verify that outbound links you pay for or exchange still work and do not redirect somewhere unexpected.

### How to use it

1. Click **Try for free** and paste domains, page URLs or sitemap URLs into **Websites, pages or sitemap URLs**.
2. Set **Max pages explored per site** and **Max links checked per site** to control the size (and cost) of the run.
3. Click **Start**. Export the results as **JSON, CSV, Excel or HTML**, or fetch them through the Apify API.

Tip: turn on **Only return broken links** and create a **Schedule**: you only receive, and pay for, new problems.

### What is checked

| Area | Details |
|---|---|
| Pages explored | Pages listed in the sitemap (up to **Max pages explored per site**), the home page, or the page URL you give |
| Links tested | Clickable links (`<a>`, `<area>`) of those pages; optionally images, scripts and stylesheets |
| Sitemap entries | Every URL listed in the sitemap is also tested (can be turned off) |
| Status | HTTP status of the final URL after following up to 20 redirects |
| Failures | Timeout, DNS error, SSL/certificate error, connection error, redirect loop, too many redirects |
| Skipped links | `mailto:`, `tel:`, `javascript:` and `#anchors` are ignored |

Categories: `ok`, `redirected`, `not-found` (404 or 410), `client-error` (other 4xx), `server-error` (5xx), `access-restricted` (401, 403, 407, 429, 999), `timeout`, `dns-error`, `ssl-error`, `connection-error`, `redirect-loop`, `too-many-redirects`, `bad-redirect`, `robots-blocked`, `blocked-address`, `other`.

### Input

```json
{
  "items": ["www.sitemaps.org", "https://example.com/blog", "https://example.org/sitemap.xml"],
  "maxPagesPerSite": 50,
  "maxLinksPerSite": 500,
  "onlyBroken": true,
  "linkScope": "all",
  "excludePattern": "linkedin\\.com|/logout"
}
```

| Field | Description | Default |
|---|---|---|
| `items` | Domains, page URLs or sitemap URLs | required |
| `maxPagesPerSite` | Pages downloaded per site to collect links | `50` |
| `maxLinksPerSite` | Unique links tested per input (caps the cost per site) | `500` |
| `onlyBroken` | Return only broken links | `false` |
| `linkScope` | `all`, `internal` or `external` links | `all` |
| `checkSitemapUrls` | Also test the URLs listed in the sitemap | `true` |
| `includeAssets` | Also test images, scripts and stylesheets | `false` |
| `excludePattern` | Regex: links matching it are not tested | none |
| `maxRedirects` | Redirects followed before `too-many-redirects` | `10` |
| `timeoutSecs` | Timeout per request | `15` |

### Output

One dataset item per unique link (real example from a test run):

```json
{
  "url": "http://www.w3.org/TR/REC-xml/",
  "finalUrl": "https://www.w3.org/TR/REC-xml/",
  "status": 200,
  "category": "redirected",
  "isBroken": false,
  "redirectCount": 1,
  "redirectChain": [{ "url": "http://www.w3.org/TR/REC-xml/", "status": 301 }],
  "sourcePage": "https://www.sitemaps.org/protocol.html",
  "foundOnPages": 1,
  "linkType": "anchor",
  "internal": false,
  "method": "HEAD",
  "responseMs": 56,
  "error": null,
  "site": "sitemaps.org",
  "checkedAt": "2026-09-18T17:27:44.158Z"
}
```

`isBroken` is `true` for a broken link, `false` for a working one, and `null` when it could not be verified (`access-restricted`, `robots-blocked`, `blocked-address`). A `SUMMARY` record in the run's key-value store gives totals per category and, for each input, the pages explored and the reason when something stopped the run (no sitemap, limit reached).

### Pricing

Pay per event: **$0.001 per link tested**, which is **$1 per 1,000 links**. Links that were not requested (blocked by robots.txt or pointing to a non-public address), invalid inputs and duplicate links are not charged. Cap your spending with the maximum charge setting of each run, with **Max links checked per site**, or with **Only return broken links**.

### Integrations

- **API**: start runs and download results from any language with the Apify API or clients.
- **Schedules and webhooks**: run weekly and trigger a webhook when a run finishes.
- **Make, Zapier, n8n**: send broken links to Slack, Google Sheets, Jira or your ticketing tool.
- **AI agents**: callable as a tool through the Apify MCP server.

### FAQ

**Does it respect robots.txt?**
Yes. Pages and links disallowed for crawlers are not requested: they appear with the category `robots-blocked` and are not charged. Requests are limited to 2 per second per domain, robots.txt of every checked domain included.

**Why do some links show `access-restricted`?**
Many sites answer 403, 429 or 999 to automated requests. The Actor does not try to get around such protections, so it reports the link as unverified (`isBroken: null`) instead of calling it broken.

**What if a site has no sitemap?**
Only the home page is explored and the run says so in the `SUMMARY` record. Give page URLs (or a sitemap URL) to explore more.

**Does it crawl the whole site?**
No. It explores the pages listed in the sitemap (up to **Max pages explored per site**) and tests the links found on them. It does not follow links to discover new pages.

**Are links in JavaScript-rendered content found?**
No. Pages are read as plain HTML, so links added by JavaScript after loading are not seen.

**Is the same link reported twice?**
No. Each unique URL is tested and billed once per run, even if it appears on many pages or on several of your sites; `foundOnPages` tells you how many pages contain it.

### Limitations

- Only links present in the downloaded HTML are tested; pages that need a login or JavaScript rendering are not covered.
- Anchors (`#section`) are not verified, only the page they belong to.
- Servers that block automated requests cannot be verified; the Actor does not bypass protections.
- Pages above about 3 MB are read only in part, and sitemap files above about 60 MB uncompressed are skipped.
- Local, private and link-local network addresses are never requested.

# Actor input Schema

## `items` (type: `array`):

Domains (example.com), page URLs (https://example.com/blog) or direct sitemap URLs (https://example.com/sitemap.xml). For a domain, the sitemap is discovered from robots.txt (or the usual sitemap paths) and its pages are explored; without a sitemap, only the home page is explored. For a page URL, only that page is explored. Every link published by the explored pages is then checked.

## `maxPagesPerSite` (type: `integer`):

How many pages of each site are downloaded to collect their links (taken from the sitemap, in order).

## `maxLinksPerSite` (type: `integer`):

Stop after this many unique links per input. Each link tested is one billed result, so this caps your cost per site.

## `onlyBroken` (type: `boolean`):

Return (and pay for) only the links that are broken: 404 or 410, other 4xx, 5xx, timeouts, DNS or SSL errors, redirect loops. Healthy links are still checked but not returned.

## `linkScope` (type: `string`):

Check internal links (same site), external links (other sites), or both.

## `checkSitemapUrls` (type: `boolean`):

Report the status of every page listed in the sitemap, not only the links found inside pages. Useful to spot sitemap entries that return 404 or redirect.

## `includeAssets` (type: `boolean`):

Also test the files referenced by img, script and stylesheet tags. Off by default: only clickable links are checked.

## `excludePattern` (type: `string`):

Optional regular expression. Links whose URL matches it are not checked, for example linkedin.com|/logout.

## `maxRedirects` (type: `integer`):

A link that redirects more often than this is reported as too-many-redirects.

## `timeoutSecs` (type: `integer`):

Network timeout for each request. A link that does not answer in time is reported as timeout (after one retry).

## Actor input object example

```json
{
  "items": [
    "www.sitemaps.org"
  ],
  "maxPagesPerSite": 10,
  "maxLinksPerSite": 100,
  "onlyBroken": false,
  "linkScope": "all",
  "checkSitemapUrls": true,
  "includeAssets": false,
  "maxRedirects": 10,
  "timeoutSecs": 15
}
```

# Actor output Schema

## `overview` (type: `string`):

Main fields for each link.

## `results` (type: `string`):

Complete result items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        "www.sitemaps.org"
    ],
    "maxPagesPerSite": 10,
    "maxLinksPerSite": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("obligate_castle/bulk-broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "items": ["www.sitemaps.org"],
    "maxPagesPerSite": 10,
    "maxLinksPerSite": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("obligate_castle/bulk-broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    "www.sitemaps.org"
  ],
  "maxPagesPerSite": 10,
  "maxLinksPerSite": 100
}' |
apify call obligate_castle/bulk-broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,obligate_castle/bulk-broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dpjY3WuUMnLoQsTgT/builds/jgLVf6Iv8FIDxd3Vi/openapi.json