# Broken Link Checker - 404s, Redirects & Slow Pages Site-Wide (`bruco3/broken-link-checker`) Actor

Check every URL a website publishes in its sitemaps: 404s, 5xx, timeouts, SSL errors, redirect chains and slow pages. No crawling needed, no proxy. Delta mode shows which problems are new since your last run and which got fixed. Pay per URL checked.

- **URL**: https://apify.com/bruco3/broken-link-checker.md
- **Developed by:** [Luca Pietrini](https://apify.com/bruco3) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.70 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker — find 404s, redirects and slow pages across a whole website

Paste a website and every URL it publishes in its **sitemap** gets checked: dead pages (404, 410), server errors (5xx), timeouts, expired SSL certificates, redirect chains and slow responses. No crawling, no proxy, no login — the sitemap already tells us what the site claims to publish, so a 10,000-page site is checked in minutes, not hours.

Or paste your own **list of URLs** (from a CMS export, a spreadsheet, another Actor) and check exactly those.

Three things set it apart from the other link checkers on Apify Store:

- **Sitemap-driven, not crawl-driven.** Crawlers miss pages that are not linked and waste time on pages that are. The sitemap is the site owner's own inventory; checking it answers the SEO question that matters: *does every URL I submit to Google actually work?*
- **Delta mode.** Turn it on and every run tells you which problems are **new** since your last run, which are **persisting**, and which got **fixed**. Schedule it weekly and the dataset is your change log.
- **Full redirect chains.** Every hop is recorded (`301 → 302 → 200`), so migration mistakes and redirect loops are visible, not hidden behind a final 200.

### What you get

By default only problems are reported, so the dataset stays small. One row per URL:

```json
{
  "url": "https://example.com/old-page",
  "status": 404,
  "category": "client-error",
  "ok": false,
  "finalUrl": null,
  "redirects": 0,
  "redirectChain": [],
  "responseMs": 212,
  "contentType": "text/html",
  "method": "HEAD",
  "error": null,
  "sitemap": "https://example.com/sitemap-pages.xml",
  "lastmod": "2026-08-30",
  "site": "https://example.com",
  "checkedAt": "2026-09-18T12:00:00+00:00",
  "change": "new"
}
```

| `category` | Meaning |
|---|---|
| `ok` | 2xx within the slow threshold, no redirect |
| `redirect` | Reached a 2xx only through one or more redirects (`redirectChain` lists every hop). Off by default? No — sitemap URLs should answer directly; turn off **Treat redirects as problems** if you disagree |
| `slow` | 2xx but slower than **Slow page threshold** (default 3 s) |
| `client-error` | 4xx — the classic broken link (404, 410, 403…) |
| `server-error` | 5xx |
| `timeout` | No response within the timeout |
| `connection` | DNS failure, refused connection, host unreachable |
| `ssl` | Certificate problem (expired, wrong host, untrusted) |
| `redirect-loop` / `too-many-redirects` | The chain never ends or exceeds **Maximum redirect hops** |
| `change` | Delta mode only: `new`, `persisting` or `fixed` |

Switch **What to report** to *All URLs* to also get the healthy pages with their status and response time — useful for a full site health export.

A run summary — URLs checked, problems by category, status-code histogram, per-site counts, delta totals, timing — is saved as the `SUMMARY` record of the run's key-value store.

### Input

| Option | Default | What it does |
|---|---|---|
| **Websites or sitemap URLs** | — | A domain (sitemap discovered via `robots.txt` and common paths) or a sitemap / sitemap index URL. Nested indexes, `.xml.gz` and text sitemaps are handled. |
| **Or: a list of URLs to check** | — | Exact URLs, one per line. Can be combined with sites. |
| **What to report** | problems only | Or all URLs. |
| **Maximum URLs to check** | `0` (no limit) | Exact cap across all sites. |
| **Check only / Skip URLs matching** | — | Case-insensitive regular expressions. |
| **Delta mode** | off | Flag problems as `new` / `persisting`, report `fixed` URLs. Snapshots live in a named key-value store, one per site. |
| **Treat redirects as problems** | on | See the `redirect` category above. |
| **Slow page threshold (ms)** | 3000 | |
| **Parallel checks** | 20 | Never more than 8 concurrent requests per host, whatever the total. |
| **Use GET instead of HEAD** | off | Each URL is probed with HEAD and falls back to GET when the server refuses HEAD (403/405/501). Bodies are never downloaded either way. |

### How the check works

1. Sitemaps are discovered and parsed **as a stream**: checks start while the sitemap files are still downloading, and memory stays flat on huge sites.
2. Each URL gets a **HEAD** request. Servers that reject HEAD get a streamed **GET** that is closed as soon as the headers arrive.
3. Redirects are followed **by hand**, one hop at a time, so the chain is recorded exactly.
4. Transient failures (timeouts, 429, 5xx) are retried once with back-off before being reported.
5. Requests are limited to **8 in flight per host** — polite by construction, fast across many hosts.

### Delta mode: new, persisting, fixed

The problem URLs of each site are remembered between runs. Next run:

- a problem not seen before → `new`
- a problem seen last time too → `persisting`
- a URL that was a problem last time and is healthy now → `fixed` (reported even in *problems only* mode, because that is exactly what you want to know)

A run cut short by **Maximum URLs** or by your maximum charge **does not overwrite the snapshot**, so a partial check can never make the next run report hundreds of false `fixed` URLs.

### Pricing

Pay per event: **one event per URL checked**. Reporting problems only or all URLs costs the same — the work is the same. Set **Maximum URLs** or a maximum charge on the run to cap the cost on very large sites; the Actor stops cleanly when the budget is reached and keeps everything already found.

### Use cases

- **SEO hygiene** — everything you submit to Google in your sitemap answers 200, directly. Find the 404s and redirect chains that waste crawl budget.
- **Site migrations** — run before and after; the `fixed` and `new` flags tell you exactly what the migration broke or repaired.
- **Weekly monitoring** — schedule it and pipe the dataset to Slack, email or a sheet through Apify integrations.
- **Link lists** — check the thousands of external links in your CMS or knowledge base from an exported list.
- **AI agents** — deterministic JSON, exact caps, no proxies, no login: safe to call programmatically and pay per event.

### Run it from code

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("bruco3/broken-link-checker").call(run_input={
    "startUrls": [{"url": "https://example.com"}],
    "deltaMode": True,
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["category"], row["status"], row["url"], row.get("change"))
```

### FAQ

**Why does it flag redirects?** A URL in a sitemap is a promise: "this page lives here". If it answers with a redirect, search engines follow it but count it against you, and the sitemap is stale. Turn **Treat redirects as problems** off if you only care about hard failures.

**Does it download the pages?** No. HEAD requests, or GET requests closed after the headers. Bandwidth is negligible and servers barely notice.

**HEAD says 404 but the page works in my browser.** Some servers answer HEAD incorrectly. Turn on **Use GET instead of HEAD** and the check uses GET for every URL.

**Can it crawl a site without a sitemap?** No, by design. That is a different tool with different costs and a different failure mode. Pair this Actor with a crawler if you need link discovery.

**Is it legal?** It requests only pages the site publishes for the public, identifies itself with a clear `User-Agent`, never downloads bodies, and never exceeds 8 concurrent requests per host.

### Related

- **[Sitemap URL Extractor](https://apify.com/bruco3/sitemap-url-extractor)** — the URL list behind this checker, with added/removed pages between runs.

### Feedback

A site this Actor cannot check, or a field you are missing? Open an issue on the **Issues** tab. If it saved you time, a review helps others find it.

### Changelog

- **0.1** — initial release: sitemap-driven and list-driven checks, HEAD with GET fallback, full redirect chains, slow/SSL/timeout detection, delta mode with `new` / `persisting` / `fixed`, exact caps, pay per URL checked.

# Actor input Schema

## `startUrls` (type: `array`):

Paste a website (<code>https://example.com</code>) and every URL in its sitemaps is checked. Or paste a sitemap / sitemap index URL directly. No crawling: the sitemap is the source of truth for what the site says it publishes.

## `urls` (type: `array`):

Check these exact URLs instead of (or in addition to) a site's sitemap. One URL per line, or upload a text file. Handy for link lists exported from a CMS, a spreadsheet or another Actor.

## `report` (type: `string`):

<b>Problems only</b> keeps the dataset small: 4xx, 5xx, timeouts, SSL errors, redirects and slow pages. <b>All URLs</b> also includes healthy pages with their status and response time.

## `maxUrls` (type: `integer`):

Stop after this many checks across all sites. <code>0</code> = no limit.

## `includeRegex` (type: `string`):

Case-insensitive regular expression, e.g. <code>/blog/</code>.

## `excludeRegex` (type: `string`):

Case-insensitive regular expression applied after the include filter, e.g. <code>.pdf$|/tag/</code>.

## `deltaMode` (type: `boolean`):

Remembers the problem URLs of each site in a named key-value store. Every reported problem is flagged <code>new</code> or <code>persisting</code>, and URLs that were broken last time but are healthy now are reported as <code>fixed</code>.

## `deltaStoreName` (type: `string`):

Named key-value store that keeps one snapshot of problem URLs per site.

## `reportRedirects` (type: `boolean`):

A URL listed in the sitemap should answer 200 directly. When on, 301/302/307/308 chains are reported with every hop.

## `slowMs` (type: `integer`):

Healthy pages slower than this are reported as <code>slow</code>.

## `concurrency` (type: `integer`):

Total concurrent requests. Never more than 8 at a time per host, regardless of this value.

## `maxRedirects` (type: `integer`):

Chains longer than this are reported as <code>too-many-redirects</code>.

## `alwaysGet` (type: `boolean`):

By default each URL gets a HEAD request, with an automatic GET fallback when the server refuses HEAD. Force GET for servers that answer HEAD incorrectly. Bodies are never downloaded.

## `maxSitemaps` (type: `integer`):

Safeguard against runaway sitemap indexes.

## `maxDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow.

## `userAgent` (type: `string`):

Sent with every request. Change it only if a site requires a specific value.

## `proxyConfiguration` (type: `object`):

Not needed for your own sites. Enable only if a site blocks datacenter traffic.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "report": "problems",
  "maxUrls": 0,
  "deltaMode": false,
  "deltaStoreName": "broken-link-checker-snapshots",
  "reportRedirects": true,
  "slowMs": 3000,
  "concurrency": 20,
  "maxRedirects": 10,
  "alwaysGet": false,
  "maxSitemaps": 2000,
  "maxDepth": 5,
  "userAgent": "BrokenLinkChecker/1.0 (+https://apify.com/bruco3/broken-link-checker)",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("bruco3/broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("bruco3/broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ]
}' |
apify call bruco3/broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bruco3/broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/85scuOI9Wni99YRm9/builds/ADCQCjC9tdyXvIlny/openapi.json