# Broken Link & Redirect Checker (`madpai/broken-link-checker`) Actor

Find broken links, redirect chains, redirect loops and soft-404s on your website. Seeds pages from sitemaps, checks internal and external links politely (robots.txt-aware, about 1 request per second per host) and lists which page links to each problem. Clean JSON for SEO audits and AI agents.

- **URL**: https://apify.com/madpai/broken-link-checker.md
- **Developed by:** [William Schultz](https://apify.com/madpai) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.80 / 1,000 link checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link & Redirect Checker

### What does this broken link checker do?

Broken Link & Redirect Checker crawls your website, collects every link it finds, and checks each one. For every link it reports the HTTP status, the full redirect chain (each hop, permanent or temporary), redirect loops, likely soft-404s, and the pages on your site that link to it, so you know exactly where to fix it. It can also seed pages from your XML sitemap, so pages with no inbound links get checked too.

### Who is it for?

SEO teams cleaning up 404s and redirect chains, site owners after a migration or redesign, agencies running link audits, and AI agents that need a structured list of a site's link problems.

### How it stays polite

- It only crawls the sites you list. Links to other sites are checked once each and never crawled.
- About 1 request per second per host (longer if robots.txt sets a Crawl-delay), and it backs off on HTTP 429 and Retry-After. A host that answers 429 three times in a row gets no further requests in that run.
- robots.txt is respected. Blocked links are reported as `blocked-by-robots` and aren't charged. If a host's robots.txt can't be reached or returns a server error, the whole host is treated as off-limits (RFC 9309), so its links are `blocked-by-robots` too. For your own sites only, `ownSiteConfirmed` lets you turn robots.txt off or go faster; external hosts always keep the 1-second limit and their robots.txt.
- It stores no page content, only the status codes and the redirect information. `mailto:` and `tel:` links are skipped, so no email addresses or phone numbers end up in the results.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` (required) | prefill `crawlee.dev` | Your site(s): domains or start pages. |
| `seedFromSitemap` | true | Also crawl pages listed in the site's XML sitemap. |
| `maxPagesPerSite` | 50 (prefill 5) | Pages per site fetched and scanned for links. |
| `maxLinks` | 1000 (prefill 100) | Cap on unique links checked. You pay per link, so this also caps cost. |
| `checkExternalLinks` | true | Check outbound links too (one request each). |
| `checkAssets` | false | Also check images, scripts, stylesheets and icons. |
| `includeRegex` | none | Report only links that match, e.g. `/docs/`. Pages are still crawled. |
| `excludeRegex` | none | Don't crawl or check URLs that match. |
| `maxRedirects` | 10 | Hops followed before a link counts as a loop. |
| `requestDelayMs` | 1000 | Delay per host. Under 1000 ms applies only to your own sites with `ownSiteConfirmed`. |
| `maxConcurrency` | 5 | Sites or hosts processed at once. Each host stays sequential. |
| `respectRobotsTxt` | true | Can be turned off only for your own sites, with `ownSiteConfirmed`. |
| `ownSiteConfirmed` | false | Tick only for sites you own. |

### Output

One item per unique link. `category` is one of `ok`, `broken`, `redirect`, `redirect-chain`, `loop`, `soft-404`, `blocked-by-site`, `rate-limited`, `blocked-by-robots`, `blocked-private`, `timeout` or `dns-error`.

- `blocked-by-site`: the link answered HTTP 401, 403 or 999, so the site refused the automated check (bot protection or a login wall). That isn't evidence the page is gone, so it's free and not counted as broken.
- `rate-limited`: the link answered HTTP 429 after retries, or wasn't requested because its host had answered 429 three times in a row. Also free and not counted as broken.
- `blocked-private`: the link points at, or redirects to, a localhost or private network address. It's never requested and never charged.

Real item from a local test run on the default prefill (1 Nov 2026), with the redirect chain and timestamp trimmed:

```json
{
  "url": "https://stackoverflow.com/questions/tagged/crawlee",
  "category": "blocked-by-site",
  "ok": false,
  "status": 403,
  "hops": 0,
  "redirectType": null,
  "error": "HTTP 403: the site refused the automated check (bot protection or login required); not a dead link as far as we know, open it in a browser",
  "isExternal": true,
  "linkType": "a",
  "anchorText": "Stack Overflow",
  "sourcePages": ["https://crawlee.dev/", "https://crawlee.dev/blog"],
  "sourcePageCount": 5,
  "method": "GET",
  "chargeable": false
}
```

Each run also saves a Markdown summary (`SUMMARY.md`) of the problems found, with a "Check manually" list of the `blocked-by-site` and `rate-limited` links to open in a browser.

### How long does it take?

On the default prefill (5 pages, 100 links on crawlee.dev), a cloud run took about 58 seconds and checked 97 links. Because each host gets about 1 request per second, a big single-site check takes longer: 1,000 links on one host is roughly 17 minutes. Raise the run timeout or lower `maxLinks` to match. The run stops cleanly before its timeout and keeps the results it has.

### How much does it cost?

Pay per event: one `link-checked` event per unique link with a definitive result (a status, a redirect chain, a loop, a soft-404 or a DNS failure). These are free: timeouts, rate-limited links (429), links the site refused (401, 403 or 999, `blocked-by-site`), robots-blocked links and links to private addresses. Price: $0.80 per 1,000 links checked ($0.0008 per `link-checked` event). `maxLinks` and your maximum charge per run both cap the total.

### Use with AI agents

Designed for limited permissions so it's callable through the Apify MCP server. Example: "Check crawlee.dev for broken links and tell me which pages link to them."

### OUTPUT_SUMMARY stop flags

Each run also writes `OUTPUT_SUMMARY` to the default key-value store (alongside `SUMMARY.md`). Monitors should read it before treating unchecked pages' links as fixed. Boolean stop flags (real field names):

| Flag | Meaning |
|---|---|
| `capHit` | `maxLinks` was reached with links still unchecked. |
| `pageHit` | `maxPagesPerSite` was reached with same-site URLs still queued. |
| `budgetHit` | The run's charge budget (`maxTotalChargeUsd`) was reached. |
| `timeHit` | The run stopped early because it was near the time limit. |

`linksNotChecked` is the count of found links that were not checked because a stop flag fired. A missing flag (older builds) should be treated as unknown, not as "did not hit".

**Dead internal links count as crawled pages.** Same-site URLs (including ones that later answer 404) are queued as pages under `maxPagesPerSite`, so an internal 404 still consumes a page slot and can set `pageHit`.

### Limits

- **A 401, 403 or 999 doesn't mean the link is dead.** Many sites block automated requests. These links are reported as `blocked-by-site`, aren't counted as broken and aren't charged. Check them in a browser before removing them. The checker never tries to get past the block.
- **Rate-limited links aren't checked.** A 429 after retries, or a host paused after three 429s in a row, gives `rate-limited` (free). Check those links in a browser or re-run later.
- **Soft-404 detection is a heuristic.** It flags 200 pages that say "not found" and deep URLs redirected to the homepage, and only for pages it fetched in full.
- Results cover the pages and links within your caps, not necessarily the whole site.
- It never logs in or gets past bot protection or CAPTCHAs.

### FAQ

**Does it check links to other websites?** Yes, once each, unless you turn off `checkExternalLinks`. It never crawls other sites.
**What counts as a redirect chain?** More than one redirect hop before the final page. `redirectType` shows whether the hops were permanent, temporary or mixed.

### Legal

Checks public links only. Stores status codes and redirect information, not page content or personal data.

### Changelog

See CHANGELOG.md.

# Changelog

This Actor's version history is a separate document: https://apify.com/madpai/broken-link-checker/changelog.md

# Actor input Schema

## `urls` (type: `array`):

Your site(s) to check: domains (example.com) or start page URLs. Only these sites are crawled; links to other sites are checked once each but never crawled.

## `seedFromSitemap` (type: `boolean`):

Also crawl pages listed in the site's XML sitemap (found via robots.txt or /sitemap.xml), so pages without inbound links are covered too.

## `maxPagesPerSite` (type: `integer`):

How many pages per site are fetched and scanned for links. Each page can contain many links.

## `maxLinks` (type: `integer`):

Total cap on unique links checked (crawled pages count too). You pay per link checked, so this also caps cost. At about 1 request per second per host, large single-site runs take longer than the default 5-minute timeout.

## `checkExternalLinks` (type: `boolean`):

Check outbound links too (one request each, robots.txt respected, about 1 request per second per host). External pages are never crawled.

## `checkAssets` (type: `boolean`):

Check img/script src and stylesheet/icon link href in addition to a\[href] links.

## `includeRegex` (type: `string`):

Check and report only links matching this case-insensitive regular expression, e.g. /docs/. Pages are still crawled to find links.

## `excludeRegex` (type: `string`):

Don't crawl or check URLs matching this case-insensitive regular expression, e.g. /tag/|?replytocom=.

## `maxRedirects` (type: `integer`):

Redirect hops followed per link before it is reported as a loop (too many redirects).

## `requestDelayMs` (type: `integer`):

Per-host delay (about 1 request per second by default). robots.txt Crawl-delay is honoured if longer (max 30 s). Values under 1000 ms only apply to your own sites with "I own these sites" checked; other hosts always get at least 1 s.

## `maxConcurrency` (type: `integer`):

How many sites or hosts are processed at once. Requests to any single host stay sequential and rate-limited.

## `respectRobotsTxt` (type: `boolean`):

Don't fetch URLs disallowed by robots.txt (reported as blocked-by-robots, free). Can only be turned off for your own sites, together with "I own these sites". External hosts' robots.txt is always respected.

## `ownSiteConfirmed` (type: `boolean`):

Only check this for your own sites. It allows turning off robots.txt and using delays under 1 second on the sites in "urls" (not on external hosts).

## Actor input object example

```json
{
  "urls": [
    "crawlee.dev"
  ],
  "seedFromSitemap": true,
  "maxPagesPerSite": 5,
  "maxLinks": 100,
  "checkExternalLinks": true,
  "checkAssets": false,
  "maxRedirects": 10,
  "requestDelayMs": 1000,
  "maxConcurrency": 5,
  "respectRobotsTxt": true,
  "ownSiteConfirmed": false
}
```

# Actor output Schema

## `links` (type: `string`):

Dataset with one item per unique link: url, category (ok, broken, redirect, redirect-chain, loop, soft-404, blocked-by-site, rate-limited, blocked-by-robots, blocked-private, timeout, dns-error), ok, status, finalUrl, redirect chain and hops, redirectType, error, isExternal, linkType, anchorText, sourcePages, sourcePageCount, method, chargeable.

## `summary` (type: `string`):

JSON summary: items pushed, charged events, counts per category, broken total, pages crawled, links found vs checked, external hosts, skipped links, caps hit and the politeness policy used.

## `report` (type: `string`):

SUMMARY.md: top problems grouped by the page that links to them, plus a 'Check manually' list of blocked-by-site and rate-limited links.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "crawlee.dev"
    ],
    "maxPagesPerSite": 5,
    "maxLinks": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("madpai/broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["crawlee.dev"],
    "maxPagesPerSite": 5,
    "maxLinks": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("madpai/broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 5,
  "maxLinks": 100
}' |
apify call madpai/broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,madpai/broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JU4mjllIjnvcytOBc/builds/cA1GSgx9JMAu65E9x/openapi.json