# Broken Link Checker – 404s, Redirects & #Anchors (`rationalistic_tail/broken-link-anchor-checker`) Actor

Find broken links on a public website: crawls pages politely (robots.txt + per-host rate limit), checks internal and external links, redirects, soft-404s and missing #anchors. Returns problem rows with source page, anchor text and a fix hint, plus a compact AI-ready summary.

- **URL**: https://apify.com/rationalistic_tail/broken-link-anchor-checker.md
- **Developed by:** [Jerry Lin](https://apify.com/rationalistic_tail) (community)
- **Categories:** Developer tools, SEO tools, Agents
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 link checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Broken Link Checker do?

**Broken Link Checker – 404s, Redirects & #Anchors** crawls a public website, checks **every internal and external link**, and tells you exactly **which links are broken, on which page, with which anchor text – and how to fix them**. Give it one URL like `https://example.com/` and get a clean, prioritized repair list in minutes.

Unlike plain status-code checkers it also finds the problems those tools miss:

- 🔗 **Broken links** – 404/410 and other 4xx, 5xx server errors, DNS failures, timeouts, TLS errors, redirect loops
- ⚓ **Missing #anchors** – `/docs/page#install` where no element with `id="install"` exists (very common after docs or CMS rewrites)
- 👻 **Soft 404s** – pages that answer `200 OK` but say "Page not found"
- ↪️ **Redirects** – every hop of the chain, with the final URL to update the link to
- 🛡️ **Fewer false alarms** – `401/403` bot-protection answers and `429` rate limits are reported separately as `access_denied` / `rate_limited`, not as "broken"
- 🧭 **Source page + anchor text** for every problem, plus a one-line **fix hint**
- 🤖 **AI-agent ready** – a compact `OUTPUT_SUMMARY` with an `llmSummary` paragraph, worst pages and top problems; works through the [Apify MCP server](https://mcp.apify.com)

### Polite and compliant by design

- ✅ **robots.txt is always honored** – for your site *and* for every external site that is checked. Disallowed URLs are listed as `blocked_by_robots` and **never charged**.
- ✅ **Rate limited per host** – default 1 request/second per host, never faster than 4/s, and a larger robots.txt `Crawl-delay` always wins. Max 5 parallel requests in total.
- ✅ **Public pages only** – no login, no cookies, no CAPTCHA solving, no proxies needed. The crawler identifies itself with a clear User-Agent (`BrokenLinkAnchorChecker`).
- ✅ External links are checked with a lightweight `HEAD` request (small `GET` fallback), never crawled.

### How to use it

1. Enter one or more **Start URLs** (e.g. your homepage or docs root).
2. Choose **mode**: `crawl` (discover internal pages up to *Max pages*) or `list` (only the links on the pages you list).
3. Optionally turn on **images/scripts/stylesheets** checks or **Include healthy links**.
4. Click **Start**. Problems appear in the **Problems** table, worst first; the summary is in `OUTPUT_SUMMARY`.
5. Schedule it weekly on Apify and connect a webhook, Slack, Make or Zapier to get alerted about new broken links.

#### Example input

```json
{
  "startUrls": ["https://docs.example.com/"],
  "mode": "crawl",
  "maxPages": 100,
  "maxLinks": 2000,
  "checkExternalLinks": true,
  "checkAnchors": true
}
```

#### Example output row

```json
{
  "verdict": "missing_anchor",
  "url": "https://docs.example.com/setup#installation",
  "httpStatus": 200,
  "fixHint": "No element with id/name \"installation\" on the target page; fix the #fragment.",
  "sourceCount": 3,
  "sources": [{ "pageUrl": "https://docs.example.com/", "anchorText": "Install guide" }],
  "isExternal": false,
  "linkType": "link"
}
```

Verdicts: `broken`, `server_error`, `network_error`, `soft_404`, `missing_anchor`, `rate_limited`, `access_denied`, `redirect`, `blocked_by_robots` (and `ok` with *Include healthy links*).

#### Summary for AI agents (`OUTPUT_SUMMARY`)

```json
{
  "pagesCrawled": 100,
  "linksChecked": 1432,
  "counts": { "ok": 1390, "broken": 12, "redirect": 21, "missing_anchor": 6, "access_denied": 3 },
  "worstPages": [{ "pageUrl": "https://docs.example.com/old-guide", "problemLinks": 7 }],
  "llmSummary": "Checked 1432 unique links from 100 page(s) of docs.example.com. Problems: 12 broken (4xx), ..."
}
```

### How much does it cost?

This Actor uses **pay-per-event** pricing – no subscription, no separate compute bill:

| Event | Price |
|---|---|
| `link-checked` – one unique URL verified (each crawled page and each distinct link counts once, however many pages link to it) | **$0.001** |
| Actor start (per GB of memory; default 512 MB = 1 event) | **$0.002** |

Examples: 300 links ≈ **$0.30**, 1,000 links ≈ **$1.00**. URLs skipped because of robots.txt are free, and dataset rows are free. Use **Max links** to cap the cost, or set a maximum cost per run – the Actor stops cleanly at the limit.

### Use it from AI agents (MCP)

Add the Actor to any MCP client (Claude, Cursor, VS Code, ChatGPT connectors):

```
https://mcp.apify.com?tools=rationalistic_tail/broken-link-anchor-checker
```

The agent sends `startUrls` (and optionally `mode`, `maxPages`), then reads `OUTPUT_SUMMARY.llmSummary` or the problem rows. The Actor runs with **limited permissions**, so it also works with prepaid agent tokens (x402 / Skyfire via Apify AGI).

### FAQ

**Does it render JavaScript?** No – it reads the HTML your server returns, which is what search engines see first. Links injected only by client-side JavaScript are not discovered.

**Why is a link `access_denied` instead of broken?** Many sites (LinkedIn, some CDNs) answer `403`/`999` to any automated request. We report them separately so your repair list contains only real problems.

**Can I check a site I don't own?** Only public pages, and only as far as robots.txt allows. Please keep the default polite settings.

**Is it legal?** It only requests public URLs, honors robots.txt, sends no personal data and collects none.

### Other tools by the same developer

- [Sitemap Extractor, Validator & Generator](https://apify.com/rationalistic_tail/sitemap-validator-generator)
- [Lighthouse Audit & Performance Budgets](https://apify.com/rationalistic_tail/lighthouse-budget-auditor)
- [Website to RSS / Change Monitor](https://apify.com/rationalistic_tail/website-to-rss-monitor)

# Actor input Schema

## `startUrls` (type: `array`):

Public page URLs to start from, e.g. \["https://example.com/"]. In crawl mode the Actor follows internal links on the same host (www. ignored). Login-protected or robots.txt-disallowed pages are skipped.

## `mode` (type: `string`):

crawl = discover internal pages from the start URLs (up to maxPages) and check all their links. list = only check the links found on the start URLs themselves.

## `maxPages` (type: `integer`):

Maximum internal HTML pages to fetch and scan for links (1-1000). Keeps runs small and polite.

## `maxLinks` (type: `integer`):

Maximum unique URLs verified over HTTP, crawled pages included (1-10000). Each verified URL is one billable 'link-checked' event.

## `checkExternalLinks` (type: `boolean`):

Also verify links pointing to other domains (HEAD request, GET fallback). External sites' robots.txt is honored too.

## `checkAnchors` (type: `boolean`):

Verify that internal links like /page#section point to an existing id or name on the target page.

## `checkResources` (type: `boolean`):

Also verify <img>, <script>, stylesheet and <iframe> URLs, not only <a href> links.

## `detectSoft404` (type: `boolean`):

Flag internal pages that return HTTP 200 but whose title or H1 says 'not found'.

## `includeOkLinks` (type: `boolean`):

false (default) = dataset lists only problems (broken, redirect, soft-404, missing anchor, unreachable, robots-skipped). true = every checked link.

## `maxConcurrency` (type: `integer`):

Parallel requests across all hosts (1-5). Requests to the same host are always spaced by the delay below.

## `requestDelayMs` (type: `integer`):

Minimum gap between two requests to the same host (250-60000 ms). A larger robots.txt Crawl-delay always wins.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout in seconds (5-60). Timeouts are reported as network_error.

## `slowThresholdMs` (type: `integer`):

Healthy links slower than this are marked slow=true (only visible with includeOkLinks).

## Actor input object example

```json
{
  "startUrls": [
    "https://info.cern.ch/"
  ],
  "mode": "crawl",
  "maxPages": 5,
  "maxLinks": 100,
  "checkExternalLinks": true,
  "checkAnchors": true,
  "checkResources": false,
  "detectSoft404": true,
  "includeOkLinks": false,
  "maxConcurrency": 2,
  "requestDelayMs": 1000,
  "requestTimeoutSecs": 20,
  "slowThresholdMs": 5000
}
```

# Actor output Schema

## `problems` (type: `string`):

One row per problem link, worst first, with source pages and fix hints.

## `summary` (type: `string`):

Counts, worst pages and an llmSummary paragraph for AI agents.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://info.cern.ch/"
    ],
    "maxPages": 5,
    "maxLinks": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("rationalistic_tail/broken-link-anchor-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://info.cern.ch/"],
    "maxPages": 5,
    "maxLinks": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("rationalistic_tail/broken-link-anchor-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://info.cern.ch/"
  ],
  "maxPages": 5,
  "maxLinks": 100
}' |
apify call rationalistic_tail/broken-link-anchor-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rationalistic_tail/broken-link-anchor-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/o9quq7pzSMqj2cf30/builds/pDNlehgqqEtOo1pgE/openapi.json
