# Broken Link Checker: Crawl Sites, Find Dead Links (`webify-lab/broken-link-checker`) Actor

Crawls whole websites and checks every link, image and optional script or stylesheet. Reports 404s, server errors, dead domains, timeouts and redirect chains with the exact page and anchor text, plus mixed content and links to localhost or staging hosts. Pay per crawled page.

- **URL**: https://apify.com/webify-lab/broken-link-checker.md
- **Developed by:** [Batu T.](https://apify.com/webify-lab) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker: 404s, Redirects, Mixed Content

Crawl a whole website and check every link on it: pages, images and, if you want, scripts and stylesheets. You get one row per problem with the exact page it sits on and the anchor text, so you can fix it in minutes. Plus one summary per website with the numbers and the worst offenders.

### What it finds

| status | meaning |
|---|---|
| `broken` | the link answers 4xx or 5xx (404 Not Found, 410 Gone, 500 Server Error ...) |
| `dns-error` | the domain does not exist any more |
| `connection-error` | connection refused or reset, TLS certificate errors |
| `timeout` | no answer within the timeout |
| `too-many-redirects` | redirect loops and chains longer than the limit |
| `invalid-url` | malformed href in the HTML |
| `redirect` | works, but via 301/302/307/308; `finalUrl` and the full `redirectChain` show where it ends |
| `blocked` | the server refused the bot (429 rate limit, or 401/403/999 on external sites). Check these by hand |
| `skipped` | disallowed by robots.txt (only listed with output "All checked links") |

Extra flags in `issues`:

- `mixed-content`: an http image, script or stylesheet on an https page
- `staging-host`: links to localhost, private IPs, `.local`/`.test` hosts, tunnels or subdomains like `staging.`, `dev.`, `preview.`

### Output

One item per problem link occurrence:

| field | example |
|---|---|
| `sourcePage` | https://www.example.de/kontakt |
| `linkUrl` | https://www.example.de/alte-seite |
| `href` | /alte-seite (as written in the HTML) |
| `anchorText` | Zur Anfahrt (alt text for images) |
| `element` | a, img, script, link |
| `isExternal` | false |
| `status`, `statusCode` | broken, 404 |
| `finalUrl`, `redirectChain` | after redirects: \["301 http://example.de/x", "301 https://example.de/x"] |
| `issues` | \["mixed-content"] |
| `error` | HTTP 404, Domain does not resolve (ENOTFOUND) |
| `countOnPage` | how often this link appears on that page |

Plus one item with `type: "summary"` per website: `pagesCrawled`, `linksFound`, `linksChecked`, `ok`, `broken`, `brokenInternal`, `externalBroken`, `redirects`, `blocked`, `timeouts`, `mixedContent`, `stagingLinks`, `pagesWithProblems`, `topBrokenUrls` and `topRedirects` (each with occurrences and example pages), `robotsTxt`, `stoppedBy`, `durationSecs`. The summaries are also saved as `SUMMARY` in the key-value store.

Filter the dataset by `type` = `link` or `summary` when you export to CSV or Excel.

### How it checks

- Crawls pages on the start host (www and non-www count as one site, subdomains optional), breadth first, up to `maxPagesPerSite`. Links on crawled pages are all checked, also those the page limit stops it from opening.
- HEAD request first; GET when the server rejects or mishandles HEAD (400, 403, 404, 405, 429, 5xx, empty response). This avoids the false 405s and 404s that simpler checkers report.
- Redirects are followed one by one, so you see every hop and loops are detected.
- Every unique external URL is checked once per run, even across several websites.
- At most 5 parallel requests per host, one slower retry on 429, honest user agent, robots.txt respected (also for redirect targets).
- `Output`: problems only (default) or every checked link. `Report redirects`: all, only for links to your own site, or none (wa.me or link shortener redirects are usually intended).
- `Exclude URLs matching` skips URLs by regular expression, for example `linkedin\.com` or `\?sort=`. `Only crawl URLs matching` limits the crawl to a section such as `/blog/`.

### Pricing

You pay per crawled page (an HTML page of your site that was opened and checked). Link checks, images and external URLs are included. Start URLs that fail (invalid, not reachable, not HTML, disallowed by robots.txt) are not charged; they appear as a summary item with an `error`.

### Limits

- No JavaScript rendering: links that only appear after client-side rendering are not found. Server-rendered sites, WordPress, Next.js static export and most CMS sites work fine.
- Some sites (LinkedIn, Instagram, sites behind bot protection) refuse automated requests. Those links are reported as `blocked`, not `broken`. A proxy can be set as fallback for them.
- No login, no forms, no cookies. Sites that need a cookie to load may show as redirect loops.

### Input example

```json
{ "startUrls": ["https://www.example.de/"], "maxPagesPerSite": 200, "checkScriptsAndStyles": true }
```

### FAQ

**What am I charged for?** One event per crawled HTML page of your website. Checking links, images and external URLs is included. Start URLs that fail are not charged and show up as a summary item with an `error`. When your charge limit is reached, the crawl stops and the summary says `stoppedBy: "charge limit"`.

**What happens when `maxPagesPerSite` is reached?** The crawler stops opening new pages, but every link found on the pages it already crawled is still checked. The summary then says `stoppedBy: "maxPagesPerSite"`.

**Do I need a proxy?** No. Requests go out directly by default. If you configure a proxy, the default fallback strategy retries links that answer 401, 403, 429 or 999 or fail to connect once through the proxy; the `always` strategy sends every request through it.

**Why is a link `blocked` and not `broken`?** Rate limits (429) and bot refusals from other websites (401, 403, 999) say nothing about whether the page exists, so they are listed as `blocked` for a manual check instead of being counted as broken.

**Does it render JavaScript or log in?** No. It reads the HTML the server sends, without login, forms or cookies. Links that only appear after client-side rendering are not found.

### Use with AI agents (MCP)

This Actor works as a tool for Claude, ChatGPT, Cursor and other MCP clients through the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp). Add only this Actor:

```
https://mcp.apify.com?tools=webify-lab/broken-link-checker
```

On first use the server signs you in with OAuth. Alternatively send the header `Authorization: Bearer <APIFY_TOKEN>`. Agents on the default Apify MCP setup can also find this Actor with `search-actors` and run it with `call-actor`. Pricing is the same as in Apify Console: you pay per result.

# Actor input Schema

## `startUrls` (type: `array`):

One start URL per website, e.g. https://www.example.com. The crawler follows links on the same host from there.

## `maxPagesPerSite` (type: `integer`):

Safety limit. Only HTML pages that were crawled count. Links on those pages are all checked, even if the limit stops the crawler from opening them.

## `includeSubdomains` (type: `boolean`):

Also crawl pages on subdomains of the start host (blog.example.com, shop.example.com). Links to subdomains are always checked.

## `checkExternalLinks` (type: `boolean`):

Check links to other websites. Each unique external URL is requested once per run, no matter how many pages link to it.

## `checkImages` (type: `boolean`):

Check img src and srcset URLs.

## `checkScriptsAndStyles` (type: `boolean`):

Check script src, stylesheets, icons and preloads.

## `outputMode` (type: `string`):

Every run also adds one summary item per website.

## `reportRedirects` (type: `string`):

Redirects of external links (for example wa.me or link shorteners) are often intended. Applies to output 'Problems only'; the summary always counts all redirects.

## `respectRobotsTxt` (type: `boolean`):

Do not crawl or check pages of the website that its robots.txt disallows for this bot.

## `includePatterns` (type: `array`):

Regular expressions matched against the full URL, e.g. /blog/. If set, only matching pages are crawled (the start URL always is). Links on crawled pages are still checked.

## `excludePatterns` (type: `array`):

Regular expressions matched against the full URL, e.g. ?sort= or linkedin.com. Matching URLs are neither crawled nor checked.

## `concurrency` (type: `integer`):

Requests running at the same time across all hosts.

## `requestTimeoutSecs` (type: `integer`):

A link that does not answer in this time is reported as timeout.

## `maxRedirects` (type: `integer`):

Longer chains and loops are reported as too-many-redirects.

## `userAgent` (type: `string`):

Sent with every request. The default names the bot honestly; robots.txt rules for BrokenLinkChecker apply.

## `proxyStrategy` (type: `string`):

Only applies when a proxy is configured below.

## `proxyConfiguration` (type: `object`):

Optional. With the fallback strategy, links that answer 401, 403, 429 or 999 or fail to connect are retried once through the proxy.

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "maxPagesPerSite": 200,
  "includeSubdomains": false,
  "checkExternalLinks": true,
  "checkImages": true,
  "checkScriptsAndStyles": false,
  "outputMode": "problems",
  "reportRedirects": "all",
  "respectRobotsTxt": true,
  "includePatterns": [],
  "excludePatterns": [],
  "concurrency": 10,
  "requestTimeoutSecs": 20,
  "maxRedirects": 10,
  "userAgent": "Mozilla/5.0 (compatible; BrokenLinkChecker/1.0; +https://apify.com/store)",
  "proxyStrategy": "fallback",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One item per problem link occurrence (or per link with output 'all'): type 'link', startUrl, sourcePage, linkUrl, href, anchorText, element, isExternal, status, statusCode, finalUrl, redirectChain, issues, error, countOnPage. Plus one item per website with type 'summary': pagesCrawled, linksChecked, broken, redirects, externalBroken, topBrokenUrls and more.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("webify-lab/broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://books.toscrape.com/"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("webify-lab/broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call webify-lab/broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,webify-lab/broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aZzfxT5qUKZbWxqJp/builds/H1SVhuQ54HasLeUCT/openapi.json