# Broken Link Checker & Bulk URL Status Code Checker (`kilndata/broken-link-checker`) Actor

Check the HTTP status of thousands of URLs, or crawl a site and test every link on it. Status code, full redirect chain, final URL, response time, error type (DNS, SSL, timeout, 4xx, 5xx) and the page each broken link sits on. HEAD with GET fallback, polite per-domain limits.

- **URL**: https://apify.com/kilndata/broken-link-checker.md
- **Developed by:** [Datakiln](https://apify.com/kilndata) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker and Bulk URL Status Checker

Give it a list of URLs, or a website to crawl, and it tells you which links work and which do not. For every URL you get the HTTP status code, the full redirect chain, the final URL, the response time and a plain error type (DNS, SSL, timeout, 4xx, 5xx). In crawl mode each row also lists the pages where the link was found, so you know where to fix it.

It uses HEAD requests and falls back to GET when a server answers HEAD badly, which many do. It reads no JavaScript, opens no browser and needs no login.

### What you can use it for

| Use | How |
|---|---|
| Find dead links on your site | Crawl mode on your home page; filter the dataset to `broken` |
| Check a redirect migration | List mode with your old URLs; compare `finalUrl` with what you expected |
| Audit redirect chains | Look at `redirectCount` and `redirects` (301 then 302 then 200 is easy to spot) |
| Clean a backlink or outreach list | List mode with the URLs; drop the rows with DNS, 404 or 410 |
| Monitor a handful of pages | Schedule the Actor on a list of key URLs and watch for `status` changes |
| Feed an AI agent or workflow | Call it through the API or MCP and read the JSON rows |

### How to use it

1. Paste URLs, one per line. A missing `https://` is added for you.
2. Pick the mode. List checks exactly what you pasted. Crawl opens each page you pasted, reads its links and checks them.
3. Start the run and read the dataset. The "Error details" view shows the reason for each failure.

Example input:

```json
{
    "mode": "crawl",
    "urls": ["https://www.sitemaps.org/"],
    "maxDepth": 1,
    "maxPagesPerSite": 4,
    "checkExternal": true
}
```

| Field | Default | What it does |
|---|---|---|
| `urls` | required | URLs to check (list mode) or pages to start from (crawl mode) |
| `mode` | `list` | `list` or `crawl` |
| `maxDepth` | 1 | Crawl only. 0 opens just the page you gave. 1 also opens the internal pages it links to. Deeper links are still checked, just not opened |
| `maxPagesPerSite` | 50 | Crawl only. Pages opened per site |
| `checkExternal` | true | Crawl only. Also check links to other sites (status only, never opened) |
| `respectRobotsTxt` | true | Crawl only. See Limits |
| `maxConcurrency` | 10 | Requests in flight in total |
| `perDomainConcurrency` | 2 | Requests in flight against one domain |
| `perDomainDelayMs` | 200 | Minimum gap between two requests to one domain |
| `requestTimeoutSecs` | 20 | Wait time per answer |
| `maxRetries` | 2 | Retries for timeouts, resets, 429 and 5xx |
| `userAgent` | empty | Empty uses `BrokenLinkCheckerBot/1.0`, which says what it is |

### Output

One row per URL. This is a real row from a crawl of sitemaps.org, shortened:

```json
{
    "url": "http://asg.web.cmu.edu/rfc/rfc3986.php",
    "status": "broken",
    "statusCode": null,
    "finalUrl": "http://asg.web.cmu.edu/rfc/rfc3986.php",
    "redirectCount": 0,
    "redirects": [],
    "errorType": "dns",
    "error": "Domain does not exist or has no address (ENOTFOUND).",
    "responseTimeMs": null,
    "totalTimeMs": 170,
    "method": "HEAD",
    "isExternal": true,
    "foundOn": ["https://www.sitemaps.org/faq.php"],
    "foundOnCount": 1,
    "charged": true
}
```

`status` is one of:

| status | Meaning |
|---|---|
| `ok` | Answered 2xx with no redirect |
| `redirected` | Ended on a 2xx after one or more redirects. Check `redirects` |
| `broken` | A real failure: 404, 410, other 4xx, 5xx, DNS, SSL, refused connection, redirect loop, too many redirects |
| `unverified` | No reliable answer: timeout, 429, connection reset, 401/403/999 (often bot protection) |
| `skipped` | Not requested because robots.txt disallows it |
| `invalid-input` | The line was not a checkable URL |

`errorType` is `dns`, `ssl`, `timeout`, `connection-refused`, `connection-reset`, `http-4xx`, `http-5xx`, `access-denied`, `rate-limited`, `redirect-loop`, `too-many-redirects`, `invalid-redirect`, `robots-blocked` and a few others. A 5xx is reported only after retries, so a one-off 503 does not show up as broken.

A run summary (rows, billed URLs, URLs not checked, time) is stored under `SUMMARY` in the key-value store.

### How much does it cost?

You pay per checked URL (the `url-checked` event; the price is on the Pricing tab). A URL is billed when it gets a real answer: an HTTP response of any status except 429, a DNS failure, an SSL error, a refused connection, a redirect loop. These are not billed:

- timeouts, 429 answers, connection resets and temporary DNS failures (we could not get an answer, so we do not charge for the attempt)
- invalid lines and pages skipped because of robots.txt
- a URL that appears twice in one run

The Actor reads your maximum cost per run and stops by itself when it is reached. It never starts a check it could not bill within that limit. The summary says how many URLs were left unchecked. In crawl mode, pages that get opened count as checked URLs too.

### How it works

Each URL is requested with HEAD. If the answer is an error, or the server hangs, the same URL is requested with GET and the GET answer wins. Redirects are followed one by one, up to 10, so the whole chain is recorded and loops are caught. Retries use a short backoff and honor `Retry-After`. After a 429 the domain is slowed down for the rest of the run.

In crawl mode the Actor opens the start page with GET, reads the `<a>` links (the `<base>` tag is honored), opens internal HTML pages down to `maxDepth`, then checks every link it found. Links to files (PDF, images and so on) are checked, never downloaded. `mailto:`, `tel:` and `javascript:` links are ignored.

### Limits, read before you run

- Public pages only. No login, no cookies, no captcha solving. Pages that need a session will report 401 or 403.
- Private, local and internal addresses (localhost, 10.x, 192.168.x, 169.254.x) are rejected on purpose.
- Links are read from the HTML the server sends. Links added later by JavaScript are not seen.
- Only `<a href>` and `<area href>` links are checked. Images, scripts and stylesheets are not.
- robots.txt: with `respectRobotsTxt` on, pages and internal links it disallows are not requested. External links are checked with a single HEAD or GET to the URL itself, without reading the other site's robots.txt, the same way a browser or link checker does when you click a link.
- Some sites block anything that is not a browser. They show up as `unverified` with 403 or 999 (LinkedIn does this), not as broken. We do not try to get around that.
- A timeout means "no answer in time from where the Actor runs". The site may still be fine for you. Try a longer timeout before you trust it.
- There is no proxy support. Sites that block data center IPs may answer differently than they would for a visitor.
- Rows come out in the order checks finish, not in input order.
- One URL is checked once per run, even if several sites link to it. In crawl mode `foundOn` keeps the first 25 pages.

### Is it legal?

It requests public URLs the way a browser does when you click a link, at a low rate: two parallel requests per domain by default, with a pause between them. It sends an honest User-Agent. It stores no personal data, only URLs and what the server answered. You are responsible for checking only sites you are allowed to test, and for the terms of any site you point it at.

### FAQ

**Why is a working link marked `unverified`?**
The server answered 401, 403 or 999, or took too long. Open the link yourself. If it works in a browser, the site is filtering robots.

**Why GET sometimes, if HEAD is lighter?**
The `method` column shows what the final answer came from. GET means HEAD failed or returned an error, and we asked again to be sure. The body is not downloaded, except for pages opened in crawl mode.

**Can it check 100,000 URLs?**
Yes in list mode. Speed depends on how many domains you have: with 2 requests and a 200 ms gap per domain, 100,000 URLs on one domain takes hours, 100,000 spread over many domains takes much less. Raise `perDomainConcurrency` only for sites you own.

**Does it check anchors (`#section`)?**
No. The fragment is removed before the request, and the page is not searched for the anchor.

**Can I use it from the API or an AI agent?**
Yes. Run it through the Apify API, a webhook or the Apify MCP server and read the dataset items.

Not affiliated with any of the websites you check.

# Actor input Schema

## `urls` (type: `array`):

One per line. In "List" mode each line is checked as it is. In "Crawl" mode each line is a page to start from (usually a home page). A missing https:// is added. Duplicates are removed. Public http(s) addresses only.

## `mode` (type: `string`):

"List" checks exactly the URLs you gave. "Crawl" opens each page you gave (and internal pages up to the depth below), then checks every link on them and tells you on which page each link was found.

## `maxDepth` (type: `integer`):

Crawl mode only. 0 = open only the page you gave and check its links. 1 = also open the internal pages it links to, and check their links. Pages deeper than this are checked for status but not opened.

## `maxPagesPerSite` (type: `integer`):

Crawl mode only. How many pages of one website are opened to read their links. Links found on them are all checked.

## `checkExternal` (type: `boolean`):

Crawl mode only. Also check links that lead to other websites. They are only checked, never opened. Turn off to test internal links only.

## `respectRobotsTxt` (type: `boolean`):

Crawl mode only. Pages and internal links that robots.txt disallows are not requested (they come back as "skipped", free). External links are only status-checked with HEAD/GET and do not read the other site's robots.txt.

## `maxConcurrency` (type: `integer`):

How many requests run at the same time in total, across all domains.

## `perDomainConcurrency` (type: `integer`):

How many requests may hit the same domain at once. Keep it low: this is what keeps other people's servers safe.

## `perDomainDelayMs` (type: `integer`):

Pause between two requests to the same domain. After a 429 answer the domain is slowed down further.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one answer before it counts as a timeout. Timeouts are reported but not billed.

## `maxRetries` (type: `integer`):

Retries for timeouts, connection resets, 429 and 5xx answers. DNS failures, SSL errors and 4xx answers are never retried.

## `userAgent` (type: `string`):

Optional. Leave empty to use the default, which says clearly that this is a link checker (BrokenLinkCheckerBot/1.0). Change it only if you own the sites you check.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://wikipedia.org",
    "https://nodejs.org/en/about",
    "https://github.com/this-page-does-not-exist-404-test"
  ],
  "mode": "list",
  "maxDepth": 1,
  "maxPagesPerSite": 50,
  "checkExternal": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "perDomainConcurrency": 2,
  "perDomainDelayMs": 200,
  "requestTimeoutSecs": 20,
  "maxRetries": 2,
  "userAgent": ""
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com",
        "https://wikipedia.org",
        "https://nodejs.org/en/about",
        "https://github.com/this-page-does-not-exist-404-test"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("kilndata/broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://example.com",
        "https://wikipedia.org",
        "https://nodejs.org/en/about",
        "https://github.com/this-page-does-not-exist-404-test",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("kilndata/broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com",
    "https://wikipedia.org",
    "https://nodejs.org/en/about",
    "https://github.com/this-page-does-not-exist-404-test"
  ]
}' |
apify call kilndata/broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kilndata/broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xaSzCDOhu6Q5lhwtb/builds/79QTqZ1treubqo2n8/openapi.json