# Broken Link Checker - Bulk URL Status Code Scraper (`neverempty/link-checker`) Actor

Checks every link on a page and says which state it is in: ok, not-found, blocked-to-us, server-error or unreachable. A 403 to this checker is reported as blocked, not broken. Falls back to GET when a server rejects HEAD, accepts every 2xx, and records the full redirect chain.

- **URL**: https://apify.com/neverempty/link-checker.md
- **Developed by:** [NeverEmpty](https://apify.com/neverempty) (community)
- **Categories:** SEO tools, Developer tools, MCP servers
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 link checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Broken Link Checker - Bulk URL Status Code Scraper

Checks every link on a page and reports which state each one is actually in. A link that answers `403` to an automated checker is reported as **blocked to us**, not as broken, because a person clicking it in a browser will very likely reach the page.

### Why that distinction is the whole product

Measured on 26 August 2026 against fifteen well-known sites, using a bot user-agent:

| What happened | How many | Examples |
|---|---|---|
| Answered `403` to the checker — not broken, just refusing automated requests | **6 of 15** | stackoverflow.com, yelp.com, tripadvisor.com, etsy.com, quora.com, glassdoor.com |
| Refused `HEAD` but answered `GET` | **2 of 15** | amazon.com (`HEAD 405` → `GET 202`), medium.com (`HEAD 403` → `GET 200`) |
| Answered with a 2xx that is not `200` | **4 of 15** | amazon.com and imdb.com and booking.com (`202`), github.com (`206`) |
| Genuinely gone | **1 of 15** | a `404` URL used as a control |

A checker that sends `HEAD` and treats "not 200" as broken would have reported **twelve of those fifteen links as broken**. One of them actually was.

This checker sends `HEAD`, retries with a short ranged `GET` when the server rejects `HEAD`, accepts every 2xx, and keeps `403` in a category of its own.

### The states

| `state` | Meaning |
|---|---|
| `ok` | Any 2xx, after redirects. `202` and `206` count. |
| `not-found` | `404` or `410`. The link is genuinely gone. |
| `blocked-to-us` | `401`, `403`, `429`, `451`. The server refused this checker. **This is not a broken link.** |
| `server-error` | 5xx. Often temporary; retried, and the state reflects the last attempt. |
| `client-error` | Another 4xx, a 3xx with no `Location`, or a redirect loop. |
| `unreachable` | No HTTP answer at all — DNS has no record, connection refused, TLS failed, or the request timed out. The reason says which. |

Every row also carries `naiveCheckWouldSayBroken`: true when a `HEAD`-and-200-only check would have called this link broken, while this checker found it alive or merely blocked. That column is the difference between the two tools, stated per link.

### What it does not claim

Attached to every row as `notClaimed`:

- A `blocked-to-us` link is not a broken link. The server refused this checker specifically.
- Page content is not inspected. A URL that returns `200` with the words "page not found" written in the body is reported as `ok`.
- JavaScript is not executed, so links created by scripts after load are not seen.
- A `server-error` can be temporary; it is retried, and the state reflects the last attempt.

### A real run, with the default input

The default input is `https://www.w3.org/History.html` — a page about the early history of the web, whose outbound links have had thirty years to rot. Measured on Apify: **72 links in 27 seconds — 58 `ok`, 10 `not-found`, 4 `unreachable`**, peak memory 73 MB.

The same run from a home connection took 50 seconds and reported one more link as `unreachable`. That is not a discrepancy to hide: reachability is measured from wherever the check runs, and a host that times out from one network can answer from another. The row says `unreachable` with the transport reason rather than `not-found`, precisely so the difference is visible.

Among them: `http://europa.eu.int/comm/index.htm` (the domain no longer resolves), `http://www.cern.ch/WWW94/` (same), and several W3C pages that now return `404`. Two links would have been misreported by a naive check.

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `startUrls` | array | `https://www.w3.org/History.html` | Pages whose links should be checked. Each distinct link is checked once, no matter how many pages point at it. |
| `urls` | array | empty | Check these URLs directly, without reading a page first. Can be combined with `startUrls`. |
| `checkInternal` | boolean | `true` | Include links back to the same site. The `www` prefix is ignored when deciding. |
| `checkExternal` | boolean | `true` | Include links to other sites. |
| `maxLinks` | number | `100` | How many links to check. Taken in page order; the run reports how many were left unchecked rather than hiding the truncation. |
| `concurrency` | number | `5` | Links checked in parallel across all servers. |
| `perHostConcurrency` | number | `2` | Never more than this many at once to one server. |
| `respectRobots` | boolean | `true` | Applies to fetching the pages links are read from. |
| `timeoutSecs` | number | `20` | Timeout for one link. Range 3–120. |
| `maxRetries` | number | `3` | Retries for `429` and 5xx. Range 1–10. |

### Two deliberate choices about speed

**Requests to one server are limited.** Sending everything at once makes a server answer `429`, and `429` is `blocked-to-us` — the checker would be manufacturing the very result it is measuring. Default is two at a time per server, five in total.

**Timeouts are retried at most twice, not `maxRetries` times.** A timeout costs the entire timeout period, and a server that did not answer in twenty seconds is rarely fixed by asking again six hundred milliseconds later. On the default page one dead host was taking 21 seconds per attempt; three attempts made the whole run 74 seconds. Capping timeout attempts at two brought it to 50 seconds with identical results. `429` and 5xx still get the full `maxRetries`, because those genuinely do clear.

### What each row contains

`url`, `finalUrl` (after redirects), `state`, `httpStatus`, `reason`, `naiveCheckWouldSayBroken`, `method` (whether `HEAD` was enough or `GET` was needed), `headRejectedStatus`, `redirectCount`, `redirectChain` (every hop with its status), `elapsedMs`, `transportError`, `isInternal`, `linkedFrom` (which pages link to it), `anchorTexts`, `notClaimed`, `checkedAt`.

Pages that could not be read for links are returned too, with `state: null` and a `reason` — robots.txt disallowed them, they were not HTML, or the fetch failed. Those rows are not charged.

### Typical uses

- **Link rot on old content.** Long-lived pages accumulate dead outbound links; this separates the genuinely dead from the merely unfriendly-to-bots.
- **Before a site migration.** Record the redirect chain of every internal link, then compare afterwards.
- **Cleaning a directory or resource list.** `not-found` and `unreachable` are actionable; `blocked-to-us` usually is not, and should not be deleted by mistake.

### FAQ

**Why is `403` not treated as broken?**
Because six of the fifteen sites measured answer `403` to any automated checker while working perfectly in a browser. Deleting those links would remove working references.

**Why send `HEAD` first?**
It avoids downloading page bodies. When a server rejects `HEAD` — amazon.com answers `405`, medium.com answers `403` — the check is repeated with a `GET` limited to the first 2 KB.

**Does it follow redirects?**
Yes, and it records them. Redirects are followed manually rather than by the HTTP client, because an automatically-followed redirect is invisible afterwards and the chain is often the thing you need to see.

**Is a `200` guaranteed to mean the page is fine?**
No, and this does not claim so. Some sites answer `200` with a "not found" message in the body. Checking that would mean reading and interpreting page content, which this deliberately does not do — `notClaimed` says so on every row.

### Other tools by NeverEmpty

Every NeverEmpty Actor follows the same rule: it never returns an empty result to mean two different things, and it only charges for rows that actually carry an answer.

- **[seo-site-audit](https://apify.com/neverempty/seo-site-audit)** - technical SEO audit: meta, canonical, robots, headings
- **[domain-lookup](https://apify.com/neverempty/domain-lookup)** - WHOIS, DNS, MX, SPF, DMARC and SSL expiry for a list of domains
- **[website-contacts](https://apify.com/neverempty/website-contacts)** - emails, phones and social profiles from any website, no invented addresses

### Support

Found a link classified wrongly? Open an issue on the Issues tab with the URL and the row you got. Reports about incorrect data are fixed first.

# Actor input Schema

## `startUrls` (type: `array`):

Pages whose links should be checked. Every http(s) link on these pages is collected and checked once, even if several pages point at it.

## `urls` (type: `array`):

Check these URLs without reading any page first. Can be used together with the pages above, or on its own.

## `checkInternal` (type: `boolean`):

Include links that point back to the same site as the page they were found on. The www prefix is ignored when deciding.

## `checkExternal` (type: `boolean`):

Include links that point at a different site.

## `maxLinks` (type: `number`):

How many links to check before stopping. Links are taken in the order they appear on the page, and the run reports how many were left unchecked.

## `concurrency` (type: `number`):

How many links to check in parallel across all servers.

## `perHostConcurrency` (type: `number`):

Never send more than this many requests at the same time to one server. Sending too many makes the server answer 429, which would turn into a 'blocked-to-us' result this checker caused itself.

## `respectRobots` (type: `boolean`):

Applies to fetching the pages the links are read from. The links themselves are checked with HEAD or a short ranged GET, which is what a browser following the link would do.

## `timeoutSecs` (type: `number`):

How long to wait for one link before treating it as unreachable.

## `maxRetries` (type: `number`):

How many times to retry HTTP 429 and 5xx. Timeouts are retried at most twice regardless, because a timeout costs the full timeout period and rarely clears on an immediate retry. A 404 or 403 is never retried, because that is already an answer.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.w3.org/History.html"
    }
  ],
  "urls": [],
  "checkInternal": true,
  "checkExternal": true,
  "maxLinks": 100,
  "concurrency": 5,
  "perHostConcurrency": 2,
  "respectRobots": true,
  "timeoutSecs": 20,
  "maxRetries": 3
}
```

# Actor output Schema

## `results` (type: `string`):

All links produced by this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.w3.org/History.html"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("neverempty/link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://www.w3.org/History.html" }] }

# Run the Actor and wait for it to finish
run = client.actor("neverempty/link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.w3.org/History.html"
    }
  ]
}' |
apify call neverempty/link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neverempty/link-checker"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xWaCJZxWV60cyg6zo/builds/HC2Tc0snrwyYnFPar/openapi.json