# Find Broken Links on Your Whole Site: 404 Link Checker (`lotebo-lab/broken-link-auditor`) Actor

In: one domain. Out: one row per broken link, with the HTTP status, the page the link sits on and the anchor text to search for in your CMS. Respects robots.txt.

- **URL**: https://apify.com/lotebo-lab/broken-link-auditor.md
- **Developed by:** [Tarcio Elyakin Agra Diniz](https://apify.com/lotebo-lab) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$10.00 / 1,000 link or page checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker: Find Every 404 on Your Whole Site

Somewhere on your site there is a link pointing at a page that does not exist any more, and today you only find out when a visitor tells you.

This Actor takes a domain, crawls the site by following internal links, and returns one row per broken link: the address that failed, the HTTP status or the network error, the page the link sits on, and the anchor text you have to search for in your CMS.

### Who runs it, and when

People who are responsible for a site and cannot click every link by hand:

- **site owners and marketing teams** after a redesign, a CMS migration or a bulk URL change, when old addresses stop answering;
- **agencies and freelancers** who take over someone else's site and need a list of what is already broken before they touch anything;
- **technical writers and documentation maintainers**, where a dead link in a tutorial is a support ticket;
- **anyone doing a routine check**, monthly or before a release, on a site whose outbound links rot on their own.

The usual moment is: you changed something, or you inherited something, and you need the list of dead links in one file instead of a browser tab per page.

### What comes out, field by field

The run writes **one dataset row per broken link found**, plus **one summary row** at the end. Both are in the same dataset, and the `rowType` field tells them apart, so you can filter with one comparison.

#### Rows with `rowType: "broken-link"`

| field | type | what it holds |
|---|---|---|
| `rowType` | string | always `broken-link` on a finding row |
| `brokenUrl` | string | the link target that failed |
| `statusCode` | integer or null | the HTTP status the target returned (404, 410, 500...), `null` when the request itself failed |
| `error` | string or null | the network level failure when there is no HTTP status: `dns_error`, `connect_timeout`, `read_timeout`, `ssl_error`, `too_many_redirects` |
| `sourceUrl` | string | the crawled page that contains the link |
| `anchorText` | string | the clickable text of the link, so you can find it in the page |
| `linkType` | string | `internal` or `external` |
| `checkedAt` | string | when the target was checked, ISO 8601 in UTC |

One row per place the link appears: a dead address linked from five pages gives five rows, one per source page, because five pages need editing.

#### The row with `rowType: "summary"`

Every successful run writes this row, findings or no findings, so a clean site never hands you an empty dataset.

| field | type | what it holds |
|---|---|---|
| `rowType` | string | always `summary` |
| `startUrl` | string | the URL the crawl started from, after normalisation |
| `pagesVisited` | integer | how many pages were opened and parsed |
| `linksFound` | integer | how many links were seen in total, counting repeats |
| `urlsChecked` | integer | how many distinct targets were actually requested |
| `brokenLinks` | integer | how many finding rows this run wrote |
| `uniqueBrokenTargets` | integer | how many distinct addresses those rows point at |
| `skippedByRobots` | integer | targets not requested because `robots.txt` disallows them |
| `durationSeconds` | number | how long the run took |
| `chargeLimitReached` | boolean | `true` when the run stopped early on your pay-per-event limit |
| `chargedEvents` | integer | how many `url-checked` events the run charged |
| `finishedAt` | string | when the run ended, ISO 8601 in UTC |
| `message` | string | the same numbers in one sentence, for a human reading the dataset |

A fuller version of the same summary is also written to the key-value store under the key `SUMMARY`. It adds the settings the run used (`maxPages`, `maxDepth`, `delaySeconds`, `concurrency`), `uniqueTargets`, `requestsCharged` and `chargeFailures`.

#### A real finding row

Lines 94 to 102 of `saidas/docs-apify-com.json` in this repository, from a run against the public site `https://docs.apify.com/` on 19/09/2026:

```json
{
  "brokenUrl": "https://docs.apify.com/residential_proxy",
  "statusCode": 404,
  "error": null,
  "sourceUrl": "https://docs.apify.com/proxy",
  "anchorText": "residential proxies",
  "linkType": "internal",
  "checkedAt": "2026-09-19T22:32:12+00:00"
}
```

Read it as: the page `/proxy` has a link whose text is "residential proxies", it points at `/residential_proxy`, and that address answers 404. That run predates the `rowType` field; current builds add `rowType` to every row, as described above.

### Input

The example below is the input this Actor is prefilled with, so you can press Start and read a real result before you point it at your own site. `books.toscrape.com` is a site published for exactly this kind of testing.

```json
{
  "startUrl": "https://books.toscrape.com",
  "maxPages": 50,
  "maxDepth": 3,
  "checkExternalLinks": true
}
```

| field | type | default | range |
|---|---|---|---|
| `startUrl` (required) | string | — | a bare domain (`example.com`, `https://` is assumed) or a full URL |
| `maxPages` | integer | 50 | 1 to 5000 pages opened by the crawler |
| `maxDepth` | integer | 3 | 0 to 20 clicks from the start page; 0 checks only the links on the start page |
| `requestDelaySeconds` | number | 0.25 | 0 to 60 seconds between two requests to the same host |
| `requestTimeoutSeconds` | integer | 15 | 3 to 120 seconds before a link is reported as a timeout |
| `concurrency` | integer | 8 | 1 to 32 targets checked in parallel; the per-host delay still applies |
| `checkExternalLinks` | boolean | true | also check links pointing to other sites; other sites are never crawled |
| `includeSubdomains` | boolean | false | treat `blog.example.com` as part of `example.com` and crawl it too |

Links found on the crawled pages are all checked, even when the crawl stops at `maxPages`.

### What this Actor does not do

- **It does not fix anything.** It does not edit your site, your CMS or your redirects.
- **It does not run JavaScript.** It reads the HTML the server returns, so a link that only appears after a script runs is not seen.
- **It does not find orphan pages.** It follows internal links from the start URL; a page nobody links to is never visited.
- **It does not check anchors inside a page (`#section`), image sources, CSS or script URLs, or redirect chains.** It checks link targets.
- **It does not judge SEO, page speed, accessibility or content quality.**
- **It does not log in, fill forms, solve captchas or get past a paywall.**
- **It does not collect personal data.** The output holds URLs, HTTP statuses and anchor text. `mailto:` and `tel:` links are skipped.
- **It does not decide for you what a 403 or a 429 means.** Some servers answer that to any crawler; the status is reported as it came back.
- **It does not promise a number of links per minute.** The pace is set by your `requestDelaySeconds` and by any `Crawl-delay` the site publishes.

### Manners, `robots.txt` and your responsibility

- **`robots.txt` is fetched before the crawl and always respected.** There is no option to turn it off. A path disallowed for our user agent is not requested and is counted in `skippedByRobots`. Requests to `robots.txt` itself are never charged.
- **If the site's `robots.txt` asks for a longer `Crawl-delay` than your `requestDelaySeconds`, the longer value wins.**
- **The Actor identifies itself** on every request as `LoteboBrokenLinkAuditor/0.1 (+https://apify.com/store; Apify Actor; contact via Apify Store page)`. That string will appear in your logs and you can write a rule for it.
- **You are responsible for having the right to access the URLs you give it.** Before you run it, check the terms of the site and its `robots.txt`, and check whether your own agreement with that site allows automated access. Point it at sites you own or are allowed to audit.

### Price

Pay per event, one event, exactly as declared in `.actor/actor.json`:

| event | price | when it is charged |
|---|---|---|
| `url-checked` | US$ 0.01 | once per HTTP request made to the audited site: a page fetch, a HEAD check of a link target, or the GET fallback when HEAD is refused. `robots.txt` requests are never charged |

Each distinct target is checked once per run, even when many pages link to it. Apify charges its own Actor start event and the platform usage of the run on top of this; those are not set by this Actor. If a run reaches your pay-per-event limit, it stops, keeps everything found so far, writes the summary row and sets `chargeLimitReached` to `true`.

### About this Actor

The code, the tests and the run logs quoted here are in this repository. The Actor is written in Python and was built with the help of AI.

### Example tasks

Each page below is a published example task of this Actor. It shows the input used and the fields the run returns. The same page is served as Markdown by adding `.md` to the URL.

- [Check for broken links after a site migration](https://apify.com/lotebo-lab/broken-link-auditor/examples/check-broken-links-after-site-migration): crawls the migrated site and returns every broken link with the page it sits on, so redirects missed in the move are fixed before traffic drops.
- [Crawl an entire website for 404 errors](https://apify.com/lotebo-lab/broken-link-auditor/examples/crawl-entire-website-for-404-errors): starts from the home page, follows every internal link and returns each URL that answers 404 or another error, with the page that links to it.
- [Find dead outbound links on a blog or docs site](https://apify.com/lotebo-lab/broken-link-auditor/examples/find-dead-outbound-links-on-blog): checks every external link your articles point to and returns the ones that died, with the source page.
- [How do I find every broken link on my website?](https://apify.com/lotebo-lab/broken-link-auditor/examples/check-a-whole-website-for-404-links): crawls one domain from its home page and returns one row per broken link, with the HTTP status, the page the link sits on and the anchor text to search for in your CMS.
- [How do I find dead outbound links in my old blog posts?](https://apify.com/lotebo-lab/broken-link-auditor/examples/find-dead-outbound-links-in-blog-posts): crawls your site and checks the links that point to other domains, so you see which external pages your posts still send readers to and which ones are gone.
- [How do I check the links on one page for 404s?](https://apify.com/lotebo-lab/broken-link-auditor/examples/check-the-links-on-a-single-page): checks every link on a single URL, internal and external, without crawling the rest of the site.
- [How do I tell a dead link from a slow one?](https://apify.com/lotebo-lab/broken-link-auditor/examples/tell-a-dead-link-from-a-slow-one): separates the HTTP status from the network error, so a slow server shows up as a read timeout instead of a 404.
- [How do I check for broken links without overloading my site?](https://apify.com/lotebo-lab/broken-link-auditor/examples/crawl-my-site-slowly-without-overloading-it): one request at a time with the delay you choose, and a longer Crawl-delay in robots.txt always wins over it.

# Actor input Schema

## `startUrl` (type: `string`):

The site to audit. A bare domain is enough (example.com) and https:// is assumed. You can also give a full URL to start from a specific page.

## `maxPages` (type: `integer`):

How many pages of the site the crawler opens. Links found on those pages are all checked, even when the crawl stops. Raise this for a full site, keep it low for a quick sample.

## `maxDepth` (type: `integer`):

How many clicks away from the start page the crawler goes. 0 checks only the links on the start page.

## `requestDelaySeconds` (type: `number`):

Seconds to wait between two requests to the same host. The default 0.25 means at most 4 requests per second to your own site, which keeps a full-site audit inside the run time limit. Raise it if you want the audit to be slower on the server. If the site's robots.txt asks for a longer Crawl-delay, the longer value always wins over this one.

## `requestTimeoutSeconds` (type: `integer`):

Seconds to wait for a response before the link is reported as a timeout.

## `concurrency` (type: `integer`):

How many link targets are checked in parallel at the end of the run. The delay above is still kept between two requests to the same host, so a higher number only helps when the links point to many different sites. Lower it to 1 for a strictly one-at-a-time audit.

## `checkExternalLinks` (type: `boolean`):

Check outbound links too. External sites are never crawled, only their status is checked.

## `includeSubdomains` (type: `boolean`):

Treat blog.example.com as part of example.com and crawl it too.

## Actor input object example

```json
{
  "startUrl": "https://books.toscrape.com",
  "maxPages": 50,
  "maxDepth": 3,
  "requestDelaySeconds": 0.25,
  "requestTimeoutSeconds": 15,
  "concurrency": 8,
  "checkExternalLinks": true,
  "includeSubdomains": false
}
```

# Actor output Schema

## `brokenLinks` (type: `string`):

One item per broken link (rowType broken-link): target URL, HTTP status or network error, the page it was found on and the anchor text. The last item of every run is the rowType summary row with the run totals.

## `summary` (type: `string`):

Pages visited, links found, URLs checked, how many were broken and how many events were charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://books.toscrape.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("lotebo-lab/broken-link-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://books.toscrape.com" }

# Run the Actor and wait for it to finish
run = client.actor("lotebo-lab/broken-link-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://books.toscrape.com"
}' |
apify call lotebo-lab/broken-link-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lotebo-lab/broken-link-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/9lpuiqQHCo8U4jDnw/builds/h25Yhl7xkNx0vHveg/openapi.json