# Broken Link Checker — Crawl Sites & Find 404s (`hoanglongteam/broken-link-checker`) Actor

Crawl each website (same domain, respects robots.txt), check every internal and external link, and get the broken ones (404, 5xx, timeouts, DNS errors) with the page where they were found, anchor text, plus one summary row per site.

- **URL**: https://apify.com/hoanglongteam/broken-link-checker.md
- **Developed by:** [LONG HOÀNG](https://apify.com/hoanglongteam) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker — Site Crawler with Source Pages & Anchor Text

Crawl one or many websites and get a clean list of **every broken link**: the broken URL, **the page where it was found**, **its anchor text**, the HTTP status or network error, plus **one summary row per site**.

- Crawls **same-domain pages** (www and non-www count as one site) up to `maxPagesPerSite`.
- Checks **internal and external links** (and optionally images): `HEAD` first, automatic `GET` fallback for servers that mishandle HEAD.
- Reports **4xx, 5xx, timeouts, DNS failures, TLS/SSL errors, redirect loops** and malformed URLs. Redirects are followed and the final URL is shown.
- Flags links that are **probably bot-blocking rather than broken** (401, 403, 429, 999 — common on LinkedIn, Stack Exchange, marketplaces), so you can filter them out.
- **Polite by default**: respects `robots.txt` (including `Crawl-delay`), at most 2 parallel requests per host, a delay between requests, and automatic slow-down when a server answers 429.

### Who it's for

- **SEO specialists & agencies** auditing client sites for 404s that waste crawl budget and hurt UX.
- **Content teams** cleaning dead outbound links from blogs and documentation.
- **Developers** running a link check after a migration or redesign (schedule it or call it via API).

### Input

```json
{
  "startUrls": ["example.com", "https://docs.example.com/guide/"],
  "maxPagesPerSite": 50,
  "checkExternalLinks": true,
  "checkImages": false,
  "maxLinksPerSite": 2000,
  "maxConcurrency": 10,
  "delayMs": 200,
  "timeoutSecs": 15
}
```

### Output

Two kinds of rows in the dataset (two views in the Output tab: **Broken links** and **Per-site summary**).

**`type: "broken-link"`** — one row per broken URL per site:
`site, url, statusCode, errorType (http-4xx | http-5xx | rate-limited | timeout | dns | tls | connection | redirect-loop | invalid-url), error, linkType (internal | external), elementType (link | image), foundOn, anchorText, occurrences, foundOnPages[{page, anchorText}] (up to 10), possiblyBlocked, finalUrl, checkMethod, checkedAt`

**`type: "site-summary"`** — one row per start URL:
`site, startUrl, pagesCrawled, linksFound, linksChecked, internalLinks, externalLinks, brokenLinks, brokenConfirmed, brokenInternal, brokenExternal, possiblyBlocked, skippedByRobots, skippedOverLimit, robotsTxt, crawlDelaySecs, durationSecs, error`

Export as JSON, CSV or Excel, or read it through the API.

### Pricing (pay per event)

- **Page crawled**: each HTML page fetched and parsed on your site.
- **Link checked**: each unique link URL checked. Links to pages already crawled are not charged twice.
- Output rows are free. A site that cannot be reached is not charged for pages.
  Set a maximum cost per run in Apify and the Actor stops cleanly when it is reached.

### How it works / what it does not do

Plain HTTP requests, no browser and no proxies. It only reads public technical information (URLs, status codes, link text). It does **not** collect emails, phone numbers or other personal data (`mailto:` and `tel:` links are skipped), and it does not log in or solve CAPTCHAs.

### Limitations

- Links created only by JavaScript are not seen (no browser rendering).
- Sites behind Cloudflare/bot protection may return 403 for every link; these are flagged `possiblyBlocked`.
- Internal URLs disallowed by robots.txt are not fetched and not reported (counted in `skippedByRobots`).
- Servers that send malformed HTTP responses which browsers still display are not reported as broken.
- Results reflect what the checker's network sees: a host that is down only from one region may show as a timeout.
- External links are checked with a single request each; robots.txt of external sites is not consulted for that one check.

# Actor input Schema

## `startUrls` (type: `array`):

Start URLs or domains, e.g. example.com or https://example.com/blog/. The crawler stays on the same domain (www and non-www count as the same site).

## `maxPagesPerSite` (type: `integer`):

Maximum number of HTML pages to crawl on each site. Every link found on these pages is checked.

## `checkExternalLinks` (type: `boolean`):

Also check links pointing to other domains (one HEAD/GET request each). Turn off to check internal links only.

## `checkImages` (type: `boolean`):

Also check <img src> URLs, not just <a href> links.

## `maxLinksPerSite` (type: `integer`):

Safety cap on unique link URLs checked per site (internal + external). Also caps cost, since each unique link checked is one charged event.

## `maxConcurrency` (type: `integer`):

Maximum number of HTTP requests in flight at once across all sites. Requests to the same host are additionally limited to 2 at a time.

## `delayMs` (type: `integer`):

Polite pause between two requests to the same host. A robots.txt Crawl-delay (up to 10 s) overrides this when larger, and the delay is raised automatically if a host answers 429 Too Many Requests.

## `timeoutSecs` (type: `integer`):

Max seconds to wait for each request before reporting the link as a timeout.

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "maxPagesPerSite": 10,
  "checkExternalLinks": true,
  "checkImages": false,
  "maxLinksPerSite": 2000,
  "maxConcurrency": 10,
  "delayMs": 200,
  "timeoutSecs": 15
}
```

# Actor output Schema

## `brokenLinks` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/"
    ],
    "maxPagesPerSite": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("hoanglongteam/broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://books.toscrape.com/"],
    "maxPagesPerSite": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("hoanglongteam/broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "maxPagesPerSite": 10
}' |
apify call hoanglongteam/broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hoanglongteam/broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RVDqBV3pBwsqWpEI4/builds/yYeQ7sx7hBpysaKjE/openapi.json