# Broken Link Checker Scraper (`automation-lab/sitewide-broken-link-checker`) Actor

Crawl public websites, test deduplicated internal and external links, and export status, redirect chains, source pages, anchor text, timing, and bounded errors.

- **URL**: https://apify.com/automation-lab/sitewide-broken-link-checker.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.48 / 1,000 item extracteds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Broken Link Checker Scraper

Run a sitewide **broken link checker** against one or more public websites. The Actor crawls bounded same-origin pages, tests each deduplicated internal and external link, follows redirects, and exports a typed audit dataset with HTTP status, redirect chain, source page, anchor text, response time, and bounded error details.

Use it after content changes, redesigns, migrations, or deployments to find link rot before visitors and search engines do.

### What does Broken Link Checker Scraper do?

For every supplied website, the Actor:

1. validates that the start URL resolves to the public internet;
2. fetches server-rendered HTML pages on the same origin;
3. discovers and normalizes HTTP and HTTPS links;
4. deduplicates targets while preserving source-page evidence;
5. reuses crawl responses for internal links already fetched;
6. tests other targets with `HEAD` and falls back to `GET` on 405/501;
7. records every redirect hop;
8. classifies each result as `ok`, `redirect`, `broken`, or `warning`;
9. writes one dataset row per tested target.

Successful links are included alongside failures. Filter `resultType = broken` for actionable dead links, or retain all rows as a release-quality baseline.

### Who is this website link audit for?

- **SEO teams** checking 404s, redirect chains, and external link rot.
- **Web developers** validating navigation after a deployment.
- **Content teams** finding stale references across documentation or blogs.
- **QA engineers** producing machine-readable link health evidence.
- **Agencies** running the same bounded audit for multiple client sites.
- **Data teams** loading recurring audit datasets into a warehouse or spreadsheet.

This Actor is designed for anonymously reachable public pages. It does not accept accounts, cookies, or private-network URLs.

### Why use this broken link check?

A browser extension checks only the page you open. This Actor performs a bounded site crawl and produces reusable data through the Apify Console, API, schedules, webhooks, datasets, and integrations.

Useful details include:

- the first page and anchor text where a target appeared;
- up to 20 distinct source pages per target;
- total occurrence count from the bounded crawl;
- final status and URL after redirects;
- every observed 301, 302, 303, 307, or 308 hop;
- internal versus external classification;
- bounded timeout, DNS, TLS, connection, and URL-safety errors;
- response time and final request method.

### What data does the Actor export?

| Field | Meaning |
| --- | --- |
| `targetUrl` | Deduplicated URL discovered in an anchor element |
| `finalUrl` | Last URL reached after redirects, or `null` on a network error |
| `sourcePage` | First crawled page containing the target |
| `sourcePages` | Up to 20 crawled pages containing the target |
| `anchorText` | Normalized text from the first anchor occurrence |
| `occurrenceCount` | Number of occurrences found during the bounded crawl |
| `linkType` | `internal` or `external` relative to the website root |
| `statusCode` | Final HTTP status, or `null` if no response arrived |
| `statusText` | HTTP status text |
| `redirectChain` | Ordered redirect URL, status, and destination objects |
| `redirectCount` | Number of observed redirects |
| `isRedirect` | Whether the target redirected |
| `isBroken` | Whether the target has an HTTP or confirmed network failure |
| `resultType` | `ok`, `redirect`, `broken`, or `warning` |
| `errorType` | Bounded network/safety error category |
| `errorMessage` | Diagnostic message capped at 500 characters |
| `responseTimeMs` | Total request and redirect time |
| `method` | `HEAD` or `GET` used for the result |
| `checkedAt` | ISO 8601 audit timestamp |

Fields that depend on an HTTP response are nullable because DNS, TLS, connection, and timeout failures may not produce a status.

### How to run your first sitewide link audit

1. Open the Actor input page.
2. Add a public website URL under **Websites to audit**.
3. Keep `maxPages` and `maxLinks` small for the first run.
4. Leave external checking enabled if off-site references matter.
5. Click **Start**.
6. Open the **Link audit** dataset view.
7. Filter `resultType` to `broken`, `redirect`, or `warning`.
8. Export the result as CSV, JSON, Excel, XML, or RSS.

A useful smoke-test input is:

```json
{
  "startUrls": [
    { "url": "https://crawler-test.com/links/broken_links_internal" }
  ],
  "maxPages": 6,
  "maxLinks": 40,
  "checkExternalLinks": false,
  "maxConcurrency": 6
}
```

### Input parameters

#### `startUrls`

Required array of 1–20 public HTTP(S) website roots. The Actor follows same-origin links from each root. URL credentials, localhost, and private/reserved network destinations are rejected.

#### `maxPages`

Maximum total same-origin HTML pages crawled across all roots. Default: `5`. Range: `1–1000`.

#### `maxLinks`

Maximum unique targets tested and exported. Default: `20`. Range: `1–5000`.

#### `checkExternalLinks`

When `true` (default), test off-origin links too. When `false`, export only internal targets.

#### `maxConcurrency`

Concurrent crawl/check requests. Default: `10`. Range: `1–30`. Reduce it for small or rate-sensitive websites.

#### `requestTimeoutSecs`

Per-request timeout. Default: `15` seconds. Range: `3–60`.

#### `maxRedirects`

Maximum redirect hops for each page or link. Default: `8`. Range: `0–15`.

### Example broken-link output

This representative record comes from the current crawler-test workflow:

```json
{
  "targetUrl": "https://crawler-test.com/links/not_found/foo1",
  "finalUrl": "https://crawler-test.com/links/not_found/foo1",
  "sourcePage": "https://crawler-test.com/links/broken_links_internal",
  "sourcePages": [
    "https://crawler-test.com/links/broken_links_internal"
  ],
  "anchorText": "Broken Internal Link 1",
  "occurrenceCount": 1,
  "linkType": "internal",
  "statusCode": 404,
  "statusText": "Not Found",
  "redirectChain": [],
  "redirectCount": 0,
  "isRedirect": false,
  "isBroken": true,
  "resultType": "broken",
  "errorType": null,
  "errorMessage": null,
  "responseTimeMs": 113,
  "method": "GET",
  "checkedAt": "2026-08-25T06:07:55.593Z"
}
```

Response times and timestamps vary by run.

### How result classifications work

- **`ok`** — a final response below HTTP 400 with no redirect.
- **`redirect`** — one or more redirect hops ended in a non-broken response.
- **`broken`** — HTTP 400+ or a confirmed DNS, TLS, connection, or unsafe-destination failure.
- **`warning`** — timeout or unknown request failure that does not confirm the target is permanently broken.

A 401 or 403 is classified as broken from the link-audit perspective because anonymous visitors cannot reach it. Review authorization-sensitive targets before removing them.

### Redirect and deduplication behavior

URL fragments are removed because they do not change the HTTP target. Query strings are preserved because they may produce different content or status. Each normalized target is tested once per run.

The Actor records the first anchor text, the first source page, up to 20 unique source pages, and a full occurrence count. This keeps records bounded without hiding how widely a link is used.

Redirect destinations are public-IP validated before each hop. Redirect loops or chains beyond `maxRedirects` become bounded warnings.

### How much does it cost to check website links?

The Actor uses pay-per-event pricing:

- one **Start** event per run;
- one **Item processed** event per tested, exported link target.

At the BRONZE rate of **$0.005 per start** and **$0.0008 per link**, examples are:

| Exported links | Estimated BRONZE calculation |
| ---: | ---: |
| 10 | 0.005 + (10 × 0.0008) = 0.013 USD |
| 100 | 0.005 + (100 × 0.0008) = 0.085 USD |
| 500 | 0.005 + (500 × 0.0008) = 0.405 USD |
| 1,000 | 0.005 + (1,000 × 0.0008) = 0.805 USD |

Apify plan tier and platform usage treatment can affect the final invoice. The Console displays the applicable event prices before each run. Set `maxLinks` to create a hard output and event bound.

### Recurring website QA workflow

Create an Apify Schedule that runs after a daily content publish or weekly maintenance window. Send the dataset to Google Sheets, Slack, a webhook, or a warehouse. Compare rows by `targetUrl`, `statusCode`, and `resultType` to detect new failures.

The Actor creates a fresh default dataset per run. It does not maintain historical diffs or send alerts itself; use Apify schedules, webhooks, or your downstream system for those steps.

### Website migration workflow

Before migration, save a baseline audit. After cutover:

1. run the same input against the new site;
2. filter for `broken` internal links;
3. inspect redirect chains for legacy paths;
4. group by `sourcePage` to assign fixes;
5. compare the new dataset against the baseline.

This catches navigation errors and incomplete redirects without requiring a browser session.

### cURL API example

Replace `APIFY_TOKEN` with your token:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~sitewide-broken-link-checker/runs?token=APIFY_TOKEN&waitForFinish=300" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://docs.apify.com/"}],
    "maxPages": 10,
    "maxLinks": 100,
    "checkExternalLinks": true
  }'
```

Fetch dataset items from the `defaultDatasetId` returned by the run API.

### JavaScript API example

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/sitewide-broken-link-checker').call({
    startUrls: [{ url: 'https://docs.apify.com/' }],
    maxPages: 10,
    maxLinks: 100,
    checkExternalLinks: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const broken = items.filter((item) => item.resultType === 'broken');
console.log(broken);
```

### Python API example

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/sitewide-broken-link-checker').call(run_input={
    'startUrls': [{'url': 'https://docs.apify.com/'}],
    'maxPages': 10,
    'maxLinks': 100,
    'checkExternalLinks': True,
})

items = client.dataset(run['defaultDatasetId']).list_items().items
broken = [item for item in items if item['resultType'] == 'broken']
print(broken)
```

### Use with Apify MCP

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/sitewide-broken-link-checker"
```

#### Claude Desktop setup

Add this server entry to the Claude Desktop MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/sitewide-broken-link-checker"
    }
  }
}
```

#### Cursor setup

Open Cursor MCP settings, add the same remote `apify` server URL, and enable the server for Agent mode.

#### VS Code setup

Add the same `apify` URL to the VS Code MCP configuration, start it from the MCP view, and confirm this Actor appears in the available tools.

Example prompts:

- “Audit the first 20 pages of this public documentation site and list broken internal links.”
- “Check external references, then group failures by source page.”
- “Show redirects with more than one hop and their final status.”

### Performance and cost tips

- Begin with 5–25 pages and 50–250 links.
- Disable external checking when only site navigation matters.
- Lower concurrency when a server starts returning 429 or 503.
- Increase timeout for slow but legitimate sites rather than treating a timeout as broken.
- Audit a focused section URL instead of the homepage when only one documentation area changed.
- Use repeated inputs and downstream comparisons for consistent regression checks.

### Limitations

- Only links present in server-rendered `<a href>` elements are discovered.
- JavaScript-generated links and browser-only navigation are not rendered.
- Authenticated pages, accounts, cookies, and login flows are unsupported.
- Private-network, localhost, credential-bearing, and non-HTTP(S) targets are blocked.
- `robots.txt` is not an access-control mechanism; users remain responsible for crawl permission and rate choices.
- Some servers reject automated `HEAD` and `GET` requests or return bot-specific statuses.
- Fragment-only failures inside a page are not checked because URL fragments are removed.
- The first 20 unique source pages are retained per target; `occurrenceCount` may be higher.
- A timeout is a warning, not proof that a link is permanently dead.

### Troubleshooting

#### The dataset is empty

Confirm the start page returns server-rendered HTML with ordinary anchor elements. An empty dataset means no supported HTTP(S) links were discovered under the selected scope. Check the run log for fetch or public-IP validation errors.

#### Many links return 403 or 429

Lower `maxConcurrency` and rerun a small sample. The Actor intentionally has no automatic residential-proxy fallback, so it cannot bypass a site that blocks anonymous direct requests.

#### An internal link appears external

Classification uses exact URL origin (scheme, hostname, and port). A link from `www` to a bare domain, or from HTTP to HTTPS, is external unless the page redirect already changed the audited root origin.

#### A working link is marked broken

Check whether it requires login, geography, browser JavaScript, cookies, or a specific user agent. Review 401/403 results manually. Increase the timeout only for timeout warnings.

### Responsible use and legality

Use the Actor only on public pages you are authorized to request. Respect site terms, applicable robots guidance, crawl-rate expectations, copyright, privacy, and computer-access laws. Do not use it to probe internal infrastructure or evade access controls. The public-IP validation is a safety boundary, not legal permission.

Dataset URLs and anchor text may contain personal or confidential information published by the target site. Apply appropriate retention and access controls to exports.

### Related Automation Lab Actors

- [Redirect Chain Analyzer](https://apify.com/automation-lab/redirect-chain-analyzer) for a focused list of known URLs.
- [Robots & Sitemap Analyzer](https://apify.com/automation-lab/robots-sitemap-analyzer) for crawl directives and sitemap structure.
- [Lighthouse Website Audit](https://apify.com/automation-lab/website-lighthouse-seo-audit) for performance, accessibility, best-practice, and SEO audits.

Choose this Actor when you need source-page and anchor context from a bounded site crawl. Choose Redirect Chain Analyzer when you already have the exact URL list and do not need link discovery.

### FAQ

#### Does it check both internal and external links?

Yes. External checking is enabled by default and can be disabled with `checkExternalLinks`.

#### Does it report healthy links?

Yes. One row is exported for every tested target, making the dataset suitable as a complete bounded audit baseline.

#### Can I audit several websites in one run?

Yes, up to 20 roots. `maxPages` and `maxLinks` are total run limits shared across them.

#### Does it follow redirects?

Yes, up to `maxRedirects`, with each hop stored in `redirectChain`.

#### Does it use a browser or proxy?

No. It uses direct HTTP and server-rendered HTML to keep runs lightweight and predictable.

#### Can it monitor changes automatically?

Schedule recurring Actor runs and compare datasets downstream. The Actor does not maintain history or send alerts by itself.

#### Is a timeout definitely a broken link?

No. Timeouts and unknown failures are `warning` results. Retry them separately before making a content decision.

# Actor input Schema

## `startUrls` (type: `array`):

One or more public HTTP(S) website roots. The crawler follows same-origin pages from each root.

## `maxPages` (type: `integer`):

Maximum total number of same-origin HTML pages to crawl across all supplied websites.

## `maxLinks` (type: `integer`):

Maximum number of unique link targets to test and export.

## `checkExternalLinks` (type: `boolean`):

Test links to other origins as well as links within each audited website.

## `maxConcurrency` (type: `integer`):

Number of concurrent page and link requests. Lower this for fragile websites.

## `requestTimeoutSecs` (type: `integer`):

Maximum seconds allowed for each HTTP request.

## `maxRedirects` (type: `integer`):

Maximum redirect hops traced for one page or link.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxPages": 5,
  "maxLinks": 20,
  "checkExternalLinks": true,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 15,
  "maxRedirects": 8
}
```

# Actor output Schema

## `dataset` (type: `string`):

Deduplicated link status, redirect, source-page, anchor, timing, and bounded error records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "maxPages": 5,
    "maxLinks": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/sitewide-broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/" }],
    "maxPages": 5,
    "maxLinks": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/sitewide-broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "maxPages": 5,
  "maxLinks": 20
}' |
apify call automation-lab/sitewide-broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/sitewide-broken-link-checker"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/D1VsA1x11YGaqHl5V/builds/iI58SfCLim2aD4neq/openapi.json