# Broken Link Checker & Internal Link Graph (Fast, No Browser) (`fast_data_asap/broken-link-internal-link-checker`) Actor

Crawl a website over plain HTTP and find broken links and images (404, 5xx, timeouts, DNS errors), redirect chains, http links and links to noindex pages, plus an internal link graph per page: inlinks, outlinks, canonical, orphan pages. $1 per 1,000 pages.

- **URL**: https://apify.com/fast_data_asap/broken-link-internal-link-checker.md
- **Developed by:** [Fast Data](https://apify.com/fast_data_asap) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 crawled pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker & Internal Link Graph (Fast, No Browser)

Crawl any website and find **broken links, broken images, redirect chains, insecure `http://` links and links to `noindex` pages**. You also get an **internal link graph** for every crawled page: inlinks, internal and external outlinks, title, canonical, noindex and **orphan-page candidates** from the sitemap.

It crawls over plain HTTP (async `httpx`, no headless browser), so it is fast and cheap. A 10-page crawl that checks 330+ unique links finishes in about 10 seconds.

### What it does

1. Starts from one or more **start URLs**, reads `robots.txt` (respects `Disallow` and `Crawl-delay`) and the **sitemap**.
2. Crawls internal pages breadth-first up to **Max pages** and **Max depth**, with polite concurrency.
3. Parses every `<a href>`, `<img src>`, `<link rel="canonical">`, `<meta name="robots">` and `<base href>`.
4. Checks every unique link target **once**: `HEAD` first, `GET` fallback when a server rejects HEAD. Redirects are followed by hand, so the full **redirect chain** is recorded. External hosts are rate-limited per host.
5. Writes three kinds of rows (`rowType`):
   - **`page`**: one row per crawled page, with `url`, `statusCode`, `depth`, `title`, `h1`, `canonical`, `canonicalIsSelf`, `noindex`, `inlinks`, `outlinksInternal`, `outlinksExternal`, `imagesOnPage`, `brokenLinksOnPage`, `inSitemap`, `isOrphanCandidate`.
   - **`link`**: **problems only**, one row per problem link on a page, with `sourcePage`, `targetUrl`, `anchorText`, `linkType` (link/image), `statusCode`, `redirectChain`, `problem` and `problems`.
   - **`summary`**: one final row with pages crawled, links checked, broken links by type and the **top 10 pages with the most broken links**.

#### Problems detected

| `problem` | Meaning |
|---|---|
| `broken_4xx` / `broken_5xx` | Target answered 404, 410, 500, 503 and so on |
| `timeout` | No answer within the timeout |
| `dns_error` | Domain does not resolve (expired or mistyped domain) |
| `connection_error` / `ssl_error` / `too_many_redirects` | Connection refused, bad certificate, redirect loop |
| `redirect_chain` | 2 or more redirect hops before the final URL |
| `redirect_to_other_host` | Link redirects to a different domain (www and non-www count as the same domain) |
| `http_link_on_https_page` | Insecure `http://` link or image on an `https://` page (mixed content) |
| `links_to_noindex` | Link points to a page marked `noindex` (meta robots or `X-Robots-Tag`) |

### Who it's for

- **SEO specialists and agencies**: regular broken-link audits, redirect cleanup and internal-linking analysis for clients.
- **Webmasters and shop owners** (Shopify, WooCommerce, WordPress) who want to find 404s before Google and customers do.
- **Content teams** after a migration or URL change.
- **Developers**: add a link check to CI/CD or a scheduled monitor through the API.
- **AI agents**: flat JSON rows are easy to read and act on ("fix the 12 broken links on /blog/").

### Why use it instead of the alternatives

- **Cheap**: $0.001 per crawled page, and every link on that page is checked at no extra cost. Comparable audit actors charge up to $0.04 per page.
- **Fast**: no browser. Async HTTP, link targets deduplicated across the whole site, HEAD requests where possible.
- **Problem rows only**: you don't get thousands of "200 OK" rows to filter. Healthy links are counted in the summary.
- **Link graph included**: inlinks and outlinks per page, plus orphan candidates (in the sitemap but not linked internally). Most link checkers don't return these.
- **Polite**: respects robots.txt and Crawl-delay, rate-limits each external host, and sends an identifying User-Agent.

### Input example

```json
{
  "startUrls": ["https://books.toscrape.com/"],
  "maxPages": 10,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "checkExternalLinks": true,
  "checkImages": true,
  "respectRobotsTxt": true,
  "excludePatterns": ["\\?sort=", "/tag/"]
}
```

Other options: `includePatterns` (regex), `includeSitemapUrls` (also crawl sitemap URLs, for real orphan-page detection), `checkUncrawledInternalLinks`, `maxConcurrency` (default 5), `maxRequestsPerExternalHost` (default 2), `maxLinkChecks`, `requestTimeoutSecs`, `userAgent` and `proxyConfiguration`.

### Output example

These values come from a real run on `https://books.toscrape.com/` (10 pages, 333 unique URLs checked, 10.4 s):

```json
{
  "rowType": "page",
  "url": "https://books.toscrape.com/",
  "statusCode": 200,
  "status": "ok",
  "depth": 0,
  "title": "All products | Books to Scrape - Sandbox",
  "h1": "All products",
  "canonical": null,
  "noindex": false,
  "inlinks": 0,
  "outlinksInternal": 73,
  "outlinksExternal": 0,
  "imagesOnPage": 20,
  "brokenLinksOnPage": 0,
  "inSitemap": null,
  "isOrphanCandidate": false
}
```

```json
{
  "rowType": "summary",
  "pagesCrawled": 10,
  "pagesOk": 10,
  "uniqueLinksChecked": 333,
  "linkOccurrences": 1087,
  "brokenLinks": 0,
  "broken4xx": 0,
  "broken5xx": 0,
  "timeouts": 0,
  "dnsErrors": 0,
  "redirectChains": 0,
  "topPagesByBrokenLinks": [],
  "robotsTxtFound": false,
  "sitemapFound": false,
  "crawlComplete": false,
  "durationSecs": 5.5
}
```

The demo site has no broken links. On a site that has some, a `link` row looks like this:

```json
{
  "rowType": "link",
  "problem": "redirect_chain",
  "problems": ["redirect_chain"],
  "sourcePage": "https://example.com/",
  "targetUrl": "https://example.com/r1",
  "linkType": "link",
  "isInternal": true,
  "anchorText": "Old pricing page",
  "statusCode": 200,
  "finalUrl": "https://example.com/pricing",
  "redirectHops": 2,
  "redirectChain": "https://example.com/r1 [301] -> https://example.com/r2 [301] -> https://example.com/pricing [200]",
  "error": null
}
```

The dataset has three views: **Overview**, **Link problems** and **Pages & internal link graph**. The summary is also stored as `OUTPUT` in the key-value store.

### Pricing

Pay per event: **$0.001 per crawled page** ($1 per 1,000 pages). Checking links and images, the summary and pages that fail (network errors, 4xx and 5xx pages) are **free**. Platform usage is minimal: the 10-page demo run used about $0.0001 of compute at 512 MB.

### Usage via API, n8n, Make, MCP

**API** (run synchronously and get the dataset items):

```bash
curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~broken-link-internal-link-checker/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://your-site.com/"],"maxPages":200}'
```

**Python**:

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
run = client.actor("fast_data_asap/broken-link-internal-link-checker").call(run_input={"startUrls": ["https://your-site.com/"], "maxPages": 200})
problems = [i for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i["rowType"] == "link"]
```

**n8n / Make / Zapier**: use the Apify node or app, choose this actor, then filter rows by `rowType = link` and send them to Slack, Sheets or your ticket system. Add a **schedule** in Apify to monitor your site weekly.

**MCP (Claude, Cursor and other AI agents)**: add `https://mcp.apify.com?tools=fast_data_asap/broken-link-internal-link-checker` as an MCP server.

### Notes and limits

- The crawler only follows links in the HTML. Links that JavaScript adds after page load are not seen.
- Some sites block bots or rate-limit `HEAD` requests. The checker falls back to `GET`, but a few sites still answer 403 or 429 to any automated client. Treat such results as "check manually".
- Only public pages are crawled. No logins, and no captcha or bot-protection bypass.

# Actor input Schema

## `startUrls` (type: `array`):

One or more pages to start crawling from, e.g. your home page. The crawl stays on these sites.

## `maxPages` (type: `integer`):

Maximum number of internal pages to crawl (each crawled page is one billable event). All links found on these pages are checked.

## `maxDepth` (type: `integer`):

How many clicks away from the start URLs the crawler goes. 0 = only the start URLs.

## `sameDomainOnly` (type: `boolean`):

Only crawl pages on the exact host of the start URLs (www and non-www count as the same). Turn off to also crawl subdomains such as blog.example.com.

## `checkExternalLinks` (type: `boolean`):

Check links to other websites (HEAD request, falls back to GET). Requests are rate-limited per host.

## `checkImages` (type: `boolean`):

Also check <img> sources for broken images.

## `checkUncrawledInternalLinks` (type: `boolean`):

Internal links that were not crawled (because of Max pages or Max depth) are still checked with a light HEAD request. Not billed.

## `includeSitemapUrls` (type: `boolean`):

Adds URLs from sitemap.xml to the crawl queue (after linked pages). Needed for meaningful orphan-page detection (in sitemap, but no internal links).

## `includePatterns` (type: `array`):

Optional regular expressions. If set, only matching internal URLs are crawled (start URLs are always crawled). Example: /blog/

## `excludePatterns` (type: `array`):

Optional regular expressions. Matching internal URLs are not crawled. Example: ?sort= or /tag/

## `respectRobotsTxt` (type: `boolean`):

Skip internal URLs disallowed by robots.txt and honor its Crawl-delay.

## `maxConcurrency` (type: `integer`):

Parallel requests to the crawled site. Keep it low for small servers.

## `maxRequestsPerExternalHost` (type: `integer`):

Politeness limit for external link checks (one host is never hit by more than this many parallel requests).

## `maxLinkChecks` (type: `integer`):

Upper limit of unique link/image URLs checked (on top of crawled pages).

## `requestTimeoutSecs` (type: `integer`):

A link that does not answer within this time is reported as a timeout.

## `userAgent` (type: `string`):

User-Agent header sent with every request. The default identifies the checker so site owners can allow it.

## `proxyConfiguration` (type: `object`):

Optional. Not needed for most sites.

## Actor input object example

```json
{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "maxPages": 10,
  "maxDepth": 3,
  "sameDomainOnly": true,
  "checkExternalLinks": true,
  "checkImages": true,
  "checkUncrawledInternalLinks": true,
  "includeSitemapUrls": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "maxRequestsPerExternalHost": 2,
  "maxLinkChecks": 2000,
  "requestTimeoutSecs": 15,
  "userAgent": "Mozilla/5.0 (compatible; FastDataLinkChecker/1.0; +https://apify.com/fast_data_asap/broken-link-internal-link-checker)"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://books.toscrape.com/"
    ],
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("fast_data_asap/broken-link-internal-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://books.toscrape.com/"],
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("fast_data_asap/broken-link-internal-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://books.toscrape.com/"
  ],
  "maxPages": 10
}' |
apify call fast_data_asap/broken-link-internal-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fast_data_asap/broken-link-internal-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Jct7Efl9k6rIw6isw/builds/JB3VPpfKkfp3P03p9/openapi.json