# Broken Link Checker & Sitemap Validator (Monitor Mode) (`mrdexter4/broken-link-monitor`) Actor

Find broken links, 404s, redirects and sitemap errors on your site. A 4xx is confirmed with GET, temporary errors are retried and bot-blocked links are re-checked in a real browser. Optional monitor mode flags only new issues. No subscription: pay per page and per link.

- **URL**: https://apify.com/mrdexter4/broken-link-monitor.md
- **Developed by:** [Mateus Correia](https://apify.com/mrdexter4) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 page crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker & Sitemap Validator (Monitor Mode)

**Find broken links, 404s, redirects and sitemap errors. In monitor mode, get alerted only about new ones.**

This broken link checker crawls your site, status-checks the links, images, scripts and stylesheets on the pages it crawls (up to `maxLinksToCheck` unique URLs), and validates your XML sitemap. A 4xx is confirmed with a GET request before it's reported, and 5xx, 408, 429 and network errors are retried. Bot-blocked links are re-checked in a real browser; if they still can't be verified, they're a warning, not an error.

**Use cases**

- **SEO agencies:** a weekly link and sitemap check per client, with an HTML report.
- **Site owners:** catch dead links after a migration or a deleted page.
- **Developers:** run it after a deploy and fail the pipeline when `errors > 0`.

**Sample output** (shortened, real run of the example input: `web-scraping.dev` with limits of 20 pages, 300 link checks and 50 sitemap URLs):

```json
{
  "category": "link",
  "issueType": "broken-link",
  "severity": "error",
  "url": "https://web-scraping.dev/api/graphql",
  "statusCode": 404,
  "explanation": "HTTP 404 Not Found.",
  "foundOnCount": 18,
  "firstFoundOn": "https://web-scraping.dev/",
  "anchorText": "Graphql",
  "checkedWith": "HTTP GET"
}
```

That run took 27 seconds: 19 pages crawled, 108 link checks (85 page links + 23 sitemap URLs not already checked; the other 27 sitemap URLs reused earlier results at no charge). Cost: 19 × $0.002 + 108 × $0.0005 = **$0.092**.

**No subscription.** You pay $0.002 per page crawled and $0.0005 per link checked, only when it runs.

### What does it check?

| Result | What it means | How it's verified |
|---|---|---|
| `broken-link` | 4xx such as 404, 410 or 400 | HEAD first; any failure is re-checked with GET |
| `server-error` | 5xx that keeps failing | Retried with backoff; `Retry-After` is respected |
| `unreachable` | DNS, connection or certificate failure | Re-checked; reason given in plain language |
| `blocked-unverified` | Possible bot protection (403, 429, 999, Cloudflare). Not proof the link is broken | Retried, then the first 25 (by default) are opened in headless Chrome. A browser 404/410 is reported as broken; the rest stay a **warning** |
| `soft-404` | Answers 200 but shows a "not found" page | Compared with your site's 404 response and "not found" titles in several languages |
| `redirect`, `redirect-chain`, `redirect-loop` | The link redirects, once, several times or in a loop | Every hop is recorded |
| `login-required`, `malformed-url` | HTTP 401 (info only); an `href` that isn't a valid URL | – |

Each unique URL is checked once per run, including sitemap URLs already checked as links. Every row lists where the link was found (up to 25 pages) and how it was checked (`checkedWith`).

### Sitemap validator

It finds sitemaps via `robots.txt` or common locations, follows indexes and `.gz` files, and reports:

- Sitemap URLs that are **broken, redirect, are soft 404s, need a login** or are **blocked by robots.txt**.
- **`noindex` pages in the sitemap**, which send search engines contradictory signals, and **canonicalised URLs** whose canonical tag points elsewhere.
- **Orphan pages** (in the sitemap, but nothing links to them) and **pages missing from the sitemap**.
- File problems: missing sitemap, no `<urlset>`/`<sitemapindex>` root, over 50,000 URLs or 50 MB uncompressed, bad `<lastmod>`, duplicates, a broken `Sitemap:` line in robots.txt.

It checks structure, not the full XML schema. Reading stops at `maxSitemapUrls`. URLs pointing to another host are flagged, not checked.

### Input

| Field | Description | Default |
|---|---|---|
| `startUrls` | Your site, usually the homepage. Other sites are checked, not crawled. | – |
| `maxPagesToCrawl` | Internal pages to crawl for links | 100 |
| `maxLinksToCheck` | Cap on unique URL checks (links, resources and sitemap URLs) | 2,000 |
| `checkExternalLinks`, `checkResources` | Also check links to other sites, and images, scripts and stylesheets | `true` |
| `validateSitemap`, `sitemapUrls`, `maxSitemapUrls` | Validate the sitemap; extra sitemap URLs; how many sitemap URLs to read | `true`, –, 500 |
| `verifyBlockedInBrowser`, `maxBrowserChecks` | Re-check bot-blocked links in headless Chrome (free) | `true`, 25 |
| `renderJavaScript` | Find links added by JavaScript (slower; use 2 GB+ memory) | `false` |
| `reportRedirects` | Which redirecting links to report: `internal`, `all` or `none` | `internal` |
| `excludeUrlPatterns` | Skip URLs matching these regular expressions or plain text | – |
| `monitorMode`, `monitorStoreName`, `outputOnlyNewIssues`, `alertOn` | Compare with the previous run; output or alert on new issues only | off, `new-errors` |
| `webhookUrl` | Slack, Discord, n8n, Make or Zapier webhook | – |
| `notificationEmail` | Optional email alert | – |
| `maxConcurrency`, `requestTimeoutSecs`, `proxyConfiguration` | Advanced | 16, 20, none |

The Console form is prefilled with smaller limits (20 pages, 300 link checks, 50 sitemap URLs) for a quick first try; the defaults apply when a field is empty. Several start URLs share one budget, so use one run or Task per site.

### Output

One dataset row per issue, sorted by severity, then a `category: "summary"` row with the totals (`pagesCrawled`, `linksChecked`, `errors`, `brokenLinks`, `crawlComplete`, and in monitor mode `newIssues` and `resolvedIssues`). It's always added, even when nothing is wrong.

The key-value store holds `report.html` (a shareable report) and `OUTPUT` (the summary as JSON).

### Monitor mode and scheduled checks

Monitor mode is off by default. For a scheduled check that reports only changes:

1. Turn on **Monitor mode** and save the input as a **Task**, one per site.
2. In Apify Console, open **Schedules** and add the Task with a cron expression, for example `0 6 * * 1` for Mondays at 06:00.
3. Add a `webhookUrl`. With `alertOn: new-errors` (the default), an alert is sent when a new error appears or when an earlier issue is resolved.

Each issue gets `isNew` and `firstSeenAt`. **The first run is the baseline: every issue on it is new, so it alerts on every error.** Set `maxPagesToCrawl` above your site's page count: when the limit is hit, different pages can be crawled each run and issues can flip between new and resolved. See [Apify schedules](https://docs.apify.com/platform/schedules).

### Use it from n8n, Make and Zapier

Apify's official integrations can **run a Task**, **get its dataset items**, or start a workflow **when a run finishes**. For example:

- **Client sheet:** append the `summary` row's totals to a Google Sheet.
- **Ticket per new broken link:** with monitor mode on, filter rows with `isNew: true` and `severity: "error"` and create a Jira, Linear or Trello card. Skip the first (baseline) run, or it will open a card for every existing issue.
- **After a deploy:** run the Task and stop the release when the summary has `errors > 0`.

The webhook sends JSON with a `text` field (Slack, n8n, Make, Zapier) and a `content` field (Discord). Guides: [n8n](https://docs.apify.com/platform/integrations/n8n), [Make](https://docs.apify.com/platform/integrations/make), [Zapier](https://docs.apify.com/platform/integrations/zapier).

### Pricing (pay per event)

| Event | Price |
|---|---|
| Page crawled (`page-crawled`): an HTML page on your site downloaded and scanned for links | **$0.002** |
| Link checked (`link-checked`): each unique URL status-checked, including sitemap URLs | **$0.0005** |

Crawled pages aren't also charged as link checks. Retries, browser re-checks, the summary row and the report are free.

**Example:** a 100-page site with 2,000 unique URLs (the 100 pages plus 1,900 other links, images, scripts and sitemap URLs) costs 100 × $0.002 + 1,900 × $0.0005 = **$1.15 per check**. That's about $5 a month weekly or $35 a month daily. With the default limits one site costs at most about $1.20 per run. Set **Maximum cost per run** for a hard cap; the Actor stops cleanly when it's reached.

### FAQ

**Will it report links that actually work?**
It tries not to, but bot protection makes some links impossible to verify. Those are warnings to check by hand; exclude sites that always block it.

**I run a shop. Why are there so many URLs?**
Each query-string variant (`?sort=`, filters, `?add-to-cart=`) is a separate URL and is charged separately. Add patterns such as `\?sort=`, `\?filter` or `add-to-cart` to `excludeUrlPatterns`.

**Do I need email alerts?**
No, they're optional. If you enter an address, the alert is sent by Apify's `apify/send-mail` Actor on your own account, which uses a little platform credit.

### Limitations

- Crawling stays on the start URL's site. Other subdomains are checked as links but not crawled.
- Links built by JavaScript are only found with `renderJavaScript`. Images referenced in CSS `url()` aren't checked.
- Every 403 is treated as possible bot blocking and reported as `blocked-unverified`, even when the page really is forbidden.
- Bursts of 403 or 429 answers from big external sites are reported as warnings. The Actor doesn't yet slow down automatically for those sites.
- Links are checked before sitemap URLs, so when `maxLinksToCheck` runs out, remaining sitemap URLs are skipped.
- Orphan pages are only reported when the whole site was crawled and the sitemap fully read; missing-from-sitemap needs a fully read sitemap.

### Keywords

Broken link checker, dead link checker, 404 checker, sitemap validator, link monitor, redirect checker, soft 404 checker.

# Actor input Schema

## `startUrls` (type: `array`):

Start page(s), usually the homepage. Pages on the same domain are crawled; links to other domains are checked but not crawled. Several start URLs share one page and link budget, so use one run or Task per site.

## `maxPagesToCrawl` (type: `integer`):

Internal HTML pages to crawl for links. Internal links beyond this limit are still status-checked, just not crawled.

## `maxLinksToCheck` (type: `integer`):

Safety cap on unique link/resource/sitemap URL checks per run.

## `checkExternalLinks` (type: `boolean`):

Also check links to other websites (not crawled, only checked).

## `checkResources` (type: `boolean`):

Check <img>, srcset, <script>, stylesheet and icon URLs too.

## `validateSitemap` (type: `boolean`):

Find the sitemap (robots.txt, /sitemap.xml, sitemap indexes, .gz) and check every listed URL: broken, redirecting, noindex, canonicalised elsewhere, blocked by robots.txt. Also reports crawled pages missing from the sitemap and orphan sitemap pages.

## `sitemapUrls` (type: `array`):

Only needed if your sitemap is not listed in robots.txt or at /sitemap.xml.

## `maxSitemapUrls` (type: `integer`):

Sitemap reading stops after this many URLs. Crawled pages and URLs already checked as links are reused; the rest are fetched once each (counted as link checks).

## `verifyBlockedInBrowser` (type: `boolean`):

Links that answer 403/429/999 or a Cloudflare challenge are opened in real Chrome before anything is reported. Working links are dropped; the rest are reported as 'blocked-unverified', never as broken. No extra charge.

## `maxBrowserChecks` (type: `integer`):

Upper limit for real-browser re-checks per run.

## `renderJavaScript` (type: `boolean`):

Load each crawled page in Chrome and collect links from the rendered DOM too. Slower; use 2048 MB+ memory.

## `reportRedirects` (type: `string`):

Which redirecting links to report. Redirects are never errors; internal redirect chains are warnings.

## `excludeUrlPatterns` (type: `array`):

URLs matching any pattern are neither crawled nor checked, e.g. "/wp-admin/", "linkedin\\.com", "\\?add-to-cart=", "\\?sort=". Shops: query-string variants are separate URLs and are charged separately.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout. Timeouts are retried with backoff before a link is reported.

## `monitorMode` (type: `boolean`):

Remember issues between runs (same start URLs + store name). Each issue gets isNew and firstSeenAt; fixed issues are listed as resolved. Schedule the Actor daily or weekly. Set 'Max pages to crawl' above your site's page count, otherwise issues can flip between new and resolved.

## `monitorStoreName` (type: `string`):

Named key-value store that keeps the history. Use different names for separate monitors of the same site.

## `outputOnlyNewIssues` (type: `boolean`):

In monitor mode, put only new issues in the dataset (the summary row is always added).

## `alertOn` (type: `string`):

What triggers the webhook/email. Alerts are also sent when an earlier issue is resolved. In monitor mode the first run is the baseline, so every error on it counts as new. Without monitor mode, 'new' means all current issues.

## `webhookUrl` (type: `string`):

POSTs JSON with a 'text' field (Slack, n8n, Make, Zapier), a 'content' field (Discord) and the full issue list, only when there is something to report.

## `notificationEmail` (type: `string`):

Optional. Sends a short HTML report when there is something to report, using the apify/send-mail Actor on your account (uses a little platform credit).

## `maxConcurrency` (type: `integer`):

Parallel requests overall. Per host it is always limited (4 for your site, 2 for each external site) to avoid rate limits and server load.

## `proxyConfiguration` (type: `object`):

Not needed for most sites. Use it if your own site's firewall blocks data-centre IPs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://web-scraping.dev/"
    }
  ],
  "maxPagesToCrawl": 20,
  "maxLinksToCheck": 300,
  "checkExternalLinks": true,
  "checkResources": true,
  "validateSitemap": true,
  "maxSitemapUrls": 50,
  "verifyBlockedInBrowser": true,
  "maxBrowserChecks": 25,
  "renderJavaScript": false,
  "reportRedirects": "internal",
  "requestTimeoutSecs": 20,
  "monitorMode": false,
  "monitorStoreName": "broken-link-monitor-history",
  "outputOnlyNewIssues": false,
  "alertOn": "new-errors",
  "maxConcurrency": 16
}
```

# Actor output Schema

## `results` (type: `string`):

One row per issue (broken link, redirect, sitemap problem) plus a final summary row.

## `report` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://web-scraping.dev/"
        }
    ],
    "maxPagesToCrawl": 20,
    "maxLinksToCheck": 300,
    "maxSitemapUrls": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("mrdexter4/broken-link-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://web-scraping.dev/" }],
    "maxPagesToCrawl": 20,
    "maxLinksToCheck": 300,
    "maxSitemapUrls": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("mrdexter4/broken-link-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://web-scraping.dev/"
    }
  ],
  "maxPagesToCrawl": 20,
  "maxLinksToCheck": 300,
  "maxSitemapUrls": 50
}' |
apify call mrdexter4/broken-link-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mrdexter4/broken-link-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eh6SuJPlSCJiyIpHc/builds/hFcyFjessuJ4wVSWT/openapi.json