# Broken Link Checker for Your Sitemap (`isnard/broken-link-checker`) Actor

Scan the pages of your own website from its sitemap (or a same-site crawl) and find broken links, images, scripts and stylesheets. Each target checked once, robots.txt respected, results compared with the previous run.

- **URL**: https://apify.com/isnard/broken-link-checker.md
- **Developed by:** [Matias Isnard](https://apify.com/isnard) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 page scanneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Broken Link Checker for Your Sitemap

Find broken links, missing images and dead scripts or stylesheets on your own website.

Give it your homepage or your sitemap URL. The Actor reads the sitemap (sitemap indexes and `.xml.gz` files work), fetches each page once, collects every `<a href>`, `<img src>`, `<link href>` and `<script src>`, and checks each unique target once. You get one result per broken or problematic link, with the page it was found on, plus a summary. Run it again later and each problem is marked as `new` or `stillBroken`, and you see how many were fixed.

This Actor is meant for checking sites you own or manage. It respects robots.txt and keeps the request rate low.

### How it works

1. **Pages.** If `startUrl` looks like a sitemap (ends in `.xml` or `.xml.gz`, or has `sitemap` in the path), that sitemap is read. Otherwise the sitemaps listed in your robots.txt are used, or `/sitemap.xml` if robots.txt lists none. Only URLs on the same site are taken (`www.` is ignored, other subdomains are not the same site). Up to `maxPages`.
2. **Crawl fallback.** If no sitemap is found, the Actor starts at `startUrl` and follows `<a href>` links on the same site until `maxPages` pages have been queued.
3. **Links.** Each HTML page is downloaded with GET (up to 5 MB) and parsed. Links with `mailto:`, `tel:`, `javascript:`, `data:` and plain `#anchors` are ignored. Fragments are removed, so `/page#a` and `/page#b` are one target.
4. **Checks.** Each unique target is requested once with HEAD. If the server answers 403, 405 or 501 to HEAD, the Actor tries GET (without reading the body). Redirects are followed, up to 10. Pages already downloaded in step 3 are not requested again.

Requests identify themselves with the User-Agent `Mozilla/5.0 (compatible; BrokenLinkChecker/1.0; +https://apify.com/isnard/broken-link-checker)`. Timeout is 15 seconds per request.

### Being polite

- **robots.txt.** Pages and links on other websites are only requested when their robots.txt allows it, using the rules for `BrokenLinkChecker` or `*`. This cannot be turned off. On your own site it is also respected by default. Set `respectRobots` to `false` only if you own the site and want to check paths you hide from crawlers.
- **Rate limit.** At most 4 requests per second to your site, and at most 1 request per second to each other host. If your robots.txt has a `Crawl-delay`, that is used instead (up to 10 seconds).
- **Concurrency.** 4 requests in flight by default.

### Input

```json
{
    "startUrl": "https://example.com",
    "maxPages": 200,
    "checkExternal": true,
    "maxLinksPerPage": 500,
    "concurrency": 4,
    "respectRobots": true
}
```

- `startUrl` (required): your homepage or your sitemap URL. `https://` is added when there is no scheme.
- `maxPages` (default 200): pages of your site to scan.
- `checkExternal` (default true): also check links to other websites. With `false` they are counted in the summary but not requested.
- `maxLinksPerPage` (default 500): links taken from each page, in document order.
- `concurrency` (default 4, max 20): requests at the same time.
- `respectRobots` (default true): see above. Only affects your own site.

### Output

One dataset item per problem, that is, per pair of page and link. A missing logo linked from every page gives one item per page, so you see every place to fix.

Real example from a test run:

```json
{
    "type": "link",
    "sourcePage": "https://crawlee.dev/blog/authors",
    "targetUrl": "https://crawlee.dev/blog/img/lukasp.webp",
    "tag": "link",
    "linkText": "",
    "isExternal": false,
    "issue": "broken",
    "status": 404,
    "error": null,
    "finalUrl": "https://crawlee.dev/blog/img/lukasp.webp",
    "redirectCount": 0,
    "redirectHops": null,
    "change": "new"
}
```

- `issue`:
  - `broken`: the final status is 400 or higher.
  - `error`: no HTTP answer (DNS failure, connection refused, TLS error, timeout) or more than 10 redirects. The reason is in `error`.
  - `blocked`: the server answered 401, 403, 429 or 999, which usually means it refuses automated requests. The page often works in a browser, so check it by hand before fixing anything.
  - `longRedirectChain`: the link works but needs more than 3 redirects. `redirectHops` lists every hop with its status.
- `tag`: `a`, `img`, `link` or `script`. `sitemap` when a URL listed in your sitemap is itself broken (then `sourcePage` is the sitemap). `startUrl` when the start URL itself fails (then `sourcePage` is null).
- `linkText`: anchor text of an `<a>` or `alt` of an `<img>`, first 100 characters.
- `change`: `new` or `stillBroken`, compared with the previous run for the same `startUrl`.

The last item is the summary:

```json
{
    "type": "summary",
    "startUrl": "https://crawlee.dev/",
    "mode": "sitemap",
    "sitemaps": ["https://crawlee.dev/sitemap.xml"],
    "pagesScanned": 3,
    "linksChecked": 46,
    "skippedByRobots": 0,
    "externalSkipped": 0,
    "problems": 1,
    "broken": 1,
    "errors": 0,
    "longRedirectChains": 0,
    "new": 1,
    "stillBroken": 0,
    "fixed": 0,
    "firstRun": true,
    "stoppedByCostLimit": false,
    "runErrors": []
}
```

- `pagesScanned`: HTML pages downloaded and parsed (the billed count).
- `linksChecked`: unique URLs requested, including the pages.
- `fixed`: problems from the previous run that are gone. A problem only counts as fixed when its page was scanned again, so a run with a smaller `maxPages` does not look like a cleanup.
- `runErrors`: sitemaps that could not be read and similar notes. The Actor does not fail because of a bad page or link.

### Comparing runs

Problems are stored in a named key-value store called `broken-link-checker-state` in your Apify account, one record per `startUrl`. Schedule the Actor weekly and filter the dataset by `change = new` to see only what broke since last time. Delete the store to start fresh.

### Pricing

Pay per event: `page-scanned`, once for each HTML page of your site that was downloaded and parsed. Checking links is not billed, and pages that fail or are not HTML are not billed (they show up as problems instead). You can cap the cost of a run with the "Maximum cost per run" setting. When the cap is reached, no more pages are scanned, but the links already found are still checked and reported.

### Limitations

- Only links in the HTML are seen. Links added by JavaScript, CSS `url(...)`, `srcset` and iframes are not checked.
- Some sites answer 403, 429 or 999 to automated requests even when the page works (LinkedIn, some shops behind bot protection). These show up as `broken` with that status. Check them by hand.
- Each target is checked once per run with no retry. A timeout from a slow server is reported as `error`.
- Checks run from Apify's servers. Pages only reachable from your network or behind a login are reported as errors.
- The run takes about 1 second per 4 internal links at the default rate limit. A site with many unique links per page takes longer.

### Run locally

```bash
python -m venv .venv
.venv/Scripts/pip install -r requirements.txt   # on Linux/macOS: .venv/bin/pip
## put input in storage/key_value_stores/default/INPUT.json
.venv/Scripts/python -m my_actor
python -m unittest discover -s tests -t .
```

Author: Matias Isnard.

# Actor input Schema

## `startUrl` (type: `string`):

Your homepage (for example https://example.com) or your sitemap (https://example.com/sitemap.xml, sitemap indexes and .xml.gz work). With a homepage, the sitemaps listed in robots.txt or /sitemap.xml are used. If no sitemap is found, same-site pages are crawled from this URL.

## `maxPages` (type: `integer`):

Pages of your site to scan. Each scanned HTML page is one billed event. Link checks are not billed.

## `checkExternal` (type: `boolean`):

Also check links to other websites. External sites are only checked when their robots.txt allows it, at most one request per second per site.

## `maxLinksPerPage` (type: `integer`):

Links taken from each page, in document order. Duplicates on a page count once.

## `concurrency` (type: `integer`):

Requests in flight at the same time. Your own site also gets at most 4 requests per second (or less if robots.txt sets a Crawl-delay).

## `respectRobots` (type: `boolean`):

Skip pages and links of your own site that robots.txt disallows. Turn off only for a site you own. robots.txt of other sites is always respected.

## Actor input object example

```json
{
  "startUrl": "https://books.toscrape.com",
  "maxPages": 200,
  "checkExternal": true,
  "maxLinksPerPage": 500,
  "concurrency": 4,
  "respectRobots": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://books.toscrape.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("isnard/broken-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://books.toscrape.com" }

# Run the Actor and wait for it to finish
run = client.actor("isnard/broken-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://books.toscrape.com"
}' |
apify call isnard/broken-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,isnard/broken-link-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/En4nNCneEED6i0M2K/builds/g3QiDy96fLKrSpsHR/openapi.json