# Broken Link Checker: Crawl a Website, Find 404s & Redirects (`charlesmarkette/broken-link-checker-crawl-website-404`) Actor

Crawl your website and check every link once: find 404s, server errors, redirect chains and loops, with the pages and anchor text where each broken link appears. Optional images, scripts and external links, sitemap seeding and robots.txt. For SEO audits and migrations.

- **URL**: https://apify.com/charlesmarkette/broken-link-checker-crawl-website-404.md
- **Developed by:** [Charles Markette](https://apify.com/charlesmarkette) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 checked links

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Broken Link Checker** crawls your website from one or more start URLs, collects every link on its pages (and, if you want, every image, script and stylesheet), checks each unique link **once**, and tells you which ones are **broken** (404, 410, 500, timeouts, DNS errors, redirect loops) and **exactly where they are**: the pages that link to them and the anchor text used. Redirects are reported with their full chain, so you can also clean up 301/302 hops after a migration.

It's a plain, fast HTML crawler: no browser, no proxies, no setup. Give it your home page and get a table you can export as CSV or Excel, send to Google Sheets, or get by email after a scheduled run.

### What you get

- **Each broken link with where it's linked from.** For each link: up to 10 of the pages it was found on with the **anchor text** (or image alt text), plus the total number of pages linking to it (`foundOnCount`).
- **Internal and external links.** Links to your own site are crawled; links to other websites are checked once and never crawled.
- **Redirect chains and loops.** Each hop's URL and status code, the final URL, and redirect loops flagged as broken.
- **Sitemap entries checked too.** URLs listed in `/sitemap.xml` (and sitemap index files) are crawled as well, so pages nothing links to are covered, and a 404 in your sitemap shows the sitemap as where it was found.
- **Images, scripts and stylesheets** (optional): `<img src>` and `srcset`, `<picture>` sources, `<script src>` and `<link rel="stylesheet">`.
- **Polite by default.** At most 2 parallel requests per host, `robots.txt` respected for crawling, and a clear User-Agent (`BrokenLinkChecker/1.0; +https://apify.com/charlesmarkette`).

### Use cases

- **SEO audit:** find the 404s and long redirect chains that waste crawl budget and leak link equity, with the pages to fix.
- **Site migration:** after moving to a new domain, CMS or URL structure, crawl the new site and list every link still pointing at an old URL (redirected) or a page that no longer exists (404).
- **Pre-launch QA:** crawl a staging or newly launched site and catch broken navigation, missing images and dead external references in the HTML before visitors do (links added by JavaScript aren't seen; see the FAQ).
- **Scheduled monitoring with an email alert:** schedule a weekly run with *Output only broken links* on, and add our [Notify Step](https://apify.com/charlesmarkette/send-email-report-slack-discord-teams-telegram-alerts) as an integration. By default it sends nothing when the dataset is empty, so you get an email (or a Slack, Teams, Discord or Telegram message) only when something is broken or couldn't be verified (add sites that always block bots, such as `linkedin\.com`, to *Ignore URLs matching* to keep those alerts quiet).
- **Content and outbound-link hygiene:** find external links to sites that went offline or now redirect somewhere unexpected.

### How to use it

1. Add your **Start URL** (usually the home page). Several start URLs are fine; the crawl covers all of their sites.
2. Pick the **Crawl scope**: *Same host* (only `www.example.com`) or *Same domain* (also `example.com`, `blog.example.com`, `shop.example.com`).
3. Set **Max pages to crawl** (default 200) and, if you like, turn on **Check images, scripts and stylesheets** or **Output only broken links**.
4. Run it. When the crawl finishes, each unique link becomes one row. Use the **Overview**, **Where each link appears** and **Redirects** views, or export the dataset.

**Input example**

```json
{
  "startUrls": ["https://www.example.com/"],
  "maxPages": 500,
  "crawlScope": "sameDomain",
  "checkResources": true,
  "ignoreUrlPatterns": ["/wp-admin/", "\\?replytocom="],
  "onlyBrokenInOutput": true
}
```

**Output example** (one row)

```json
{
  "url": "https://www.example.com/pricing-2023",
  "statusCode": 404,
  "ok": false,
  "statusText": "Not Found",
  "isBroken": true,
  "isUnverified": false,
  "linkType": "internal",
  "resourceType": "link",
  "finalUrl": "https://www.example.com/pricing-2023",
  "redirectCount": 0,
  "redirectChain": [],
  "isRedirectLoop": false,
  "responseTimeMs": 87,
  "contentType": "text/html; charset=utf-8",
  "error": null,
  "foundOnCount": 14,
  "firstFoundOn": "https://www.example.com/",
  "foundOn": [
    {"page": "https://www.example.com/", "anchorText": "Pricing", "nofollow": false},
    {"page": "https://www.example.com/blog/launch", "anchorText": "see our plans", "nofollow": false}
  ],
  "nofollow": false,
  "depth": 1,
  "pageCrawled": false,
  "robotsDisallowed": false,
  "checkedAt": "2026-10-16T09:41:12Z"
}
```

**All columns**

- `url`: the link, normalised (lower-case scheme and host, default port removed, `#fragment` dropped, `/./` and `/../` resolved, percent-encoding normalised, query string kept).
- `statusCode`, `ok` (2xx), `statusText`: the final response, after redirects.
- `isBroken`: true for status 400 or higher (except the four below), any `error` (timeout, DNS failure, refused connection, TLS error, too many redirects, invalid URL…) and redirect loops.
- `isUnverified`: true for 401, 403, 429 and 999. These usually mean the site refused an automated check (login required, bot protection, rate limit, LinkedIn's 999), not that the page is gone, so they're not counted as broken. Open them in a browser to be sure.
- `linkType`: `internal` (inside the crawl scope) or `external`. `resourceType`: `link`, `image`, `script` or `stylesheet`.
- `finalUrl`, `redirectCount`, `redirectChain` (`[{url, statusCode, location}]`), `isRedirectLoop`.
- `responseTimeMs`, `contentType`, `error`, `checkedAt`.
- `foundOn` (first 10 `{page, anchorText, nofollow}`), `foundOnCount` (all pages), `firstFoundOn`. A start URL's `foundOn` lists only the pages that link to it; sitemap URLs show the sitemap file as the page.
- `nofollow`: at least one link to it had `rel="nofollow"`. `depth`: clicks from a start URL. `pageCrawled`: the page was downloaded and read for links. `robotsDisallowed`: robots.txt doesn't allow crawling it (it was still checked, see FAQ).

A run summary is saved as `OUTPUT` in the key-value store: `pagesCrawled`, `linksChecked`, `broken`, `internalBroken`, `externalBroken`, `unverified`, `redirects`, `byStatus` (count per status code, `error` for no response), `outputRows`, `notChecked`, `ignored`, `externalNotChecked`, `sitemapUrls`, `robotsTxtUnreachable` and `stoppedBy` (`null`, `max cost`, `max results`, `time limit`, or `charging error` if the platform's charging API kept failing; links whose charge couldn't be confirmed are then left out).

### How much does it cost?

Pay per event: **$0.001 per checked link + $0.005 per run start, platform usage included.**

**Apify Store discounts:** the price above is for Apify's Free plan. Paid plans pay less per 1,000 checked links: $0.90 on Starter, $0.80 on Scale, $0.70 on Business. The "from" price at the top of the Store page is the Business price.

A "checked link" is each unique URL that gets an HTTP check, and that includes the pages of your site that are crawled (each page is fetched once, and that fetch is its check). A dead link is a result, so links that return 404 or 500, or fail with a DNS error, timeout or refused connection, are charged. These are **free**: links that aren't valid http(s) URLs (reported as broken with `Invalid URL`), URLs refused by the private-address guard, `mailto:`/`tel:`/`javascript:` links (skipped), repeats of a link already checked in the run, links matching *Ignore URLs matching*, and external links when *Check external links* is off.

With *Output only broken links* on, every link is still checked and charged; the working ones are just left out of the dataset (broken and unverified ones are kept).

**Worked examples:**

*Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default memory); Apify Proxy (where used) is billed separately.*

- A 200-page site with 1,500 unique links (pages included): 1,500 × $0.001 + $0.005 = **$1.505**.
- A weekly check of a 50-page site with 400 unique links: $0.405 per run, about **$1.76 per month**.

If you set a maximum cost per run, the crawl stops cleanly at that limit and you get every row already paid for, plus the summary. A link is only checked when the remaining budget can pay for it.

### FAQ and limits

#### Does it see links added by JavaScript?

No. It reads the HTML your server sends, without a browser, so links that only appear after scripts run (some single-page apps, menus built in JavaScript) aren't seen, and `<meta http-equiv="refresh">` and JavaScript redirects aren't followed. Server-rendered sites (WordPress, Shopify, most CMSs and static sites) put their links in that HTML, so they're covered.

#### Why does a link show 403, 429 or 999 when it works in my browser?

Some sites (LinkedIn, Amazon, sites behind Cloudflare bot protection…) block automated requests or requests from data-centre IP addresses. The Actor doesn't use proxies or solve challenges, so it reports the status it actually got, but marks 401, 403, 429 and 999 as `isUnverified: true` instead of broken: the check couldn't tell whether the page works. This applies to your own site too, if its firewall blocks bots. Use *Ignore URLs matching* (e.g. `linkedin\.com`) to leave such domains out.

#### How are links checked?

External links and resources get a HEAD request first; if the server answers HEAD with any error (4xx/5xx) or drops the connection, the link is checked again with GET, so servers that mishandle HEAD aren't reported as broken. Pages of your site that are crawled are fetched with GET (their HTML is needed). Busy answers (429, 502, 503, 504) are retried twice with backoff, honouring `Retry-After` up to 10 seconds. Each request waits at most *Timeout per request*; a whole link, with its redirect hops and retries, gets at most 4× that (at least 60 seconds).

#### How does robots.txt work here?

With *Respect robots.txt* on (the default), pages that robots.txt disallows are **not crawled**: they're never downloaded and read for more links. A link pointing to such a page is still **checked once** with a single HEAD request (GET only if HEAD fails), the same as a visitor clicking it, and marked `robotsDisallowed: true`. We chose this because robots.txt governs crawling, and a broken link to a disallowed page (a members area, a checkout step) is still a broken link on your site. If a robots.txt can't be read because of a server error or timeout, crawling that host is treated as disallowed (RFC 9309), and you'll see it in `robotsTxtUnreachable`. Rules are matched for the `BrokenLinkChecker` user-agent, then `*`, with `*` and `$` wildcards and longest-match precedence. `Crawl-delay` is not used; politeness comes from the per-host limit.

#### What about rel="nofollow"?

It's recorded (`nofollow` on the row and on each `foundOn` entry) and the link is still checked, and crawled if it's internal. A broken nofollow link is still broken for visitors.

#### Why do rows appear only at the end?

A link's `foundOn` list keeps growing while the crawl finds more pages that link to it, so rows are written once the crawl has finished. While it runs, the crawl state lives on disk, not in memory, and it's saved to the key-value store regularly: if the platform restarts the run (for example a server migration), it continues from the last save, without charging any link twice and without writing rows twice (links checked after the last save may be checked again, free of charge). Near the run's timeout the crawl stops early (`stoppedBy: "time limit"`) to leave time to write the rows.

#### Other limits

- Up to 10,000 crawled pages per run (*Max pages to crawl*); all links on them are checked. At most 5,000 links are read from one page and at most 250,000 unique links are kept per run. The default 256 MB of memory suits typical sites; give the run 1 GB for crawls with around 100,000 unique links.
- At most 2 MB of HTML is read per page; links further down a larger page aren't seen. Pages are requested with gzip/deflate only.
- The sitemap: `/sitemap.xml` plus `Sitemap:` lines in robots.txt, following sitemap index files, `.xml.gz` and plain-text sitemaps, at most 25 sitemap files (20 MB each), and at most *Max pages* URLs taken from them. Sitemap URLs are crawled like start URLs (depth 0).
- URLs with different query strings are different links (`?utm_source=…` variants are checked separately); use *Ignore URLs matching* to leave tracking variants out. `#fragments` are dropped, so `page#a` and `page#b` are one link.
- `<base href>` is honoured. `<iframe>`, `<video>`, CSS `url()` and `<link rel="canonical">`/`hreflang` targets aren't collected.
- No login: cookies are kept only within one link's redirect chain, never between pages, so pages behind a login show up as whatever the server returns to a visitor who isn't logged in.
- Private, loopback, link-local and cloud-metadata addresses are never requested (free rows with `Blocked: private address`), including when a public URL redirects to one. That also means intranet sites can't be crawled.
- HTTP/1.1 only. Response times are measured from Apify's servers, not your location.
- Certificates are verified like Python does, not like a browser: a site whose server leaves out an intermediate certificate (which some browsers fill in themselves) is reported with a `TLS error`.

### Related Actors

| Actor | Use it to |
|---|---|
| [Bulk URL Status Checker](https://apify.com/charlesmarkette/bulk-url-status-checker-redirects-ssl) | Check a list of URLs you already have (no crawl), with SSL expiry and page metadata. |
| [PageSpeed & Lighthouse Audit](https://apify.com/charlesmarkette/pagespeed-lighthouse-audit-core-web-vitals) | Audit the site's key pages for Core Web Vitals, SEO and accessibility, no API key. |
| [Sitemap URL Extractor & New Page Monitor](https://apify.com/charlesmarkette/sitemap-url-extractor-new-page-monitor) | List every URL in the site's sitemaps, to compare with what the crawl found. |
| [Notify Step](https://apify.com/charlesmarkette/send-email-report-slack-discord-teams-telegram-alerts) | Email or Slack the broken links after a scheduled run, only when there are some. |
| [Google Sheets Import & Export](https://apify.com/charlesmarkette/google-sheets-sync) | Send the broken links to a Google Sheet your team works from. |
| [Dataset Filter & Transform](https://apify.com/charlesmarkette/dataset-filter-transform-rename-flatten-fields) | Keep only internal 404s, pick and rename columns, or flatten `foundOn` for a spreadsheet. |

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl starts, usually your home page. One per line. The crawl stays on these sites (see Crawl scope); bare domains (example.com) become https://example.com/.

## `maxPages` (type: `integer`):

How many HTML pages of your site to download and read for links (1–10,000). Links found on them are all checked, so the number of checked links is usually much larger.

## `maxDepth` (type: `integer`):

How many clicks away from a start URL a page may be and still be crawled. 0 = only the start URLs (and sitemap URLs) are read; their links are still checked.

## `crawlScope` (type: `string`):

Which pages count as your site (internal) and get crawled. Same host: exactly the start URL's host name (www.example.com). Same domain: also every subdomain (example.com, blog.example.com, shop.example.com). Everything else is an external link: checked once, never crawled.

## `includeSitemap` (type: `boolean`):

Also start from the URLs in /sitemap.xml and the sitemaps listed in robots.txt (sitemap index files are followed; at most 25 sitemap files, and at most 'Max pages' URLs are taken from them). Finds pages that nothing links to, and reports broken sitemap entries (foundOn shows the sitemap).

## `respectRobotsTxt` (type: `boolean`):

Don't crawl (download and read) pages that robots.txt disallows. Links to them are still checked once, like a visitor clicking them, and marked robotsDisallowed.

## `checkExternalLinks` (type: `boolean`):

Also check links that point to other websites (one request each; they're never crawled). Off: external links are skipped and not output.

## `checkResources` (type: `boolean`):

Also check <img src> and srcset, <picture> sources, <script src> and <link rel=stylesheet> files, not only <a href> links.

## `ignoreUrlPatterns` (type: `array`):

Regular expressions (case-insensitive, up to 500 characters each). Links matching any of them are neither checked nor crawled, e.g. /wp-admin/, ?replytocom=, ^https://(www.)?facebook.com/.

## `onlyBrokenInOutput` (type: `boolean`):

On: only broken links (status 400 or higher, an error such as a timeout, DNS failure or refused connection, or a redirect loop) and unverified ones (401, 403, 429 or 999: the site blocked the check) are output. Every link is still checked and charged; the others are just left out of the dataset.

## `maxResults` (type: `integer`):

Optional. Stop once this many rows are ready for output (each row is one unique link).

## `timeoutSecs` (type: `integer`):

How long to wait for each request (each redirect hop) before reporting 'Timed out'. A whole link, with all its redirect hops and retries, gets at most 4 times this (at least 60 seconds).

## `concurrency` (type: `integer`):

How many links to check at the same time in total (1–50).

## `perHostConcurrency` (type: `integer`):

At most this many requests at a time to any one host, including your own site, to stay polite (1–10).

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com"
  ],
  "maxPages": 20,
  "maxDepth": 10,
  "crawlScope": "sameHost",
  "includeSitemap": true,
  "respectRobotsTxt": true,
  "checkExternalLinks": true,
  "checkResources": false,
  "onlyBrokenInOutput": false,
  "timeoutSecs": 15,
  "concurrency": 10,
  "perHostConcurrency": 2
}
```

# Actor output Schema

## `items` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://example.com"
    ],
    "maxPages": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("charlesmarkette/broken-link-checker-crawl-website-404").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://example.com"],
    "maxPages": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("charlesmarkette/broken-link-checker-crawl-website-404").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://example.com"
  ],
  "maxPages": 20
}' |
apify call charlesmarkette/broken-link-checker-crawl-website-404 --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,charlesmarkette/broken-link-checker-crawl-website-404"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wRaz2EpHi30tdxDjc/builds/4hQJU0IHsWfYIh8Ur/openapi.json