# Dead Link Finder (Site Crawl): 404 and Redirect Report (`transparent_meteorite/dead-link-finder-site-crawl`) Actor

Crawl your website and get one row per broken or redirecting link: 404, 410, 5xx, DNS, timeout, SSL errors, redirect loops and long chains, with source page, anchor text and how many pages link to it. robots.txt honoured.

- **URL**: https://apify.com/transparent_meteorite/dead-link-finder-site-crawl.md
- **Developed by:** [Detroit Web Agent - Open Data](https://apify.com/transparent_meteorite) (community)
- **Categories:** SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 link checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dead Link Finder (Site Crawl): 404 and Redirect Report

Give it your home page. It crawls your website page by page, checks every link it finds, and returns **one row per broken or redirecting link**: the page it is on, the anchor text, the status code, the error type and how many pages link to it. Plain HTTP, robots.txt honoured, no browser, no login, no proxy needed.

### What data you get

- Broken links: 404, 410, other 4xx, 5xx, DNS failures, timeouts, SSL certificate errors, connection errors
- Redirect problems: loops, chains longer than 3 hops, every hop listed; simple redirects optional
- Source page and anchor text, internal or external, pages linking to it, check method, response time, checked-at time

### Use cases

- Monthly SEO hygiene: schedule a crawl and fix the 404s that leak link equity, most-linked first.
- Site migrations: run before and after a redesign or domain move to catch broken internal links and redirect chains.
- Agencies: hand clients a CSV of broken links with the exact page and anchor text to edit.
- Content teams: find dead outbound links in old blog posts before readers do.
- Automation: send new broken links to Slack, a ticket queue or a sheet with n8n, Make or Zapier.

### What this actor does

1. Reads robots.txt for your site and follows its rules (Allow, Disallow, wildcards, Crawl-delay).
2. Opens up to `maxPages` HTML pages of your site and extracts every link (optionally images, scripts and stylesheets too).
3. Checks each unique URL once: a HEAD request first, then GET when a server refuses or mishandles HEAD. Redirects are followed by hand so every hop is recorded and loops are caught.
4. Writes one row per problem link, sorted errors first and then by how many pages link to it.

#### Key features

- **Same-site crawl** with www and non-www treated as one site; subdomains optional.
- **External links checked too** (one status request each, never crawled), with a per-host rate limit so a page with 200 links to one site never floods it.
- **Polite by default**: 3 requests per second to your site, 30 per minute per external host, 8 in parallel, retries with backoff on 429 (Retry-After honoured) and on timeouts.
- **Fails fast**: an invalid start URL ends the run in a second with a clear message and nothing charged.
- **Cost caps you control**: `maxLinksChecked` and Apify's maximum cost per run. Skipped links are never charged.

### Input

| Field | What it does | Default |
|---|---|---|
| `startUrls` | Pages to start from (one site each) | crawler-test.com demo |
| `maxPages` | HTML pages to open and read links from | 25 |
| `maxLinksChecked` | Cap on unique URLs checked (the billed count) | 1000 |
| `checkExternalLinks` | Also check links to other websites | true |
| `includeSubdomains` | Treat subdomains as the same site | false |
| `checkResources` | Also check images, scripts, stylesheets, media | false |
| `excludeUrlPatterns` | Skip URLs containing these texts (`*` wildcard) | none |
| `internalRequestsPerSecond` | Crawl speed on your site | 3 |
| `externalRequestsPerMinutePerHost` | Rate limit per external host | 30 |
| `maxConcurrency` | Requests in flight at once | 8 |
| `requestTimeoutSecs` | Timeout per request (one retry) | 15 |
| `reportRedirects` | `all`, `chains_only` or `none` | all |
| `includeSummaryRow` | Append a totals row to the dataset | false |
| `proxyConfiguration` | Optional proxy; off by default | off |

#### Example input

```json
{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "maxPages": 500,
  "maxLinksChecked": 5000,
  "checkExternalLinks": true,
  "excludeUrlPatterns": ["/wp-admin/", "?replytocom="],
  "reportRedirects": "chains_only"
}
```

### Output

One row per broken or redirecting link. Healthy links are counted but not listed. The crawl summary (pages crawled, links checked, counts by error type, robots.txt notes, why a run stopped early) is saved as the `OUTPUT` record and, if you ask for it, as a final `rowType: summary` row.

| Field | Meaning |
|---|---|
| `linkUrl` | The broken or redirecting URL |
| `sourcePage` | First page where it was found |
| `anchorText` | Link text (or image alt) on that page |
| `statusCode` | Final HTTP status (empty for DNS, timeout, SSL) |
| `errorType` | 404, 410, 4xx, 5xx, forbidden, rate_limited, blocked, dns, timeout, ssl, connection_error, redirect_loop, invalid_redirect, invalid_url, redirect_chain, redirect |
| `severity` | error, warning or notice |
| `finalUrl`, `redirectChain`, `redirectHops` | Where redirects lead, every hop |
| `linkType` | internal or external |
| `foundOnPageCount`, `foundOnPages` | How many crawled pages link to it (up to 20 listed) |
| `checkMethod`, `responseTimeMs`, `checkedAt` | How and when it was checked |

#### Sample output

```json
{
  "rowType": "link",
  "linkUrl": "https://crawler-test.com/redirects/redirect_to_404",
  "sourcePage": "https://crawler-test.com/",
  "anchorText": "Redirect to 404",
  "statusCode": 404,
  "errorType": "404",
  "severity": "error",
  "errorMessage": "HTTP 404",
  "finalUrl": "https://crawler-test.com/status_codes/status_404",
  "redirectChain": ["301 https://crawler-test.com/redirects/redirect_to_404"],
  "redirectHops": 1,
  "linkType": "internal",
  "resourceType": "link",
  "foundOnPageCount": 1,
  "foundOnPages": ["https://crawler-test.com/"],
  "checkMethod": "GET",
  "responseTimeMs": 412,
  "checkedAt": "2026-11-08T06:00:01.000Z"
}
```

### Pricing

Pay per event: **$0.20 per 1,000 links checked** ($0.0002 per unique URL whose status was checked, crawled pages included), plus Apify's $0.00005 start fee per run. You pay for the work done, not for the number of problems found, so a clean site costs the same as a broken one of the same size. Links skipped by robots.txt, your exclusions, the external-links switch or the link limit are never charged. Example: a 300-page site with 1,800 unique links costs about $0.36.

### Use with the API, n8n, Make, Zapier or AI agents

**API** (one call, returns the rows as JSON):

```bash
curl -X POST "https://api.apify.com/v2/acts/transparent_meteorite~dead-link-finder-site-crawl/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://crawler-test.com/"}],"maxPages":1,"maxLinksChecked":450,"checkExternalLinks":true}'
```

The call waits up to 5 minutes. For bigger inputs, start the run with `/runs` instead and read the dataset when it finishes.

**Python**

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("transparent_meteorite/dead-link-finder-site-crawl").call(run_input={"startUrls": [{"url": "https://crawler-test.com/"}], "maxPages": 1, "maxLinksChecked": 450, "checkExternalLinks": True})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)
```

- **n8n, Make, Zapier:** use the Apify app's "Run Actor" step with actor `transparent_meteorite/dead-link-finder-site-crawl`, paste the input JSON, then map the dataset items to your sheet or CRM.
- **Clay:** add an HTTP API column that calls the run-sync endpoint above.
- **Schedules:** in Apify Console, Schedules, run it daily or weekly with the same input for a steady feed.
- **AI agents (MCP):** connect the Apify MCP server at `https://mcp.apify.com?tools=transparent_meteorite/dead-link-finder-site-crawl` to Claude, Cursor or any MCP client, and your agent can run Dead Link Finder (Site Crawl) on request.

### FAQ

**Why HEAD and then GET?** HEAD is fast and light. Some servers answer HEAD with 403, 404 or 405 while the page works fine, so any error on HEAD is confirmed with a GET before it is reported.

**Does it run JavaScript?** No. It reads the HTML your server sends, which is what search engines see first. Links injected only by JavaScript are not found.

**Why does LinkedIn, X or a shop show as forbidden or blocked?** Some sites refuse automated requests (403, 429 or 999). These are reported as warnings, not errors, because the link usually works in a browser.

**Does it respect robots.txt?** Yes, for every page on your own site. External links get one status request each and are never crawled.

**Can I schedule it?** Yes. Use Apify schedules and a webhook or integration to send new rows to Slack, email or a sheet.

### Limits

- Pages are read as HTML only (no JavaScript rendering); single-page apps that build links in the browser will show few links.
- A redirect chain longer than 10 hops is reported as a redirect loop.
- A site whose robots.txt returns a server error is not crawled (only the start URL is checked), as robots.txt rules require.
- Very slow hosts can stop a run early to keep costs sane; the summary says so and names the reason.
- Up to 100 start URLs, 10,000 pages and 100,000 links per run.

### Related actors

- [Sitemap Link Inventory](https://apify.com/transparent_meteorite/sitemap-link-inventory): every URL in a site's sitemaps, ready to compare with what the crawl finds.
- [Technical SEO Page Audit](https://apify.com/transparent_meteorite/technical-seo-page-audit): titles, meta tags, canonicals and headings per page.
- [Wayback Page Change Tracker](https://apify.com/transparent_meteorite/wayback-page-change-tracker): find what a dead page used to say, to choose a redirect target.

### Support

Found a bug or need a field? Open an issue on the Issues tab with the run URL and we will reply there. If the actor saved you time, a short honest review on the [Reviews tab](https://apify.com/transparent_meteorite/dead-link-finder-site-crawl/reviews) helps other people judge it.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start from, for example https://example.com/. The crawl stays on the same website (www and non-www count as the same site). Several start URLs = several sites in one run.

## `maxPages` (type: `integer`):

How many HTML pages of your site to open and read links from, for example 25 for a quick check or 2000 for a full audit. Links found on those pages are all checked, even when they point to pages beyond this limit.

## `maxLinksChecked` (type: `integer`):

Hard cap on unique URLs checked in one run (crawled pages included), for example 1000. This is also your cost cap: you pay per link checked.

## `checkExternalLinks` (type: `boolean`):

Also check links that point to other websites (one status request each, never crawled). Turn off to audit internal links only.

## `includeSubdomains` (type: `boolean`):

When on, blog.example.com and shop.example.com count as part of example.com and get crawled too.

## `checkResources` (type: `boolean`):

Check img, script, stylesheet, icon, iframe and media URLs as well as normal links. Finds broken images; adds more links to check.

## `excludeUrlPatterns` (type: `array`):

URLs that contain any of these texts are skipped and never charged. Use \* as a wildcard, for example /wp-admin/, ?replytocom= or https://twitter.com/\*.

## `internalRequestsPerSecond` (type: `integer`):

Polite crawl speed on your own site, for example 3. A robots.txt Crawl-delay slows it further (up to 10 seconds between requests).

## `externalRequestsPerMinutePerHost` (type: `integer`):

Rate limit for each external website you link to, for example 30 per minute, so a page with 200 links to the same site never floods it.

## `maxConcurrency` (type: `integer`):

How many requests may be in flight at once across all hosts, for example 8. The per-host limits above still apply.

## `requestTimeoutSecs` (type: `integer`):

A link that does not answer within this many seconds (one retry) is reported as a timeout, for example 15.

## `reportRedirects` (type: `string`):

Which redirecting links get a row, for example all to clean up every 301. Redirect loops are always reported as errors.

## `includeSummaryRow` (type: `boolean`):

Append one rowType=summary row with totals by error type to the dataset. The same summary is always saved as the OUTPUT record.

## `proxyConfiguration` (type: `object`):

Not needed for most sites (plain HTTP from Apify). Use a proxy only if your site blocks data-centre IPs, for example Apify Proxy.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawler-test.com/"
    }
  ],
  "maxPages": 1,
  "maxLinksChecked": 100,
  "checkExternalLinks": true,
  "includeSubdomains": false,
  "checkResources": false,
  "excludeUrlPatterns": [],
  "internalRequestsPerSecond": 3,
  "externalRequestsPerMinutePerHost": 30,
  "maxConcurrency": 8,
  "requestTimeoutSecs": 15,
  "reportRedirects": "all",
  "includeSummaryRow": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `links` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawler-test.com/"
        }
    ],
    "maxPages": 1,
    "maxLinksChecked": 100,
    "checkExternalLinks": true,
    "excludeUrlPatterns": [],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("transparent_meteorite/dead-link-finder-site-crawl").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawler-test.com/" }],
    "maxPages": 1,
    "maxLinksChecked": 100,
    "checkExternalLinks": True,
    "excludeUrlPatterns": [],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("transparent_meteorite/dead-link-finder-site-crawl").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawler-test.com/"
    }
  ],
  "maxPages": 1,
  "maxLinksChecked": 100,
  "checkExternalLinks": true,
  "excludeUrlPatterns": [],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call transparent_meteorite/dead-link-finder-site-crawl --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,transparent_meteorite/dead-link-finder-site-crawl"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iRGXTqcGuHlcgxsKL/builds/qXOCEKTUTF3ommsy0/openapi.json