# SEO Audit & Broken Link Checker – Site Crawler (`far003/broken-link-seo-site-audit`) Actor

Crawl a site over HTTP: find broken links, redirect chains and on-page SEO issues. 0-100 score per page and what changed since the last run.

- **URL**: https://apify.com/far003/broken-link-seo-site-audit.md
- **Developed by:** [Francesco Antonio Russo](https://apify.com/far003) (community)
- **Categories:** SEO tools, Developer tools, Marketing
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$20.00 / 1,000 page audits

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SEO Audit & Broken Link Checker – Site Crawler

Crawls a website over plain HTTP and returns **one record per page**: every link and resource on it verified, the redirect chains, the on-page SEO problems, a **0-100 score** with a letter grade — and, if you schedule it, **what changed since the last run**.

| Area | What is checked |
|---|---|
| **Links** | every `<a>` link, internal and external, requested once per run: broken (404/410, 5xx, DNS, refused, TLS, redirect loop), redirecting (with the final URL and number of hops), or *unverified* (see below) |
| **Resources** | every image, script, stylesheet and iframe URL; `http://` resources on `https://` pages (mixed content) |
| **Redirects** | the full chain of each page, hop by hop, with status codes; loops and chains of 2+ hops |
| **On-page** | title (missing, multiple, too long/short), meta description, H1, canonical, `lang`, viewport, images without `alt`, word count |
| **Indexing** | `noindex` (meta and `X-Robots-Tag`), canonical pointing elsewhere, pages missing from the sitemap, sitemap pages nobody links to (orphans), `noindex` pages listed in the sitemap |
| **Site level** | duplicate titles and meta descriptions, robots.txt status, sitemap files read, security headers of the home page |

**Three verdicts, not two.** A link is *broken* only when that is certain — and a 5xx answer is asked a second time before it counts. A target that answers 401, 403, 429, 503 or times out is reported as *unverified*: the server exists but will not show the page to a bot (LinkedIn, Stack Overflow and most bot-protected sites do this). Unverified links are listed separately and never lower the score as broken links.

**Polite by design.** It reads and obeys `robots.txt` (RFC 9309, including `Crawl-delay`), identifies itself as `BrokenLinkAuditBot`, sends at most 2 requests per second to the audited site (configurable 1-5) and never more than 1 per second to any other host. No browser, no cookies, no login.

**Typical uses:** find and fix broken links before they cost rankings · pre-launch and post-migration checks (did every old URL survive the redesign?) · a weekly scheduled audit that reports only new problems · agency health checks across client sites · checking a list of URLs from a spreadsheet.

### Quick start

1. Put your site in **Sites or URLs** (`example.com` is enough).
2. Set **Max pages per site** — in crawl mode it is also your cost cap.
3. Run. Open the **Overview** table, or the `BROKEN_LINKS` record for a fix list grouped by broken target.

```json
{
  "startUrls": ["https://example.com/"],
  "maxPagesPerSite": 200
}
```

### Input

| Field | Default | Meaning |
|---|---|---|
| `startUrls` | — | Crawl mode: one entry per site (a path such as `example.com/blog/` starts there). List mode: the exact URLs to audit |
| `mode` | `crawl` | `crawl` follows internal links and the sitemap · `list` audits only the URLs given, one record each, whatever their status |
| `maxPagesPerSite` | 200 | Crawl mode: stop after this many pages per site. List mode audits every URL you give |
| `checkExternalLinks` | `true` | Verify links that leave the site |
| `checkResources` | `true` | Verify images, scripts, stylesheets, iframes |
| `useSitemap` | `true` | Read the sitemaps declared in robots.txt (or `/sitemap.xml`): index files, `.gz` and text sitemaps are supported |
| `trackChanges` | `false` | Compare each page with the previous run (see *Monitoring*) |
| `stateKey` | `default` | Name of the monitor: runs with the same name share one memory |
| `maxDepth` | 20 | Clicks from the start URL; `0` audits only the start URL and verifies its links (sitemap pages are not audited either) |
| `maxLinksPerPage` | 500 | Links beyond this number on one page are counted as skipped |
| `stayInStartFolder` | `true` | A start URL inside a folder (`example.com/blog/`) audits only pages under that folder; links elsewhere are checked but not audited or charged. A start URL at the root audits the whole site |
| `includeSubdomains` | `false` | Treat `blog.example.com` as part of `example.com` (`www` always is) |
| `excludeUrlPatterns` | — | URLs containing these texts are neither crawled nor checked; `*` is a wildcard (`*/tag/*`, `?replytocom=`) |
| `maxRequestsPerSecond` | 2 | 1-5, to the audited site only |
| `timeoutSecs` | 15 | Per request; link checks use at most 10 s |

### Output

#### Dataset — one record per page

57 typed fields, all documented in the dataset schema (*Storage → Dataset → Schema*). A real record (2026-10-06, the first web page ever published, which has one dead link):

```json
{
  "url": "http://info.cern.ch/hypertext/WWW/TheProject.html",
  "finalUrl": "http://info.cern.ch/hypertext/WWW/TheProject.html",
  "site": "info.cern.ch",
  "discoveredVia": "start",
  "depth": 0,
  "statusCode": 200,
  "redirectHops": 0,
  "responseTimeMs": 133,
  "score": 72,
  "grade": "C",
  "issueCodes": [
    "BROKEN_INTERNAL_LINK",
    "META_DESCRIPTION_MISSING",
    "VIEWPORT_MISSING",
    "LANG_MISSING",
    "CANONICAL_MISSING"
  ],
  "title": "The World Wide Web project",
  "titleLength": 26,
  "h1": "World Wide Web",
  "h1Count": 1,
  "wordCount": 169,
  "canonical": null,
  "indexable": true,
  "linksTotal": 23,
  "internalLinks": 23,
  "externalLinks": 0,
  "resourceLinks": 0,
  "linksChecked": 23,
  "brokenLinksCount": 1,
  "brokenLinks": [
    {
      "url": "http://info.cern.ch/hypertext/WWW/Administration/Mailing/Overview.html",
      "type": "link",
      "internal": true,
      "status": 404,
      "reason": "HTTP 404",
      "anchorText": "Mailing lists",
      "occurrences": 1,
      "finalUrl": null
    }
  ],
  "redirectedLinksCount": 0,
  "unverifiedLinksCount": 0,
  "skippedLinksCount": 0,
  "changeStatus": null,
  "coverageComplete": true,
  "issues": [
    {
      "code": "BROKEN_INTERNAL_LINK",
      "severity": "high",
      "area": "links",
      "message": "1 link to this site is broken.",
      "count": 1
    }
  ]
}
```

(shortened: the full record also carries `redirectChain`, `redirectedLinks`, `unverifiedLinks`, `mixedContent`, `metaDescription`, `metaRobots`, `xRobotsTag`, `inSitemap` and the change fields.)

#### Key-value store

- **`BROKEN_LINKS`** — the fix list: one entry per broken target with its status, the pages that link to it (`foundOn`), the anchor texts and how many pages carry it.
- **`SUMMARY`** — per site: pages audited, average score, grades, count of every issue code, duplicate titles and meta descriptions, robots.txt and sitemap report (files read, URLs listed, URLs not reached), security headers of the home page, and the change report.

#### Score and grade

`score = 100 − Σ penalties`, one penalty per issue code present on the page: **critical 25 · high 12 · medium 6 · low 2 · info 0**, floored at 0.
Grade: **A** ≥ 90 · **B** ≥ 75 · **C** ≥ 60 · **D** ≥ 40 · **F** below 40 **or any critical issue**.
Only analysed HTML pages get a computed score. A start or input URL that is certainly dead gets 0 / F; one that was not analysed (unreachable, refused the bot, robots.txt, not HTML) gets `score: null` and is left out of the site average.

#### Issue codes

| Code | Severity | Meaning |
|---|---|---|
| `HTTP_4XX` (404/410) · `HTTP_5XX` · `REDIRECT_LOOP` | critical | the URL itself is certainly dead: score 0 |
| `FETCH_FAILED` | critical | the URL could not be reached at all (DNS, timeout, refused): no score, not charged |
| `NOT_VERIFIABLE` | info | the URL answered 401/403/429/503 or another 4xx to the bot: it may work in a browser. No score, not charged |
| `BROKEN_INTERNAL_LINK` · `BROKEN_RESOURCE` | high | links to your own site, or images/scripts/stylesheets/iframes, that do not load |
| `BROKEN_EXTERNAL_LINK` | medium | links to other sites that are certainly dead |
| `MIXED_CONTENT` | high | `http://` resources on an `https://` page |
| `REDIRECT_CHAIN` / `REDIRECTED` | medium / info | the URL needs 2+ redirects / one redirect to reach its page |
| `INTERNAL_LINK_REDIRECTS` | low | internal links that point to a redirecting URL: link to the final address |
| `LINKS_UNVERIFIED` | info | targets that block bots, ask for a login or timed out |
| `TITLE_MISSING` | high | no `<title>` |
| `META_DESCRIPTION_MISSING` · `H1_MISSING` · `VIEWPORT_MISSING` | medium | |
| `TITLE_TOO_LONG` (> 60) · `TITLE_TOO_SHORT` (< 10) · `TITLE_MULTIPLE` · `META_DESCRIPTION_TOO_LONG` (> 160) · `H1_MULTIPLE` · `IMG_ALT_MISSING` · `LANG_MISSING` · `CANONICAL_MISSING` | low | |
| `NOINDEX_IN_SITEMAP` | medium | the page says `noindex` but the sitemap lists it |
| `NOT_LINKED_INTERNALLY` | low | in the sitemap, but no crawled page links to it |
| `NOINDEX` · `CANONICAL_POINTS_ELSEWHERE` · `NOT_IN_SITEMAP` | info | facts worth knowing, no penalty |
| `NOT_HTML` · `BLOCKED_BY_ROBOTS` | info | a start/input URL that is a file (PDF, image…) or is forbidden by robots.txt: no score |
| `SLOW_RESPONSE` (> 3 s) · `HTML_TOO_LARGE` (> 3 MB) | low | |

Codes are stable: filter on `issueCodes` in your integration.

### Monitoring: what changed since the last run

Turn on `trackChanges` and schedule the Actor (daily, weekly). The first run is the baseline; from the second run every record carries:

- `changeStatus`: `new` · `changed` · `unchanged`
- `newIssueCodes` / `resolvedIssueCodes`
- `newBrokenLinks` / `fixedBrokenLinks`

and `SUMMARY.sites[].changes` lists the new and fixed broken links of the whole site. Send only the news to Slack or email: filter the dataset on `changeStatus` = `changed` or `new`. The memory lives in the named key-value store `broken-link-seo-site-audit-state` in your account; use a different `stateKey` for independent monitors of the same site. Every page is audited (and charged) on every run — the comparison needs the fresh data.

### Pricing

Pay per event, **one event: `page-audit`** — one page analysed with all its links verified. No start fee, no per-link fee: a page with 300 links costs the same as a page with 3.

Not charged (`coverageComplete: false`): URLs that could not be reached, URLs that refused the bot (401/403/429/503…), URLs forbidden by robots.txt, pages left unfinished because the run was stopped. Invalid input entries produce no record at all.

What counts as one event besides an analysed page: a **start or input URL whose status is established** — a 404/410, a 5xx confirmed twice, a redirect loop, or a file that is not HTML. In crawl mode that can only be one of your start URLs; in list mode it is every URL you list, so list mode doubles as a status checker at the page price. Pages found by the crawl are charged only when they are HTML pages that were fully analysed.

Example: a 150-page site audited every Monday is 150 events per run, about 650 a month — multiply by the price shown in the **Pricing** tab. `maxPagesPerSite` (crawl mode) and the run's maximum cost both cap the bill: the Actor stops when either is reached. In list mode the cap is the length of your list, and the run's maximum cost.

### How long does it take

Speed is bounded by politeness, not by compute: each unique URL of the audited site is one request, at 2 per second by default. Roughly: 100 pages with 400 unique internal URLs and 150 external links ≈ 4-5 minutes. A `Crawl-delay` in robots.txt overrides the setting (capped at 10 s per request): a site that asks for `Crawl-delay: 10` takes about 5 minutes per 30 URLs, and the log says so. When the run timeout approaches, the Actor stops starting new pages and still writes `SUMMARY`.

### Limits and honesty

- **No JavaScript rendering.** Links and content injected by client-side scripts are not seen. Server-rendered sites, static sites and most CMS sites (WordPress, Shopify, Webflow, Hugo, Docusaurus …) are fully covered; a single-page app that renders everything in the browser is not a good fit.
- Links inside CSS files, `srcset`, `<video>`/`<audio>` sources and inline styles are not checked. Fragments (`#section`) are not verified against the target page.
- `robots.txt` is obeyed for the audited site: disallowed URLs are neither crawled nor checked (they count as skipped). If robots.txt answers 5xx the site is treated as fully disallowed, as RFC 9309 requires. External link targets are checked with a single HEAD/GET, as any link checker does.
- "Broken" for an external link means the answer at the time of the run. A server that is down for a minute looks the same as a dead one: with `trackChanges` you see whether it stays broken.
- Soft 404s (a "not found" page served with status 200) cannot be told from real pages.
- Two addresses that serve the same content (`/` and `/index.html`, or the same page with different query strings) are two pages: two records, two events. The second one shows `CANONICAL_POINTS_ELSEWHERE` when the site declares a canonical. Use `excludeUrlPatterns` for tracking or filter parameters (`?utm_`, `?sort=`), and for calendars or faceted navigation that generate endless URLs.
- `rel="nofollow"` links are followed and checked like any other link.
- A page served without a `text/html` (or XHTML) Content-Type is treated as a file, not analysed.
- An incomplete TLS chain on a link target is reported as *unverified*, not broken: most browsers repair it on their own. An expired, self-signed or wrong-name certificate is broken.
- If the site starts refusing the bot (many 403/429 answers on its own pages), `SUMMARY` says so in `warnings` and `internalPagesNotReadable`, and the site status becomes `partial` when that is a fifth of the site or more — a quiet "all clean" on a blocked crawl would be a lie.
- Size: built for sites up to a few tens of thousands of pages per run (about 35 KB of memory per page). For larger sites split the audit by section (start URLs `example.com/blog/` and `example.com/shop/` in separate runs: with `stayInStartFolder` on, each run audits and charges only its own folder) or raise the run memory.
- With `trackChanges`, a page first reached in a later run (because an earlier run stopped at `maxPagesPerSite`) is reported as `new`; the memory holds the 20,000 most recently audited pages per site.
- Duplicate titles and descriptions are compared among indexable pages, exact match ignoring case.
- Title and description limits are character counts (60 / 160), an approximation of what search results show.
- Private and loopback addresses (`localhost`, `10.x`, `192.168.x` …) are never requested.
- This is a technical screening, not a ranking guarantee.

### API & integrations

```bash
curl -X POST "https://api.apify.com/v2/acts/far003~broken-link-seo-site-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://example.com/"],"maxPagesPerSite":50}'
```

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('far003/broken-link-seo-site-audit').call({ startUrls: ['https://example.com/'], maxPagesPerSite: 50, trackChanges: true });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const fixList = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('BROKEN_LINKS');
```

Works with Apify schedules, webhooks, Make, Zapier, n8n and the Apify MCP server: every field is typed and described, so an AI agent can read the result without guessing.

# Actor input Schema

## `startUrls` (type: `array`):

Crawl mode: one entry per site to audit (example.com or https://example.com/blog/). List mode: the exact URLs to audit. A bare domain gets https://.

## `mode` (type: `string`):

Crawl follows internal links (and the sitemap) from each start URL. List audits exactly the URLs you give, one record each, without following links.

## `maxPagesPerSite` (type: `integer`):

Crawl mode: the crawl stops after this many pages per site, so this is also your cost cap (one page audit = one charged event). List mode audits every URL you give.

## `checkExternalLinks` (type: `boolean`):

Verify every link that leaves the site (one HEAD/GET per unique URL, never more than 1 request per second per host).

## `checkResources` (type: `boolean`):

Verify that every resource URL referenced by the page loads.

## `useSitemap` (type: `boolean`):

Crawl mode: reads the sitemaps declared in robots.txt (or /sitemap.xml), audits pages no link reaches (orphans) and flags sitemap contradictions.

## `trackChanges` (type: `boolean`):

Remembers each page's issues and broken links in a named key-value store and adds changeStatus, newIssueCodes, resolvedIssueCodes, newBrokenLinks and fixedBrokenLinks to every record. The first run is the baseline.

## `stateKey` (type: `string`):

Runs with the same name share one memory. Use different names for independent schedules on the same site.

## `maxDepth` (type: `integer`):

Clicks from the start URL. 0 audits only the start URL: its links are still verified, sitemap pages are not audited.

## `maxLinksPerPage` (type: `integer`):

Links beyond this number on a single page are counted as skipped.

## `stayInStartFolder` (type: `boolean`):

On: an audit started at example.com/blog/ audits only pages under /blog/ (links elsewhere are still checked, not audited or charged). Off: the whole site is crawled from that page. A start URL at the root always means the whole site.

## `includeSubdomains` (type: `boolean`):

Off: blog.example.com is an external site when auditing example.com (www is always the same site).

## `excludeUrlPatterns` (type: `array`):

URLs containing one of these texts are neither crawled nor checked. \* is a wildcard, e.g. */tag/* or ?replytocom=

## `maxRequestsPerSecond` (type: `integer`):

1-5. Speed of the audit: a site with 1,000 unique internal URLs takes about 8 minutes at 2 per second. Raise it only on your own site. A Crawl-delay in robots.txt always wins; other sites are never asked more than once per second.

## `timeoutSecs` (type: `integer`):

3-60. Link checks use at most 10 seconds.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "mode": "crawl",
  "maxPagesPerSite": 5,
  "checkExternalLinks": true,
  "checkResources": true,
  "useSitemap": true,
  "trackChanges": false,
  "stateKey": "default",
  "maxDepth": 20,
  "maxLinksPerPage": 500,
  "stayInStartFolder": true,
  "includeSubdomains": false,
  "maxRequestsPerSecond": 2,
  "timeoutSecs": 15
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `brokenLinks` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev/"
    ],
    "maxPagesPerSite": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("far003/broken-link-seo-site-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev/"],
    "maxPagesPerSite": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("far003/broken-link-seo-site-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxPagesPerSite": 5
}' |
apify call far003/broken-link-seo-site-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,far003/broken-link-seo-site-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/yImUwM8hhoJcYXgx1/builds/vRYNkzyHfmi6CAYpF/openapi.json