# SEO Audit & Broken Link Checker: Site Crawler (`vantum/seo-audit-broken-links`) Actor

Crawl a website and get a per-page SEO audit plus every broken link, redirect and blocked URL: titles, meta, H1, alt, canonicals, duplicates, sitemap & robots checks.

- **URL**: https://apify.com/vantum/seo-audit-broken-links.md
- **Developed by:** [vantum](https://apify.com/vantum) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SEO Audit & Broken Link Checker

Crawl any website and get, in **one run**:

- **a technical SEO audit of every page**: title, meta description, H1, image alt text, canonical, robots/noindex, hreflang, lang, viewport, thin and duplicate content, mixed content, slow and heavy pages, each with a 0–100 score;
- **every broken link, broken image and broken script/stylesheet**, with the pages that link to it and the anchor text;
- **every internal redirect and redirect chain**, so you can update links to point at the final URL;
- **a site summary**: issues ranked by how many pages they affect, duplicate title/description/content groups, sitemap quality (broken, redirecting, noindex and orphan URLs), robots.txt status and HTTP status codes.

Plain HTTP and no browser, so it is fast and cheap. **$2 per 1,000 audited pages.** Link checks, link rows, summaries, dry runs and failed sites are free.

### Why this one

| | This Actor | Typical SEO audit actors | Typical broken-link checkers |
|---|---|---|---|
| Price | **$0.002 per page**, no start fee | $0.005–$0.04 per page | $0.001 per link checked, often plus a start fee |
| Per-page SEO audit (44 checks) | ✅ | ✅ | ❌ |
| Broken links with source page and anchor text | ✅ free | partial | ✅ (paid per link) |
| External links, images, scripts and CSS checked | ✅ free | rarely | some |
| HEAD first, then GET fallback (no false "405 broken") | ✅ | ❌ | some |
| 403/429/999 reported as **blocked**, not broken | ✅ | ❌ | ❌ |
| Canonical target validated (broken, redirect, noindex) | ✅ | ❌ | ❌ |
| Sitemap quality: broken, redirecting, noindex and orphan URLs | ✅ | rarely | ❌ |
| Duplicate titles, descriptions and content, grouped | ✅ | partial | ❌ |
| Respects robots.txt | ✅ (option) | varies | varies |
| Free dry run with a cost estimate | ✅ | ❌ | ❌ |
| Stops safely before the run timeout and still reports | ✅ | ❌ | ❌ |

### Quick start

1. Enter one or more websites, e.g. `https://crawlee.dev`.
2. Set **Max pages per site** (default 100). Not sure how big the site is? Tick **Dry run** first: it is free and tells you how many pages the sitemap lists and what a full audit would cost.
3. Run. Open the **Pages overview**, **Broken links & redirects** and **Site summary** tabs in the output.

```json
{
  "startUrls": [{ "url": "https://crawlee.dev" }],
  "maxPagesPerSite": 200,
  "checkExternalLinks": true,
  "linkRows": "brokenAndInternalRedirects"
}
```

### How it crawls

- It starts at your URL, reads `robots.txt` and the sitemaps (including sitemap indexes and `.xml.gz` files), then follows links breadth-first: closest pages first.
- It stays on the same site. `www.` and the bare domain count as one site. Subdomains are optional.
- It audits HTML pages that answer 200. Redirecting, broken and non-HTML URLs are recorded as link results and are **not charged**.
- Tracking parameters (`utm_*`, `gclid`, `fbclid`…) are always stripped. **Ignore query strings** also merges `?sort=` style variants.
- **Only crawl / never crawl patterns** take plain text or `/regex/`.
- After crawling, it checks every unique URL found on the audited pages once. Canonical targets go first, then internal links, internal images/scripts/CSS, external links and external resources. It tries `HEAD` and falls back to `GET` when a server mishandles `HEAD`. It retries `429`/`503`, honouring `Retry-After`.
- If the run is about to time out, it stops crawling early and still writes every row and the summary.

**Not covered:** pages that render their content only with JavaScript. These are flagged `JS_RENDERED` so you know. Also not covered: Core Web Vitals, and pages behind a login.

### Output

All rows go to one dataset. `rowType` tells them apart, and the tabs filter the columns.

#### `page` row (charged, one per audited page)

```json
{
  "rowType": "page",
  "url": "https://crawlee.dev/blog/page/2",
  "statusCode": 200,
  "score": 86,
  "errors": 0, "warnings": 2, "notices": 2,
  "issueCodes": ["TITLE_TOO_LONG", "TITLE_DUPLICATE", "META_DESCRIPTION_DUPLICATE", "CANONICAL_MISSING"],
  "issues": [{ "code": "TITLE_TOO_LONG", "severity": "warning", "message": "Title is 108 characters (over 60); search engines will cut it.", "count": 108 }],
  "title": "Crawlee Blog - learn how to build better scrapers | Crawlee …",
  "titleLength": 108,
  "metaDescriptionLength": 52,
  "h1": "Crawlee Blog",
  "h1Count": 1,
  "canonical": null,
  "canonicalStatus": "missing",
  "indexable": true,
  "inlinks": 3,
  "inSitemap": true,
  "brokenLinkCount": 0,
  "brokenLinks": null,
  "linksToRedirects": 0,
  "imagesMissingAlt": 0,
  "wordCount": 1210,
  "responseTimeMs": 182
}
```

Page rows also include: `finalUrl`, `redirectChain`, `depth`, `foundOn`, `notIndexableReason`, `metaDescription`, `h2Count`, `metaRobots`, `xRobotsTag`, `lang`, `hreflang`, `images`, `imagesMissingAltSample`, `internalLinks`, `externalLinks`, `structuredDataTypes`, `ogTitle`, `ogImage`, `htmlBytes`, `contentType` and `crawledAt`.

#### `link` row (free, one per problem URL)

```json
{
  "rowType": "link",
  "url": "https://crawlee.dev/blog/img/lukasp.webp",
  "state": "broken",
  "statusCode": 404,
  "internal": true,
  "kind": "img",
  "linkedFromCount": 1,
  "linkedFrom": [{ "page": "https://crawlee.dev/blog/…", "anchor": null, "kind": "img" }],
  "checkMethod": "GET"
}
```

`state` takes one of these values:

- `broken`: 4xx/5xx, a redirect loop or too many redirects.
- `error`: DNS, connection, SSL or timeout failure.
- `blocked`: 401/403/429/999, meaning the server refused the checker. LinkedIn, Stack Overflow and Cloudflare-protected sites often do this, so check these by hand; they are not counted as broken.
- `redirect`: the URL works but redirects; the row shows the full `redirectChain`.

**Link rows** decides which states get rows.

#### `summary` row (free, one per site)

The summary row holds:

- `pagesAudited`, `averageScore`, `pagesWithErrors`, `nonIndexablePages`;
- `brokenInternalUrls`, `brokenExternalUrls`, `blockedUrls`, `redirectingInternalUrls`, `linksChecked`, `urlsDiscovered`;
- `issues`: every issue code with the number of pages it affects, worst first;
- `duplicateTitles`, `duplicateMetaDescriptions`, `duplicateContent`: groups of pages;
- `sitemaps`: URLs listed, plus broken, redirecting, noindex and orphan URLs (in the sitemap but linked from no audited page, checked only when the whole site was crawled);
- `robotsTxt`, `statusCodes`, `stoppedBy` (`maxPages`, `costLimit`, `timeout` or null).

The same summaries are saved as JSON in the key-value store record `RUN_SUMMARY`.

### Issue codes

Score = 100 − 10 per error − 4 per warning − 1 per notice (minimum 0).

| Code | Severity | Meaning |
|---|---|---|
| `TITLE_MISSING` | error | The page has no \<title>. |
| `TITLE_EMPTY` | error | The \<title> is empty. |
| `TITLE_MULTIPLE` | warning | The page has N \<title> tags. |
| `TITLE_TOO_LONG` | warning | Title is N characters (over 60/160); search engines will cut it. |
| `TITLE_TOO_SHORT` | notice | Title is only N characters (under 30/70). |
| `TITLE_DUPLICATE` | warning | Same title as N other page(s). |
| `META_DESCRIPTION_MISSING` | warning | No meta description. |
| `META_DESCRIPTION_EMPTY` | warning | The meta description is empty. |
| `META_DESCRIPTION_MULTIPLE` | warning | The page has N meta descriptions. |
| `META_DESCRIPTION_TOO_LONG` | notice | Meta description is N characters (over 60/160). |
| `META_DESCRIPTION_TOO_SHORT` | notice | Meta description is only N characters (under 30/70). |
| `META_DESCRIPTION_DUPLICATE` | notice | Same meta description as N other page(s). |
| `H1_MISSING` | warning | No \<h1> heading. |
| `H1_EMPTY` | warning | The \<h1> is empty. |
| `H1_MULTIPLE` | notice | N \<h1> headings. |
| `IMG_ALT_MISSING` | warning | N image(s) without an alt attribute. |
| `CANONICAL_MISSING` | notice | No canonical link. |
| `CANONICAL_MULTIPLE` | error | N canonical links with different targets; search engines may ignore all of them. |
| `CANONICAL_RELATIVE` | notice | Canonical uses a relative URL (…). |
| `CANONICAL_INVALID` | error | Canonical is not a valid http(s) URL (…). |
| `CANONICAL_OTHER_DOMAIN` | warning | Canonical points to another domain (…). |
| `CANONICAL_TO_OTHER_PAGE` | notice | Canonical points to a different page (…); this page will not be indexed itself. |
| `CANONICAL_INSECURE` | warning | Canonical uses http:// on an https:// page. |
| `CANONICAL_TARGET_BROKEN` | error | Canonical target returns …. |
| `CANONICAL_TARGET_REDIRECTS` | warning | Canonical target redirects to …. |
| `CANONICAL_TARGET_NOINDEX` | error | Canonical target is noindex. |
| `CANONICAL_ON_NOINDEX` | notice | Page is noindex but also has a canonical to another page (mixed signals). |
| `NOINDEX` | notice | Page is noindex (…). |
| `NOFOLLOW` | notice | Page is nofollow (…). |
| `BROKEN_LINKS` | error | N broken link(s) on this page. |
| `BROKEN_IMAGES` | warning | N broken image(s) on this page. |
| `BROKEN_RESOURCES` | warning | N broken script, stylesheet or frame URL(s) on this page. |
| `LINKS_TO_REDIRECTS` | notice | N internal link(s) go through a redirect. |
| `PAGE_REDIRECTED` | notice | The URL redirects (…). |
| `REDIRECT_CHAIN` | warning | The URL goes through N redirects. |
| `LANG_MISSING` | notice | No lang attribute on \<html>. |
| `VIEWPORT_MISSING` | warning | No mobile viewport meta tag. |
| `THIN_CONTENT` | notice | Only N words of text. |
| `CONTENT_DUPLICATE` | warning | Exactly the same text as N other page(s). |
| `MIXED_CONTENT` | warning | N resource(s) loaded over http:// on an https:// page. |
| `SLOW_RESPONSE` | notice | Server took N ms to respond. |
| `LARGE_PAGE` | notice | HTML is N KB. |
| `JS_RENDERED` | notice | Very little text in the HTML; the page is probably rendered by JavaScript, which this audit does not run. |
| `HREFLANG_INVALID` | warning | Invalid hreflang value(s): …. |

Thresholds: title 30–60 characters, meta description 70–160, thin content under 200 words, slow over 3 s, large HTML over 3 MB. Pages that are noindex or canonicalised elsewhere are left out of duplicate checks.

### Pricing

**$0.002 per audited page saved** ($2 per 1,000). Nothing else is charged:

- link checks and `link` rows: free, however many;
- `summary` rows and the `RUN_SUMMARY` record: free;
- dry runs: free;
- sites that fail to open: free;
- broken, redirecting and non-HTML URLs: free (they are not audited pages);
- with **Save only pages with issues**, pages without issues: free.

Set a **maximum cost per run** in Apify and the Actor stops before going over it. The summary then shows `stoppedBy: "costLimit"`.

Examples: a 50-page brochure site costs $0.10. A 1,000-page blog costs $2. A 10,000-page shop audited on 2,000 sample pages costs $4.

### Tips

- **Big sites**: run a dry run, then audit a sample with **Max pages per site** and an **Only crawl** pattern per section (e.g. `/blog/`).
- **Shops and faceted navigation**: tick **Ignore query strings**. Exclude `?filter=`, `/tag/` and similar.
- **Fragile servers**: lower **Max parallel requests** to 2–3.
- **Only want broken links?** Set **Link rows** to "Broken links only" and **Save only pages with issues**. Use the **Broken links & redirects** tab.
- **Scheduled monitoring**: schedule a weekly run and compare `summary` rows, or connect the dataset to Google Sheets, Slack or Zapier via Apify integrations.

### FAQ

**Does it render JavaScript?** No. It reads the HTML that servers send, which is what most search-engine first passes see. Pages whose text appears only after JavaScript are flagged `JS_RENDERED`.

**Why is a link "blocked" rather than broken?** Some sites (LinkedIn returns 999, Stack Overflow 403, many Cloudflare sites 403/429) refuse automated requests even though the link works in a browser. Calling these broken would be wrong, so they are reported separately.

**Does it obey robots.txt?** Yes, by default, for internal pages. External links are checked with a single request each.

**Is it legal?** It requests public pages, like a search-engine crawler, at a polite concurrency. Audit sites you own or are allowed to audit.

### Support

Open an issue in the Actor's **Issues** tab with the run ID. Feature requests are welcome.

# Actor input Schema

## `startUrls` (type: `array`):

One or more websites (home page or any start page). Each site is crawled separately, staying on its own domain. Example: https://crawlee.dev

## `maxPagesPerSite` (type: `integer`):

Stop after this many HTML pages per site (each saved page = 1 charged event). 0 = no limit. Use a dry run first to see how big the site is.

## `maxDepth` (type: `integer`):

How many clicks away from the start page to crawl. Empty = unlimited. Sitemap URLs count as depth 1.

## `includeSubdomains` (type: `boolean`):

Also crawl blog.example.com, shop.example.com… when starting from example.com. www and the bare domain are always treated as the same site.

## `respectRobotsTxt` (type: `boolean`):

Skip internal URLs that robots.txt disallows (they are counted in the summary).

## `useSitemaps` (type: `boolean`):

Read sitemap.xml (and sitemaps listed in robots.txt) to find pages and to check sitemap quality: broken, redirecting, noindex and orphan URLs.

## `ignoreQueryStrings` (type: `boolean`):

Treat /page?sort=x and /page as the same page. Useful for shops and faceted navigation. Tracking parameters (utm\_\*, gclid, fbclid…) are always ignored.

## `includeUrlPatterns` (type: `array`):

Crawl only pages whose URL contains one of these texts (e.g. /blog/). Wrap in slashes for a regular expression: //p/\d+$/. The start page is always audited.

## `excludeUrlPatterns` (type: `array`):

Skip pages whose URL contains one of these texts (e.g. /tag/, ?replytocom=). Regex allowed in /slashes/. Links to them are still checked.

## `checkExternalLinks` (type: `boolean`):

Check that links to other websites still work (HEAD, then GET if needed). Turn off for a faster internal-only audit.

## `checkImagesAndAssets` (type: `boolean`):

Also check every image, script, stylesheet and iframe URL used on the audited pages.

## `maxLinkChecksPerSite` (type: `integer`):

Safety cap on how many unique URLs are checked per site (internal pages first, then resources, then external). 0 = no cap. Link checks are free.

## `linkRows` (type: `string`):

Which problem URLs get their own row (with the pages linking to them). Link rows are always free.

## `onlyPagesWithIssues` (type: `boolean`):

Save (and pay for) only pages that have at least one issue. All pages are still crawled and counted in the summary.

## `dryRun` (type: `boolean`):

Only open the start page, robots.txt and sitemaps and estimate how many pages would be audited and what it would cost. Nothing is charged.

## `maxConcurrency` (type: `integer`):

Lower this (e.g. 3) for small or fragile servers.

## `userAgent` (type: `string`):

Custom User-Agent header. Empty = a current desktop Chrome string.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxPagesPerSite": 20,
  "includeSubdomains": false,
  "respectRobotsTxt": true,
  "useSitemaps": true,
  "ignoreQueryStrings": false,
  "checkExternalLinks": true,
  "checkImagesAndAssets": true,
  "maxLinkChecksPerSite": 5000,
  "linkRows": "brokenAndInternalRedirects",
  "onlyPagesWithIssues": false,
  "dryRun": false,
  "maxConcurrency": 10
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `links` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `allFields` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "maxPagesPerSite": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("vantum/seo-audit-broken-links").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "maxPagesPerSite": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("vantum/seo-audit-broken-links").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxPagesPerSite": 20
}' |
apify call vantum/seo-audit-broken-links --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vantum/seo-audit-broken-links"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tc1GLY7YZcFai93pE/builds/JZAzgDQvFOQjwUA4P/openapi.json