# SEO Site Audit: Broken Links, Sitemap, Robots.txt (`fernandosord/seo-site-audit`) Actor

Crawl a website and list its technical SEO problems: broken links, redirect chains, missing or duplicate titles and descriptions, canonicals, noindex, hreflang, alt text, structured data, sitemap and robots.txt errors, llms.txt and HTTPS redirects. Includes a summary and a fix-first checklist.

- **URL**: https://apify.com/fernandosord/seo-site-audit.md
- **Developed by:** [Plainsource](https://apify.com/fernandosord) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What this Actor does

SEO Site Audit crawls a website from one start URL and lists its technical SEO problems, page by page, with a fix-first checklist at the end.

For each page it records the HTTP status, redirects, response time and size, title, meta description, H1s, canonical tag, noindex and nofollow (meta tag and `X-Robots-Tag` header), hreflang, `lang`, word count, images without alt text, structured data types (JSON-LD and microdata) and Open Graph tags.

It also checks:

- **Links:** every internal link it crawls and every external link (up to a limit) for 404s, server errors, timeouts, redirect chains and redirect loops, with the pages that link to each one.
- **robots.txt:** whether it exists, lines crawlers do not understand, rules that block Googlebot from the whole site or the start page, and which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) it allows.
- **Sitemaps:** found through robots.txt or `/sitemap.xml`, including sitemap indexes and `.gz` files. Sitemap URLs that return errors, redirect, are noindex, point their canonical elsewhere or are blocked by robots.txt are flagged, and so are crawled pages that are missing from the sitemap.
- **llms.txt and llms-full.txt:** whether they exist and whether `llms.txt` follows the [llms.txt format](https://llmstxt.org/).
- **HTTPS and www:** whether `http://` redirects to `https://`, and whether the www and non-www homepages end at the same URL.

Use it to check your own site before a launch or migration, to find broken links on a schedule, or to audit a client's site.

### Input

Only **Start URL** is required. All settings:

| Field | Default | What it does |
|---|---|---|
| **Start URL** (`startUrl`) | none | Where the crawl starts, usually the homepage. Links on the same host are followed; www and non-www count as the same site. |
| **Maximum pages to audit** (`maxPages`) | `100` | The crawl stops after this many HTML pages (1 to 5,000). |
| **Check external links** (`checkExternalLinks`) | `true` | Checks links to other sites with a HEAD request, then a GET if HEAD fails. |
| **Maximum external links to check** (`maxExternalLinks`) | `500` | Unique external URLs checked per run (0 to 10,000). At most 100 per external site. |
| **Include subdomains** (`includeSubdomains`) | `false` | Treats subdomains of the start host (such as `blog.example.com`) as part of the site. |
| **Also audit pages listed in the sitemap** (`crawlSitemapUrls`) | `true` | After the pages found through links, audits sitemap URLs too. This finds pages nothing links to. |
| **Respect robots.txt** (`respectRobotsTxt`) | `true` | Skips URLs the site's robots.txt does not allow for this crawler and follows its Crawl-delay. Turn off only for a site you own or may audit. |
| **Requests at once** (`maxConcurrency`) | `5` | Parallel requests (1 to 20). Requests to one host still start at least 1 second apart. |

Example:

```json
{
    "startUrl": "https://www.example.com/",
    "maxPages": 300,
    "checkExternalLinks": true,
    "maxExternalLinks": 1000
}
```

### Output

The dataset has three kinds of rows, told apart by `rowType`. The dataset has a view for each: **Pages**, **Links** and **Site checks**.

- **`site`** (one row): `robotsTxt`, `sitemap`, `llmsTxt`, `llmsFullTxt`, `homepageRedirects`, `aiCrawlers` and the site-level `issues`.
- **`page`** (one row per crawled URL): the page facts listed above, `brokenLinks` and `redirectedLinks` found on the page, `inlinks` (how many crawled pages link to it), `inSitemap`, `indexable`, `depth` (clicks from the start URL) and `issues`. `status` is `audited`, `error` (4xx, 5xx, timeout or redirect loop), `not_allowed` (redirects to a private address) or `skipped` (the run hit your cost limit).
- **`link`** (one row per linked URL with a problem): `linkType` (`internal` or `external`), `status` (`broken`, `unreachable`, `timeout`, `redirect`, `redirect_loop`, `blocked` or `not_checked`), `statusCode`, `finalUrl`, `redirectChain`, `linkedFromCount`, `linkedFrom` (up to 20 pages) and `anchorText`. Links that work are counted in the summary but get no row.

Every row has `issues`: a list of `{ severity, code, message }`, worst first. Severity is `error` (fix first), `warning` (should fix) or `notice` (worth a look).

This is a real page row from a test run on jvns.ca (both broken links and the second H1 were checked by hand):

```json
{
    "rowType": "page",
    "url": "https://jvns.ca/blogroll",
    "status": "audited",
    "statusCode": 200,
    "finalUrl": "https://jvns.ca/blogroll/",
    "redirectChain": [
        {
            "url": "https://jvns.ca/blogroll",
            "statusCode": 301
        }
    ],
    "contentType": "text/html; charset=UTF-8",
    "responseTimeMs": 63,
    "sizeBytes": 14286,
    "depth": 1,
    "foundVia": "link",
    "title": "Blogs I like",
    "titleLength": 12,
    "metaDescription": "Blogs I like",
    "metaDescriptionLength": 12,
    "h1Count": 2,
    "h1": "Julia Evans",
    "canonicalUrl": "https://jvns.ca/blogroll/",
    "robotsMeta": null,
    "xRobotsTag": null,
    "noindex": false,
    "nofollow": false,
    "indexable": true,
    "hreflang": [],
    "lang": "en",
    "wordCount": 499,
    "imagesTotal": 0,
    "imagesMissingAlt": 0,
    "imagesMissingAltExamples": [],
    "structuredDataTypes": [],
    "openGraph": {
        "title": "Blogs I like",
        "description": null,
        "image": null,
        "url": "https://jvns.ca/blogroll/",
        "type": "website"
    },
    "twitterCard": null,
    "internalLinks": 11,
    "externalLinks": 67,
    "brokenLinks": [
        "https://mickens.seas.harvard.edu/wisdom-james-mickens",
        "https://blog.safia.rocks/post/171448973555/getting-into-git-init"
    ],
    "redirectedLinks": [
        "https://jvns.ca/about",
        "https://jvns.ca/talks",
        "https://jvns.ca/blogroll",
        "https://jvns.ca/bookshelf",
        "http://kamalmarhubi.com/blog/",
        "http://kamalmarhubi.com/blog/2016/04/25/segfaults-are-our-friends-and-teachers/",
        "http://decomposition.al/",
        "http://decomposition.al/blog/2019/04/26/an-example-run-of-the-chandy-lamport-snapshot-algorithm/",
        "https://www.harihareswara.net/ces.shtml",
        "https://www.harihareswara.net/sumana/2018/07/11/0",
        "https://www.recurse.com/scout/click?t=546ea46360584b522270b8c3e5d830f8"
    ],
    "inlinks": 97,
    "inSitemap": true,
    "issueCount": 5,
    "issues": [
        {
            "severity": "error",
            "code": "broken-links-on-page",
            "message": "2 broken links."
        },
        {
            "severity": "notice",
            "code": "description-too-short",
            "message": "Meta description is 12 characters."
        },
        {
            "severity": "notice",
            "code": "h1-multiple",
            "message": "2 H1 headings."
        },
        {
            "severity": "notice",
            "code": "og-missing",
            "message": "Missing og:image."
        },
        {
            "severity": "notice",
            "code": "redirected-links-on-page",
            "message": "11 links to redirected URLs."
        }
    ],
    "error": null
}
```

#### Summary and checklist

The `OUTPUT` record in the run's key-value store has:

- `pagesAudited`, `pagesWithErrors`, `pagesSkipped`, `urlsBlockedByRobotsTxt`, `internalUrlsNotCrawled`
- `externalLinks`: found, checked, broken, redirected, blocked, timed out, not checked
- `issueCounts` by severity, `topIssues` (the 10 worst issue types) and `issuesByType` (every issue type with up to 5 example URLs and what to do)
- `checklist`: one plain-language line per issue type, worst first, for example `[error] Broken link. 3 links: Fix or remove the link on the pages listed in linkedFrom.`
- `site`: the same site checks as the `site` row
- `stoppedEarly`: why the crawl stopped before the page limit, if it did

### Issues it reports

Page issues:

| Code | Severity | Meaning |
|---|---|---|
| `page-4xx` | error | Page returns a 4xx error |
| `page-5xx` | error | Page returns a server error |
| `page-blocked` | warning | Page returns 401, 403, 429 or 503 to the crawler |
| `page-unreachable` | error | Page timed out or the connection failed |
| `redirect-loop` | error | Redirect loop or more than 10 redirects |
| `redirect-chain` | warning | 2 or more redirects before the page |
| `title-missing` | error | Missing or empty title |
| `title-duplicate` | warning | Same title as another indexable page |
| `title-too-long` / `title-too-short` | notice | Over 60 or under 10 characters |
| `title-multiple` | warning | More than one `<title>` tag |
| `description-missing` | warning | Missing or empty meta description |
| `description-duplicate` | warning | Same meta description as another indexable page |
| `description-too-long` / `description-too-short` | notice | Over 160 or under 50 characters |
| `h1-missing` | warning | No H1 |
| `h1-multiple` | notice | More than one H1 |
| `canonical-missing` | notice | No canonical tag |
| `canonical-multiple` | warning | More than one canonical tag |
| `canonical-other` | notice | Canonical points to another URL |
| `canonical-broken` | error | Canonical points to a URL that is broken or redirects |
| `canonical-chain` | warning | Canonical points to a crawled page whose own canonical points elsewhere, including two pages that point to each other |
| `noindex` | warning | noindex in meta robots or `X-Robots-Tag` |
| `nofollow-page` | notice | nofollow for the whole page |
| `blocked-for-google` | warning | robots.txt blocks Googlebot from the page |
| `hreflang-invalid` | warning | hreflang value is not a valid language code |
| `lang-missing` / `lang-invalid` | notice | Missing or malformed `<html lang>` |
| `viewport-missing` | warning | No viewport meta tag |
| `low-word-count` | notice | Fewer than 200 words of visible text |
| `images-missing-alt` | warning | `<img>` tags without an `alt` attribute (`alt=""` counts as present) |
| `structured-data-invalid` | warning | A JSON-LD block that does not parse |
| `og-missing` | notice | No `og:title` or `og:image` |
| `slow-response` | notice | Over 3 seconds to download |
| `large-html` | notice | HTML over 2 MB |
| `page-truncated` | warning | HTML over 10 MB; only the first 10 MB were audited |
| `duplicate-content` | warning | Exactly the same visible text as another indexable page |
| `broken-links-on-page` | error | The page links to broken URLs |
| `redirected-links-on-page` | notice | The page links to URLs that redirect |
| `http-links-on-https-page` | warning | Internal links that use `http://` on an HTTPS site |
| `not-in-sitemap` | notice | Indexable page missing from the sitemap |
| `orphan-page` | notice | Found only in the sitemap; no crawled page links to it |
| `non-html-start` | error | The start URL is a file, not an HTML page |

Link issues: `link-broken` (error), `link-redirect-loop` (error), `link-unreachable` (warning: timeout, refused connection or private address), `link-redirect-chain` (warning), `link-redirect` (notice) and `link-blocked` (notice: the site answered 401, 403, 429 or 503, or its robots.txt does not allow checks). A link whose domain does not exist is reported as `link-broken`.

Site issues: `robots-missing`, `robots-server-error`, `robots-html`, `robots-invalid-lines`, `robots-rules-before-user-agent`, `robots-blocks-site`, `robots-blocks-start`, `robots-no-sitemap`, `robots-too-large`, `robots-disallows-auditor`, `sitemap-missing`, `sitemap-invalid`, `sitemap-url-broken`, `sitemap-url-redirect`, `sitemap-url-noindex`, `sitemap-url-non-canonical`, `sitemap-url-blocked`, `sitemap-url-other-host`, `sitemap-invalid-lastmod`, `llms-txt-invalid`, `http-not-redirected`, `https-unavailable`, `homepage-variants-differ` and `homepage-redirect-chain`. A missing llms.txt is shown in the `llmsTxt` field but is not counted as an issue, since the file is optional.

### Pricing

You pay per event, not for compute time:

| Event | Price |
|---|---|
| Run start | $0.002 per run |
| Page audited | $0.002 per HTML page |
| External link checked | $0.0005 per unique external URL |

Included at no extra charge: the robots.txt, sitemap, llms.txt and HTTPS checks, and internal link checks. Not charged: pages that return errors, are not HTML (PDFs, images) or are blocked by robots.txt, and external links that time out, refuse the connection, block automated checks or are not allowed by robots.txt.

Example: a 200-page site with 300 unique external links costs 0.002 + 200 × 0.002 + 300 × 0.0005 = $0.552.

If a run reaches the maximum cost you set, the crawl stops, pages it did not get to are listed with status `skipped`, and only work that was done is charged. Charges are made at the end of the run, so an aborted run costs only the start event.

### Limits and good manners

- **Speed:** requests to one host start at least 1 second apart, or further apart if robots.txt sets a `Crawl-delay` (up to 10 seconds). A 100-page audit takes about 2 to 3 minutes; 1,000 pages take about 20 minutes. Give long audits a run timeout to match. If a run gets close to its timeout, the crawl stops early and the results so far are saved.
- **Robots.txt is respected by default,** for page fetches and external link checks. The crawler identifies itself as `PlainsourceSEOAudit`.
- **No JavaScript rendering.** Pages are read as the server sends them. A site that builds its content in the browser shows little text, no links and missing tags; Google renders JavaScript, so results for such sites can differ from what Google sees.
- **Status codes are reported as sent.** Some single-page apps send 404 for pages that still render in a browser, and some sites send 403 to all bots. Those show up as errors or as `blocked`.
- **What is checked:** `<a>` and `<area>` links only. Image, script and stylesheet URLs are not checked.
- **One site per run,** up to 5,000 pages and 10,000 external links. Internal links beyond the page limit are counted in `internalUrlsNotCrawled` but not checked. Sitemaps: up to 50 files, 50 MB each, and 50,000 URLs.
- **No proxies, logins or bot-check workarounds.** LinkedIn, Facebook, Instagram, X and TikTok links are not checked, because their terms forbid automated access.
- **Private networks are refused.** Private and local addresses are not fetched, including through redirects.

### FAQ

**Why does a page show `blocked` or `page-blocked`?** The server answered 401, 403, 429 or 503. Usually that is bot protection (Cloudflare and similar), which may also affect search engines. Open the URL in a browser to check.

**Why are some external links "not checked"?** They are over the limit you set, over 100 links to one site, on a site whose robots.txt does not allow checks, or on LinkedIn, Facebook, Instagram, X or TikTok.

**Does it follow nofollow links?** Links on pages marked nofollow for the whole page are not followed while robots.txt is respected. Individual `rel="nofollow"` links are still crawled and checked.

**Can I audit a staging site that blocks all crawlers?** Yes, if you own it: turn off **Respect robots.txt**. The staging site must be reachable from the internet.

**Can I call it from code or an AI agent?** Yes. Use the Apify API or the Apify MCP server with the same input. With the API client: `client.actor("fernandosord/seo-site-audit").call({ startUrl: "https://www.example.com/" })`.

Found a bug or need a check that is missing? Open an issue on this Actor's Issues tab.

# Actor input Schema

## `startUrl` (type: `string`):

The page where the crawl starts, usually the homepage, for example <code>https://www.example.com/</code>. The audit follows links on the same host (www and non-www count as the same site) and reads the site's robots.txt, sitemaps and llms.txt.

## `maxPages` (type: `integer`):

The crawl stops after this many HTML pages. Pages on the same site are requested at least 1 second apart (longer if robots.txt sets a Crawl-delay), so 100 pages take about 2 minutes.

## `checkExternalLinks` (type: `boolean`):

Also check links to other sites for 404s, server errors, timeouts and redirects. Each link gets a HEAD request, and a GET if HEAD fails.

## `maxExternalLinks` (type: `integer`):

Unique external URLs checked per run. Links beyond this number, and beyond 100 per external site, are counted but not checked.

## `includeSubdomains` (type: `boolean`):

Treat subdomains of the start URL's host (such as <code>blog.example.com</code> for <code>example.com</code>) as part of the site, so their pages are crawled and their links count as internal.

## `crawlSitemapUrls` (type: `boolean`):

After the pages found through links, audit URLs from the sitemap too. This finds pages that nothing links to and checks that sitemap URLs work. Turn off to audit only pages reachable by links.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs that the site's robots.txt does not allow for this crawler, and follow its Crawl-delay. Turn off only for a site you own or have permission to audit, such as a staging site that blocks all crawlers.

## `maxConcurrency` (type: `integer`):

How many requests run in parallel (1 to 20). Requests to the same host still start at least 1 second apart, so this mainly speeds up external link checks.

## Actor input object example

```json
{
  "startUrl": "https://books.toscrape.com/",
  "maxPages": 10,
  "checkExternalLinks": true,
  "maxExternalLinks": 500,
  "includeSubdomains": false,
  "crawlSitemapUrls": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One site row (robots.txt, sitemap, llms.txt, redirects), one row per page, and one row per broken, redirected or unverifiable link.

## `summary` (type: `string`):

Issue counts by severity and type, the top issues, and a plain-language checklist.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://books.toscrape.com/",
    "maxPages": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("fernandosord/seo-site-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrl": "https://books.toscrape.com/",
    "maxPages": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("fernandosord/seo-site-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://books.toscrape.com/",
  "maxPages": 10
}' |
apify call fernandosord/seo-site-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fernandosord/seo-site-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RcZd5N9TAdO6eirmA/builds/uQIexmdeM61p4qa7L/openapi.json