# Website SEO Audit: Crawler, Broken Links & AI Readiness (`northledgertools/website-seo-audit`) Actor

Crawl a website and audit every page for SEO problems: titles, meta descriptions, headings, canonicals, Open Graph, structured data, broken links, redirects, duplicates, plus robots.txt, sitemap, llms.txt and AI crawler access. HTTP only, no browser, fast and cheap.

- **URL**: https://apify.com/northledgertools/website-seo-audit.md
- **Developed by:** [NorthLedger Tools](https://apify.com/northledgertools) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website SEO Audit: Crawler, Broken Links & AI Readiness

**Website SEO Audit** crawls your site with plain HTTP requests (no browser, so it is fast and cheap) and checks every page for the problems that cost you search traffic. For each page you get a **0-100 score** and a list of issues sorted by severity. For the whole site you get a **summary**: top issues, broken links, redirects, duplicate titles and descriptions, plus **robots.txt, sitemap, llms.txt and AI crawler access** (GPTBot, ClaudeBot, PerplexityBot and more).

Pay only for pages audited: **US$0.002 per page**, 20 times less than the most-used full-site SEO audit Actor on Apify Store (US$0.04 per page; checked 2026-10-26). External link checks, robots.txt, sitemap and llms.txt checks are free.

### What does it check?

**On every page**

- Title: missing, too short, too long, duplicated.
- Meta description: missing, too short, too long, duplicated.
- Headings: missing or multiple H1, skipped heading levels.
- Canonical tag: missing, multiple, pointing to another URL.
- Indexing: `noindex` and `nofollow` in the robots meta tag and `X-Robots-Tag` header.
- Mobile: viewport meta tag. Language: `lang` attribute. Charset.
- Social: Open Graph (title, description, image) and Twitter card tags.
- Structured data: JSON-LD found, types detected, invalid JSON flagged.
- Images: images without `alt`. Content: word count, thin content.
- Links: internal and external counts, nofollow, empty anchor text.
- Security and speed: http vs https, mixed content, redirect chains, slow responses, uncompressed HTML, oversized HTML.

**Across the site**

- Broken internal links, with the pages that link to them.
- Broken external links (404, 410, 5xx, unreachable), with the pages that link to them.
- Redirecting links that should point straight to the final URL.
- Duplicate titles, meta descriptions and H1 headings.
- `robots.txt` present and which pages it blocks.
- Sitemap found, number of URLs, and pages that are in the sitemap but not linked.
- `llms.txt` present or missing.
- **AI crawler access**: whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and others are allowed, partly blocked or blocked.

### How to use it

1. Click **Try for free**.
2. Enter your home page URL. Leave the defaults (50 pages, depth 3) for a first look.
3. Click **Start**. When it finishes, open the **Output** tab:
   - **Audit results**: one row per page with score, issue counts, title, description length and more.
   - **Site summary**: the site-wide findings.
   - **Markdown report**: a readable summary you can paste into a ticket or email.

#### Example input

```json
{
    "startUrls": [{ "url": "https://example.com" }],
    "maxPages": 200,
    "maxDepth": 4,
    "useSitemap": true,
    "checkExternalLinks": true,
    "respectRobotsTxt": true
}
```

#### Example output (one page)

```json
{
    "type": "page",
    "url": "https://example.com/pricing",
    "finalUrl": "https://example.com/pricing",
    "status": 200,
    "depth": 1,
    "responseMs": 214,
    "score": 82,
    "errors": 0,
    "warnings": 3,
    "notices": 2,
    "issues": [
        { "severity": "warning", "code": "description_missing", "message": "No meta description." },
        { "severity": "warning", "code": "canonical_missing", "message": "No canonical link tag." }
    ],
    "title": "Pricing | Example",
    "titleLength": 17,
    "h1": "Simple pricing",
    "wordCount": 412,
    "structuredDataTypes": ["Organization"],
    "internalLinks": 38,
    "externalLinks": 4
}
```

The last dataset item has `"type": "site-summary"` and contains `averageScore`, `topIssues`, `brokenInternalLinks`, `brokenExternalLinks`, `redirectedLinks`, `duplicateTitles`, `duplicateDescriptions`, `duplicateH1` and `sites` (robots.txt, sitemap, llms.txt and AI crawler access per site). URLs that fail to load are dataset items with `"type": "error"` and an `errorType` such as `HTTP_ERROR`, `TIMEOUT`, `DNS_ERROR`, `CONNECTION_ERROR`, `BLOCKED_URL` or `INVALID_URL`.

#### How the score works

Every page starts at 100. Each **error** subtracts 10, each **warning** 4 and each **notice** 1, down to a minimum of 0. Use the score to sort pages and track progress, and the issue list to know what to fix.

### Pricing

Pay per event, no subscription.

| Event | Price |
| --- | --- |
| Page audited (an HTML page that loaded successfully and was analysed) | **US$0.002** |
| Actor start | US$0.001 |

Pages that fail to load, pages blocked by robots.txt, files such as PDFs and images, and redirects to a page already audited are **not charged**. Use **Maximum pages** and Apify's **Maximum cost per run** to cap spend: the Actor stops cleanly when the budget is used.

| Site size | Cost |
| --- | --- |
| 50 pages | about US$0.10 |
| 500 pages | about US$1.00 |
| 2,000 pages | about US$4.00 |

### Use it from AI agents, MCP and the API

**Apify MCP server**: add `https://mcp.apify.com/?actors=northledgertools/website-seo-audit` to your MCP client and ask your agent to audit a site.

**HTTP API**

```bash
curl -X POST "https://api.apify.com/v2/acts/northledgertools~website-seo-audit/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"maxPages":25}'
```

**Python**

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
run = client.actor("northledgertools/website-seo-audit").call(run_input={"startUrls": [{"url": "https://example.com"}], "maxPages": 25})
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
summary = [i for i in items if i["type"] == "site-summary"][0]
print(summary["averageScore"], summary["topIssues"][:5])
```

**JavaScript**

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('northledgertools/website-seo-audit').call({ startUrls: [{ url: 'https://example.com' }], maxPages: 25 });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.at(-1).topIssues);
```

Schedule it weekly from the Apify Console and send the summary to Slack, email or a Google Sheet to catch regressions after every release.

### Use cases

- **Agencies and freelancers**: quick technical audits for prospects and clients.
- **In-house SEO**: weekly checks for broken links, missing metadata and accidental `noindex`.
- **Site migrations**: compare redirects, titles and canonicals before and after launch.
- **AI search readiness**: see if AI crawlers are blocked and whether you publish an `llms.txt` file.
- **Developers and AI agents**: structured findings that a script or LLM can act on.

### Limitations

- The crawler reads the HTML the server returns and **does not run JavaScript**. Pages rendered only in the browser will show little content. Use a browser-based audit for those.
- It does not measure Core Web Vitals, rankings, backlinks or keyword data.
- Pages behind logins and sites that block cloud IP addresses cannot be audited.
- External links are checked with a light request. Some sites answer automated requests with errors, so confirm external findings before acting.
- Heuristics such as title length are guidelines, not Google rules.

### FAQ

**Is it polite to the site?** Yes. It identifies itself as `NorthLedgerAuditBot`, respects robots.txt by default and uses a small number of parallel requests. Only audit sites you own or have permission to crawl.

**Will it audit my whole site?** It stops at **Maximum pages** (up to 2,000 per run). Raise the limit or run it on sub-sections of the site by using different start URLs.

**Does it follow subdomains?** Only if you turn on **Include subdomains**.

**Need a screenshot of the pages too?** Try our [Website Screenshot API Actor](https://apify.com/northledgertools/website-screenshot).

**Missing a check?** Open an issue in the **Issues** tab and we will take a look.

# Actor input Schema

## `startUrls` (type: `array`):

One or more pages to start from, usually your home page. Links are followed within the same site.

## `maxPages` (type: `integer`):

Maximum number of pages to fetch and audit. Each audited HTML page is billed once.

## `maxDepth` (type: `integer`):

How many clicks away from the start URL to crawl. 0 audits only the start URLs.

## `useSitemap` (type: `boolean`):

Seed the crawl with URLs from sitemap.xml (found through robots.txt). Helps find pages that are not linked.

## `includeSubdomains` (type: `boolean`):

Treat subdomains of the start URL's domain (for example blog.example.com) as part of the site.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that robots.txt disallows for this crawler. Keep this on unless you own the site.

## `checkExternalLinks` (type: `boolean`):

Check outgoing links to other sites and report the ones that return 404, 410 or a server error.

## `maxExternalLinks` (type: `integer`):

Cap on distinct external URLs checked. External checks are free.

## `concurrency` (type: `integer`):

How many pages are fetched at once. Keep it low for small sites.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "useSitemap": true,
  "includeSubdomains": false,
  "respectRobotsTxt": true,
  "checkExternalLinks": true,
  "maxExternalLinks": 100,
  "concurrency": 8
}
```

# Actor output Schema

## `pages` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("northledgertools/website-seo-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("northledgertools/website-seo-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call northledgertools/website-seo-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,northledgertools/website-seo-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BFmMy8DwlpQNBQWqe/builds/7bb1gsUC4lq5VBTfJ/openapi.json