# SEO Site Audit Crawler (90+ Checks, PDF Report) · SiteVerdict (`siteverdict/siteverdict-seo-site-auditor`) Actor

Crawl a whole website politely (robots.txt, sitemap seeding) and run 90+ technical and on-page SEO checks: titles, meta, canonicals, indexability, duplicates, thin content, headings, images, internal links, orphans. Site score, prioritised fixes, PDF/CSV report, monitoring.

- **URL**: https://apify.com/siteverdict/siteverdict-seo-site-auditor.md
- **Developed by:** [Aman Thakran](https://apify.com/siteverdict) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SiteVerdict — SEO Site Audit Crawler

**Find out what is stopping your site from ranking. You get a prioritised fix list, a 0–100 site score, a PDF report for your client, and alerts when something new breaks.**

The Actor crawls a whole website politely: it obeys robots.txt and starts from your XML sitemap. It runs **94 technical and on-page SEO checks** on every page: status codes, redirects, indexability, canonicals, titles, meta descriptions, headings, thin and duplicate content, images, internal links, orphan pages, sitemap hygiene, HTTPS, hreflang, structured data and more. Every issue comes with **why it matters, how to fix it, how much effort the fix takes, and example URLs**.

Who it is for: SEO agencies running monthly client audits, in-house SEO and web teams checking releases, developers who want an SEO gate in CI, and AI agents that need a structured technical audit (MCP-ready).

### Three ways people use it

1. **Client audit in 5 minutes.** Paste a domain and download `REPORT-PDF`, which you can white-label in the HTML version. The CSV of every finding goes to the dev team.
2. **Release guard / monitoring.** Schedule it weekly with `monitoring` on (the default). Each run compares against the previous one and lists **new** and **fixed** issues with the score change. A Slack or webhook alert fires only when a new critical/high issue appears or the score drops.
3. **Migration QA.** Crawl the staging or new site with include/exclude patterns. Catch noindex left on, canonicals pointing at the old domain, redirect chains, 404s from old links and sitemap URLs that redirect.

### Before / after (sample-mode output, abridged)

```json
{ "type": "summary", "site": "https://demo-shop.siteverdict.test/", "score": 41, "grade": "D",
  "htmlPagesAnalysed": 14, "urlsCrawled": 19, "orphanCandidates": 1,
  "bySeverity": { "critical": 1, "high": 9, "medium": 14, "low": 22 },
  "topIssues": [
    { "checkId": "status-5xx", "title": "Page returns a server error (5xx)", "severity": "critical", "affectedUrls": 1 },
    { "checkId": "sitemap-url-noindex", "title": "Sitemap lists noindex pages", "severity": "high", "affectedUrls": 1 },
    { "checkId": "orphan-page", "title": "Orphan page (in sitemap, no internal links)", "severity": "high", "affectedUrls": 1 }
  ] }
```

After you fix them, the next monitored run reports `"scoreDelta": +27`, `fixedIssues: [...]` and no alert.

### What it checks (94 rules in 15 categories)

| Category | Examples |
|---|---|
| Status & redirects | 5xx, 4xx, fetch errors, redirect chains and loops, temporary redirects, internal links to redirects, meta refresh |
| Indexability | noindex (meta and X-Robots-Tag), nofollow, missing/multiple/relative/cross-domain/http canonicals, canonical in body, canonical target 4xx/redirect/noindex, robots-blocked pages that are linked |
| Titles & meta | missing, multiple, too long/short, duplicates across the site, title equals H1 |
| Headings | missing/empty/multiple/long/duplicate H1, skipped heading levels, no subheadings |
| Content | thin content, low text ratio, exact duplicates (hash), **near-duplicates (SimHash)**, lorem ipsum left in |
| Links | broken internal links, internal nofollow, too many links, dead ends, **orphan pages** (in the sitemap, no internal links), deep pages, single-inlink pages, empty and generic anchors, http links on https pages |
| Images | missing alt, missing width/height (CLS), broken images, very large images (HEAD size) |
| Technical | lang, viewport, charset, doctype, deprecated HTML, long/uppercase/underscore/parameter/non-ASCII URLs, trailing-slash duplicates, www vs non-www duplicate, favicon |
| Security | http pages, mixed content, missing HSTS, http not redirected to https |
| Sitemap & robots | missing robots.txt/sitemap; sitemap URLs that error, redirect, are noindex, non-canonical or blocked by robots.txt |
| Structured data & social | invalid JSON-LD, no structured data, Open Graph, Twitter card |
| International | invalid hreflang codes, no self-reference |
| Performance | slow TTFB, HTML over 1 MB, no compression, heavy inline JS |
| Site probes | soft 404 (a random URL that returns 200), http→https and www canonical host probes |

The full list with severities and fixes is in `src/catalog.ts`. Each finding row says which rule fired and why.

### Input

| Field | Default | Notes |
|---|---|---|
| `startUrls` | — | One or more sites (home page or a section). Up to 20 per run. |
| `maxPagesPerSite` | 200 | Up to 5,000. |
| `maxDepth` | 10 | Click depth from the start URL. |
| `useSitemap` | true | Seeds the crawl from robots.txt sitemaps / `/sitemap.xml`. This is required for orphan detection. |
| `includePatterns` / `excludePatterns` | — | Globs (`/blog/**`) or `/regex/`. |
| `includeSubdomains`, `ignoreQueryParameters` | false | Crawl scope. |
| `checkImages` | true | Checks images with a HEAD request (broken, size). |
| `siteProbes` | true | Soft-404, http→https and www probes (3–4 extra requests). |
| `maxConcurrency` / `requestDelayMs` | 2 / 500 ms | Polite per-host pacing. robots.txt `Crawl-delay` is honoured when it is larger. |
| `monitoring`, `projectName` | true | Keeps one snapshot per site in the named store `siteverdict-seo-audit-history`. |
| `alertWebhookUrl` (secret), `alertOn`, `scoreDropThreshold` | — | HTTPS Slack or any webhook. |
| `aiFixes`, `aiProvider`, `aiApiKey` (secret), `aiModel`, `maxAiFixes` | off | Optional fix snippets (e.g. a rewritten title tag, a redirect rule) written by **your own** Anthropic or OpenAI key. |
| `sampleMode` | false | Audits a built-in demo shop from memory. No network requests and no charge. |

### Output

- **Dataset**, with views *Prioritised issues*, *Pages* and *Site summary*:
  - `type: "issue"`: one row per issue type with `priority`, `severity`, `category`, `affectedUrls`, `sampleUrls`, `fix`, `effort` and `newSinceLastRun`.
  - `type: "page"`: one row per crawled URL with status, depth, inSitemap, title, meta description, H1, canonical, word count, TTFB and the issue IDs.
  - `type: "summary"`: score, grade, category scores, severity counts, top issues, the monitoring diff and report links.
  - `type: "ai-fix"`: only when AI fixes are on.
- **Key-value store**: `REPORT` (HTML), `REPORT-PDF`, `ISSUES-CSV` (every finding: URL, rule, severity, detail) and `SUMMARY` (agent-ready JSON). With several sites, keys are prefixed `1-`, `2-`, and so on.

### Pricing (pay per event)

| Event | Price | When |
|---|---|---|
| `page-audited` | $0.002 | Each HTML page that was fetched and analysed |
| `site-report` | $0.03 | Once per site: score, prioritised issues, HTML/PDF/CSV report, monitoring diff and alert |
| `ai-fix` | $0.01 | Each AI fix suggestion delivered (your own AI key pays the AI provider) |

Examples: a 100-page site ≈ **$0.23**. A 1,000-page site ≈ **$2.03**. A weekly 300-page monitor ≈ **$2.52/month**. Errors, robots-blocked URLs and non-HTML URLs are never charged, and nothing is charged in sample mode. The run stops cleanly at your max-charge limit.

### FAQ

**Does it render JavaScript?** No. It audits the HTML your server returns, which is what most crawlers see first. JS-only content shows up as thin content or missing tags, and that is usually a real SEO risk.

**Will it hammer my server?** No. It sends 2 requests at a time at most, waits at least 500 ms per host, honours robots.txt `Crawl-delay` (up to 30 s), backs off on 429/5xx and honours `Retry-After`.

**Can it crawl pages behind a login or bot protection?** No, by design. Blocked pages are reported, not bypassed.

**Is the score comparable with Ahrefs/Semrush?** It is not the same formula. It is a transparent weighted score: severity × share of pages affected, per category. Track it over time rather than across tools.

**Can it run on my staging site?** Yes, if it is reachable on the public internet. Private/intranet addresses are refused for security.

### Limits

- Up to 5,000 pages per site and 20 sites per run. Pages over 5 MB are truncated.
- Orphan detection needs a sitemap. "Orphan" means *no link found within the crawled pages*.
- Near-duplicate detection compares pages within one run (SimHash, distance ≤ 3).
- Image size checks use `Content-Length` from HEAD. Some CDNs omit it.
- Core Web Vitals are not measured (use our Speed & Revenue scanner for that).

### Sources & compliance

- Data source: the public pages of the site you choose to audit, fetched with the honest user agent `SiteVerdictBot/1.0 (+aman.thakran@bharatfare.com)`. robots.txt is parsed per RFC 9309 and obeyed for every request, including redirect hops. Sitemaps are discovered from robots.txt or `/sitemap.xml`. No third-party data sources are used. Checked 11 Oct 2026.
- Never bypasses CAPTCHAs, logins, paywalls or bot protection. No proxies or fingerprint evasion.
- SSRF-safe: private, loopback, link-local and cloud-metadata addresses are refused, and every redirect hop is re-checked.
- Personal data: the Actor does not collect personal data. Page text is analysed in memory, and only SEO facts (titles, headings, counts) are stored. If the pages you audit contain personal data, you are the controller of that audit (UK GDPR / PECR). Only audit sites you own or are authorised to audit.
- Secrets (webhook URL and AI key) are marked secret, redacted from logs and output, and used only for the call you enabled.
- Google requirements referenced: Google Search Central documentation (canonicalisation, robots meta, sitemaps, hreflang), read 11 Oct 2026. This is not legal advice.

# Changelog

This Actor's version history is a separate document: https://apify.com/siteverdict/siteverdict-seo-site-auditor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Start URL of each site, e.g. https://www.example.com/. Each site is crawled and scored separately (max 20 per run).

## `maxPagesPerSite` (type: `integer`):

Upper limit of URLs crawled per site (1-5,000). You are charged only for HTML pages actually analysed.

## `maxDepth` (type: `integer`):

How many clicks away from the start URL to follow links (0 = start page only). Sitemap URLs are still checked.

## `useSitemap` (type: `boolean`):

Read the sitemap(s) listed in robots.txt (or /sitemap.xml). Needed for orphan-page and sitemap-hygiene checks.

## `includeSubdomains` (type: `boolean`):

Also crawl blog.example.com, shop.example.com... when auditing example.com. www and the bare domain always count as the same site.

## `ignoreQueryParameters` (type: `boolean`):

Treat /page?a=1 and /page as the same URL. Tracking parameters (utm\_\*, gclid, fbclid...) are always ignored.

## `includePatterns` (type: `array`):

Optional globs or /regex/ (matched against the full URL), e.g. /blog/\*\* or //products//. Empty = whole site.

## `excludePatterns` (type: `array`):

Globs or /regex/ to skip, e.g. /tag/\*\* or *?replytocom=*.

## `checkImages` (type: `boolean`):

Send HEAD requests to internal images to find broken (4xx/5xx) and heavy (>200 KB) files.

## `siteProbes` (type: `boolean`):

Three extra requests per site: http→https redirect, www vs non-www duplication and a random URL to detect soft 404s.

## `maxConcurrency` (type: `integer`):

1-4. Requests to one host are still spaced by the delay below.

## `requestDelayMs` (type: `integer`):

Minimum 250 ms. A robots.txt Crawl-delay is honoured if larger (capped at 30 s).

## `requestTimeoutSecs` (type: `integer`):

Per request, 5-60.

## `outputPageRows` (type: `boolean`):

Adds a 'page' row per crawled URL (status, title, meta, H1, canonical, word count, issue IDs). Issue and summary rows are always output.

## `monitoring` (type: `boolean`):

Keeps a snapshot per site in the named store 'siteverdict-seo-audit-history' and reports new and fixed issues plus the score change.

## `projectName` (type: `string`):

Optional. Separates monitoring history, e.g. 'client-a' (useful when auditing the same site with different settings).

## `alertWebhookUrl` (type: `string`):

Optional. Posts an alert when monitoring finds new issues. Slack incoming-webhook URLs get a Slack message; other URLs get JSON. Stored as a secret.

## `alertOn` (type: `string`):

Which changes trigger the webhook.

## `scoreDropThreshold` (type: `integer`):

Alert when the site score falls by at least this many points.

## `aiFixes` (type: `boolean`):

Off by default. Uses YOUR Anthropic or OpenAI key to write concrete fix snippets (rewritten titles, tags, redirect rules) for the top issues. Charged per fix delivered; your AI provider bills tokens separately.

## `aiProvider` (type: `string`):

Which API your key belongs to.

## `aiApiKey` (type: `string`):

Your own API key. Stored encrypted as an Apify secret, never logged or written to the output.

## `aiModel` (type: `string`):

Defaults: claude-haiku-4-5 (Anthropic) or gpt-4.1-mini (OpenAI).

## `maxAiFixes` (type: `integer`):

1-25, one per top-priority issue type.

## `sampleMode` (type: `boolean`):

Audits a built-in demo site from memory instead of your URLs: no network requests, nothing charged. Great for seeing the output format.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "maxPagesPerSite": 50,
  "maxDepth": 10,
  "useSitemap": true,
  "includeSubdomains": false,
  "ignoreQueryParameters": false,
  "checkImages": true,
  "siteProbes": true,
  "maxConcurrency": 2,
  "requestDelayMs": 500,
  "requestTimeoutSecs": 20,
  "outputPageRows": true,
  "monitoring": true,
  "alertOn": "newCriticalOrHigh",
  "scoreDropThreshold": 5,
  "aiFixes": false,
  "aiProvider": "anthropic",
  "maxAiFixes": 10,
  "sampleMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `files` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://books.toscrape.com/"
        }
    ],
    "maxPagesPerSite": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("siteverdict/siteverdict-seo-site-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://books.toscrape.com/" }],
    "maxPagesPerSite": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("siteverdict/siteverdict-seo-site-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://books.toscrape.com/"
    }
  ],
  "maxPagesPerSite": 50
}' |
apify call siteverdict/siteverdict-seo-site-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,siteverdict/siteverdict-seo-site-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/klcIbgIUe7jH4u6IF/builds/SQphTNzG3ufTQHSnH/openapi.json