# Website SEO Audit Crawler – Broken Links, Schema & llms.txt (`openharvest/website-seo-audit`) Actor

Website SEO audit crawler: on-page issues (title, meta description, H1, canonical, hreflang, noindex), broken internal and external links, schema.org JSON-LD errors, sitemap and robots.txt checks, AI-crawler rules (GPTBot, ClaudeBot…) and llms.txt. One row per page plus a site summary.

- **URL**: https://apify.com/openharvest/website-seo-audit.md
- **Developed by:** [Enes Bulanık](https://apify.com/openharvest) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 audited pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website SEO Audit Crawler – Broken Links, Schema & llms.txt

Website SEO audit crawler that checks a whole site in one run: on-page SEO issues, broken internal and external links, schema.org JSON-LD errors, canonical and hreflang mistakes, sitemap and robots.txt problems, **which AI crawlers (GPTBot, ClaudeBot, PerplexityBot…) your robots.txt blocks, and whether the site has a valid llms.txt**. Use it instead of a generic web crawler when you need a ready-made technical SEO report with issue codes, not raw HTML.

**Price: $3 per 1,000 audited pages + $0.50 per 1,000 checked external links.** The site summary row and all error rows are free. No start fee.

### What does Website SEO Audit Crawler do?

It crawls each website you enter (plain HTTP, no browser, same host only), starting from your URL and the XML sitemap, and returns:

- 📄 **One row per page** with status code, indexability and a list of issue codes ranked error → warning → notice
- 🏷 **On-page SEO** – title and meta description (missing, too long, too short, duplicate), H1/H2, word count (thin content), images without alt text, Open Graph and Twitter tags, `lang`, viewport, mixed content, response time and HTML size
- 🔗 **Broken links** – internal pages that answer 4xx/5xx (with the page that links to them) and every external link checked: `ok`, `redirect`, `broken`, `blocked` (bot protection), `timeout`
- 🧩 **Structured data** – JSON-LD, microdata and RDFa types; JSON-LD that does not parse; required properties missing for rich results (Product, Article, Event, FAQPage, JobPosting, Recipe, LocalBusiness, BreadcrumbList, VideoObject…)
- 🌍 **Canonical & hreflang** – missing/multiple/relative/cross-domain canonicals, canonicals pointing to broken or noindex pages, invalid hreflang codes (`en-UK`, `en_US`), missing self-reference or x-default, duplicate codes and **missing return links** between language versions
- 🤖 **robots.txt & AI crawlers** – status for 18 AI bots (OpenAI GPTBot / OAI-SearchBot / ChatGPT-User, Anthropic ClaudeBot / Claude-SearchBot / Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot and more): `allowed`, `partial`, `blocked`; plus Googlebot and Bingbot access, `noindex` / `noai` meta and X-Robots-Tag
- 📝 **llms.txt** – found or not, title, summary, sections, link count and format issues; llms-full.txt presence and size
- 🗺 **Sitemap** – sitemap files from robots.txt (index files and .gz supported), URL count, and pages that are in the sitemap but return errors, are noindex or canonicalised
- 📊 **Site summary** – health score, AI-readiness score, issue totals, top issues, duplicate titles and descriptions, broken link lists

### Why use this SEO audit tool?

- **All in one run** – on-page audit, broken link checker, schema validator, hreflang checker and AI-crawler / llms.txt check. No need to chain four tools.
- **Cheap** – $3 per 1,000 pages, about 10× less than the most popular audit crawler, with no start fee.
- **Fair** – pages blocked by Cloudflare or another firewall, unreachable sites and invalid URLs produce a free `error` row with an explanation; you never pay for them.
- **Fast & polite** – plain HTTP requests (no headless browser), low default concurrency, robots.txt respected. 50 pages + 100 external links take about 30 seconds.
- **Made for automation** – flat JSON with stable issue codes, ready for Google Sheets, Looker Studio, Make, Zapier, n8n or an AI agent.

### Use cases

- **Technical SEO audits** for clients – export the issue list to a spreadsheet and fix the errors first.
- **Broken link monitoring** – run it weekly on a schedule and get alerts when a page or outbound link breaks.
- **AI search readiness (GEO)** – see whether ChatGPT, Claude, Perplexity and Google AI can crawl the site and whether `llms.txt` is valid.
- **International SEO** – find hreflang codes and return-link errors before Google ignores them.
- **Rich results QA** – catch broken JSON-LD and missing Product / Article / Event properties after a release.
- **Pre-launch and migration checks** – compare status codes, canonicals and redirects before and after a site move.
- **Competitor research** – see which competitors block AI crawlers or publish llms.txt.

### How to run an SEO audit

1. Click **Try for free** / **Start**.
2. Enter one or more websites in **Websites to audit** (e.g. `example.com`).
3. Set **Max pages per site** (default 50) and, if you like, **Max external links to check per site**.
4. Click **Start**. Download the results as JSON, CSV, Excel or HTML, or use the ready-made views: **Overview**, **On-page SEO**, **Schema & hreflang**, **External links** and **Site summary & AI readiness**.

### Input

| Field | Type | Description |
|---|---|---|
| `startUrls` | array of strings | Websites to audit. Each URL is a separate site (same host only). Default `["https://crawlee.dev"]`. |
| `maxPagesPerSite` | integer | Max HTML pages to audit per site. Default `50`, max `10000`. |
| `checkExternalLinks` | boolean | Check outgoing links for broken/redirected targets. Default `true`. |
| `maxExternalLinksPerSite` | integer | Max unique external links to check per site. Default `100`. `0` = skip. |
| `useSitemap` | boolean | Read the XML sitemap(s), queue sitemap URLs and flag sitemap problems. Default `true`. |
| `respectRobotsTxt` | boolean | Skip URLs disallowed for `User-agent: *`. Default `true`. |
| `includeSubdomains` | boolean | Crawl subdomains as part of the site. Default `false`. |
| `maxConcurrency` | integer | Parallel requests. Default `5`. |
| `proxyConfiguration` | object | Optional proxy, only for sites that block datacenter IPs. |

Example input:

```json
{
    "startUrls": ["https://www.python.org", "https://crawlee.dev"],
    "maxPagesPerSite": 100,
    "checkExternalLinks": true,
    "maxExternalLinksPerSite": 200
}
```

### Output

Each row has a `type`: `page`, `link`, `site` or `error`. Real sample output (shortened):

**Page**

```json
{
    "type": "page",
    "url": "https://crawlee.dev/js/docs/guides",
    "statusCode": 200,
    "indexable": true,
    "issuesCount": 1,
    "errorsCount": 0,
    "warningsCount": 1,
    "issues": ["TITLE_TOO_LONG"],
    "issueDetails": [{ "code": "TITLE_TOO_LONG", "severity": "warning", "message": "Title longer than 60 characters (may be truncated in search results)." }],
    "title": "Guides | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "titleLength": 64,
    "metaDescriptionLength": 146,
    "h1": ["Guides"],
    "canonical": "https://crawlee.dev/js/docs/guides",
    "canonicalStatus": "self",
    "noindex": false,
    "lang": "en",
    "hreflang": [{ "lang": "en", "url": "https://crawlee.dev/js/docs/guides" }, { "lang": "x-default", "url": "https://crawlee.dev/js/docs/guides" }],
    "jsonLdTypes": ["BreadcrumbList", "ListItem"],
    "jsonLdErrors": 0,
    "structuredDataWarnings": [],
    "wordCount": 234,
    "imagesMissingAlt": 0,
    "internalLinksCount": 50,
    "externalLinksCount": 9,
    "responseTimeMs": 319,
    "inSitemap": true,
    "depth": 1,
    "foundOn": "https://crawlee.dev/"
}
```

**External link**

```json
{
    "type": "link",
    "url": "https://sdk.apify.com/docs/guides/getting-started",
    "linkStatus": "redirect",
    "statusCode": 200,
    "finalUrl": "https://docs.apify.com/sdk/js/docs/overview",
    "foundOn": "https://crawlee.dev/js/docs/guides",
    "foundOnCount": 4,
    "anchorText": "2.2"
}
```

**Site summary (free)**

```json
{
    "type": "site",
    "host": "crawlee.dev",
    "pagesCrawled": 50,
    "healthScore": 68,
    "aiReadinessScore": 95,
    "topIssues": [{ "code": "TITLE_TOO_LONG", "severity": "warning", "pages": 46 }, { "code": "H1_MISSING", "severity": "warning", "pages": 3 }],
    "duplicateMetaDescriptions": [{ "value": "Crawlee helps you build and maintain your crawlers…", "count": 16, "urls": ["https://crawlee.dev/", "…"] }],
    "brokenInternalLinksCount": 0,
    "brokenExternalLinksCount": 0,
    "hreflangMissingReturnLinks": [],
    "structuredDataTypes": { "BlogPosting": 31, "BreadcrumbList": 8 },
    "robotsTxtFound": true,
    "googlebotAccess": "allowed",
    "aiCrawlers": [{ "bot": "GPTBot", "owner": "OpenAI", "purpose": "training", "status": "allowed", "matchedGroup": "wildcard" }],
    "aiCrawlersBlocked": [],
    "sitemapUrlCount": 4836,
    "llmsTxtFound": true,
    "llmsTxtLinkCount": 361,
    "llmsTxtIssues": ["LLMS_TXT_NO_SUMMARY"],
    "llmsFullTxtFound": true
}
```

On a news site that blocks AI bots (tested on a large US newspaper), `aiCrawlersBlocked` lists 16 bots, including `GPTBot`, `ClaudeBot`, `PerplexityBot` and `Google-Extended`.

**Error (free)** – `errorCode` is one of `BLOCKED` (403/429/Cloudflare challenge), `SITE_UNREACHABLE`, `ROBOTS_DISALLOWED`, `FETCH_FAILED`, `INVALID_INPUT`.

#### Issue codes

| Severity | Codes |
|---|---|
| error | `HTTP_4XX`, `HTTP_5XX`, `TITLE_MISSING`, `CANONICAL_MULTIPLE`, `CANONICAL_INVALID`, `HREFLANG_INVALID_CODE`, `STRUCTURED_DATA_INVALID`, `ERROR_IN_SITEMAP` |
| warning | `TITLE_TOO_LONG`, `TITLE_TOO_SHORT`, `TITLE_MULTIPLE`, `META_DESCRIPTION_MISSING`, `H1_MISSING`, `NOINDEX`, `CANONICAL_RELATIVE`, `HREFLANG_NO_SELF_REFERENCE`, `HREFLANG_DUPLICATE_CODE`, `HREFLANG_RELATIVE_URL`, `VIEWPORT_MISSING`, `IMAGES_MISSING_ALT`, `STRUCTURED_DATA_WARNINGS`, `SLOW_RESPONSE`, `MIXED_CONTENT`, `NOT_HTTPS`, `NOINDEX_IN_SITEMAP` |
| notice | `REDIRECTED`, `META_DESCRIPTION_TOO_LONG`, `META_DESCRIPTION_TOO_SHORT`, `H1_MULTIPLE`, `CANONICAL_MISSING`, `CANONICAL_TO_OTHER_URL`, `CANONICAL_CROSS_DOMAIN`, `NOFOLLOW`, `HREFLANG_NO_X_DEFAULT`, `HTML_LANG_MISSING`, `STRUCTURED_DATA_MISSING`, `OPEN_GRAPH_MISSING`, `THIN_CONTENT`, `LARGE_HTML`, `URL_TOO_LONG`, `URL_UPPERCASE`, `URL_UNDERSCORES`, `CANONICALISED_IN_SITEMAP` |

**Health score** = 100 − (pages with errors + 0.35 × pages with only warnings) ÷ pages × 100.
**AI-readiness score** (0–100) = llms.txt (25, or 15 if it has format issues) + AI search/user bots not blocked (35) + share of pages with structured data (20) + share of indexable pages with real text (20).

### Pricing

Pay-per-event – you pay only for results. Platform usage is included.

| Event | Price | When it is charged |
|---|---|---|
| `page` | **$0.003** per page ($3 / 1,000) | Each audited page saved to the dataset (including 404/5xx pages – they are findings) |
| `external-link` | **$0.0005** per link ($0.50 / 1,000) | Each external link checked (only if *Check external links* is on) |
| Site summary | **Free** | One row per website |
| Errors | **Free** | Blocked pages, unreachable sites, invalid input |

Examples:

| Job | Cost |
|---|---|
| 20 pages + 30 external links (prefilled input) | 20 × $0.003 + 30 × $0.0005 = **$0.075** |
| 50 pages + 100 external links (defaults) | **$0.20** |
| 1,000 pages + 2,000 external links | **$4.00** |

Tip: set **Max cost per run** in the run options – the crawler stops cleanly at your budget and still writes the site summary.

### Use with AI agents (MCP)

This Actor works as a tool for AI agents through the [Apify MCP server](https://mcp.apify.com) (Claude, ChatGPT, Cursor, VS Code, n8n…). Agents find it with `search-actors` ("website SEO audit", "broken links", "llms.txt"). The input is small and typed; the `site` row gives the agent a compact answer, the `page` rows the details. Example tool input:

```json
{
    "startUrls": ["https://example.com"],
    "maxPagesPerSite": 30,
    "maxExternalLinksPerSite": 50
}
```

Typical prompts: *"Audit example.com and list the 10 most important SEO fixes."*, *"Does our robots.txt block ChatGPT or Claude, and is our llms.txt valid?"*, *"Find broken links on our docs site and the pages that contain them."*

### FAQ

**Does it render JavaScript?**
No. It reads the HTML the server sends, which is what most search engines and all AI crawlers see first. Pages that build their content only in the browser show up as `THIN_CONTENT` / `H1_MISSING` – a useful signal in itself.

**Which pages are crawled first?**
The start URL, then the links on it, then sitemap URLs, then deeper links (breadth-first), until *Max pages per site* is reached. Only the same host is crawled (add subdomains with *Include subdomains*).

**Are internal links beyond the page limit checked?**
No – an internal URL is checked when it is crawled. Increase *Max pages per site* for a complete broken-link check of large sites.

**What does `blocked` mean for an external link?**
The target answered 401/403/405/429/999 – usually bot protection (LinkedIn, Stack Overflow, Cloudflare). The link probably works in a browser, so it is not counted as broken.

**A site returns 403 / Cloudflare challenge. What now?**
Those pages are free `BLOCKED` error rows. Enable **Proxy configuration** (Apify Proxy, ideally the RESIDENTIAL group) and run again. Sites behind an interactive challenge may still refuse plain HTTP crawlers.

**How is AI crawler access decided?**
robots.txt is parsed per RFC 9309: a group that names the bot wins over `User-agent: *`; the longest matching rule wins and `Allow` wins ties. `blocked` = `/` is disallowed, `partial` = some paths are disallowed. Firewall-level AI blocking (e.g. Cloudflare's "Block AI bots" setting) cannot be seen from robots.txt.

**Can I schedule it?**
Yes – use Apify Schedules (e.g. weekly) and a webhook or integration to get the results in Slack, email or Google Sheets.

### Is it legal to crawl websites?

The Actor only reads publicly available pages and files (HTML, robots.txt, sitemaps, llms.txt) – no login, no forms. It respects robots.txt by default and uses low concurrency. It does not collect personal data; page content is only measured (lengths, counts, tags). Audit sites you own or have permission to audit, and follow each site's terms. If you are unsure, consult a lawyer.

### Related actors

Related actors by the same developer:

- **Sitemap URL Extractor – All URLs from sitemap.xml & robots.txt** – list every URL of a site before or after the audit
- **Website Tech Stack Detector – Bulk CMS, Analytics & Framework Lookup** – see which CMS, analytics and frameworks the audited sites use
- **Bulk WHOIS Lookup – RDAP, DNS Records, SSL & Redirects** – domain, DNS, SSL certificate and redirect checks
- **PDF to Markdown – DOCX, PPTX, XLSX & HTML for LLM & RAG** – turn the audited pages into clean Markdown for AI
- **Keyword Suggestions – Google, YouTube, Amazon & Bing Autocomplete** – find keywords for the pages you fix
- **Google Trends Scraper – Interest Over Time & Related Queries** – check demand for your topics

### Support & feedback

Found a false positive or need another check? Open an issue in the **Issues** tab with the URL – we usually respond within a day.

# Actor input Schema

## `startUrls` (type: `array`):

Homepage or any start URL of each website. <code>example.com</code>, <code>https://www.example.com/</code> and <code>https://example.com/blog/</code> all work. Each URL is audited as a separate site (same host only).

## `maxPagesPerSite` (type: `integer`):

Maximum number of HTML pages to crawl and audit per website. You pay per audited page, so this also caps your cost.

## `checkExternalLinks` (type: `boolean`):

Check outgoing links to other domains for broken (4xx/5xx, dead domain), redirected, blocked and timed-out targets. Each checked link is a separate 'external-link' event.

## `maxExternalLinksPerSite` (type: `integer`):

Upper limit of unique external links checked per website (in the order they were found). Set to 0 to skip the check.

## `useSitemap` (type: `boolean`):

Read the sitemap(s) listed in robots.txt (or /sitemap.xml). Sitemap URLs are queued right after the links on the start page, and every page is flagged when it is in the sitemap but returns an error, is noindex or canonicalised.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that robots.txt disallows for generic crawlers (User-agent: \*). Turn off only for sites you own.

## `includeSubdomains` (type: `boolean`):

Treat subdomains (e.g. blog.example.com when auditing example.com) as part of the site: crawl them and do not count links to them as external.

## `maxConcurrency` (type: `integer`):

Maximum parallel page requests (all sites together). Keep it low to be polite to the audited server.

## `proxyConfiguration` (type: `object`):

Usually not needed. If a site answers HTTP 403 / Cloudflare challenges (reported as free BLOCKED rows), try Apify Proxy, ideally the RESIDENTIAL group.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev",
    "https://www.python.org"
  ],
  "maxPagesPerSite": 50,
  "checkExternalLinks": true,
  "maxExternalLinksPerSite": 100,
  "useSitemap": true,
  "respectRobotsTxt": true,
  "includeSubdomains": false,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped items in the default dataset (JSON, CSV, Excel and more via the dataset export).

## `files` (type: `string`):

Records saved in the default key-value store, e.g. the run summary.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxPagesPerSite": 20,
    "maxExternalLinksPerSite": 30,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("openharvest/website-seo-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxPagesPerSite": 20,
    "maxExternalLinksPerSite": 30,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("openharvest/website-seo-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxPagesPerSite": 20,
  "maxExternalLinksPerSite": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call openharvest/website-seo-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,openharvest/website-seo-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5Jin8BE5yfj8XRHds/builds/fwLqnqzWVS2GCU7M5/openapi.json