Website SEO Audit: Crawler, Broken Links & AI Readiness

Website SEO Audit crawls your site with plain HTTP requests (no browser, so it is fast and cheap) and checks every page for the problems that cost you search traffic. For each page you get a 0-100 score and a list of issues sorted by severity. For the whole site you get a summary: top issues, broken links, redirects, duplicate titles and descriptions, plus robots.txt, sitemap, llms.txt and AI crawler access (GPTBot, ClaudeBot, PerplexityBot and more).

Pay only for pages audited: US$0.002 per page, 20 times less than the most-used full-site SEO audit Actor on Apify Store (US$0.04 per page; checked 2026-10-26). External link checks, robots.txt, sitemap and llms.txt checks are free.

What does it check?

On every page

• Title: missing, too short, too long, duplicated.
• Meta description: missing, too short, too long, duplicated.
• Headings: missing or multiple H1, skipped heading levels.
• Canonical tag: missing, multiple, pointing to another URL.
• Indexing: noindex and nofollow in the robots meta tag and X-Robots-Tag header.
• Mobile: viewport meta tag. Language: lang attribute. Charset.
• Social: Open Graph (title, description, image) and Twitter card tags.
• Structured data: JSON-LD found, types detected, invalid JSON flagged.
• Images: images without alt. Content: word count, thin content.
• Links: internal and external counts, nofollow, empty anchor text.
• Security and speed: http vs https, mixed content, redirect chains, slow responses, uncompressed HTML, oversized HTML.

Across the site

• Broken internal links, with the pages that link to them.
• Broken external links (404, 410, 5xx, unreachable), with the pages that link to them.
• Redirecting links that should point straight to the final URL.
• Duplicate titles, meta descriptions and H1 headings.
• robots.txt present and which pages it blocks.
• Sitemap found, number of URLs, and pages that are in the sitemap but not linked.
• llms.txt present or missing.
• AI crawler access: whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and others are allowed, partly blocked or blocked.

How to use it

1. Click Try for free.
2. Enter your home page URL. Leave the defaults (50 pages, depth 3) for a first look.
3. Click Start. When it finishes, open the Output tab:
   • Audit results: one row per page with score, issue counts, title, description length and more.
   • Site summary: the site-wide findings.
   • Markdown report: a readable summary you can paste into a ticket or email.

Example input

{
    "startUrls": [{ "url": "https://example.com" }],
    "maxPages": 200,
    "maxDepth": 4,
    "useSitemap": true,
    "checkExternalLinks": true,
    "respectRobotsTxt": true
}

Example output (one page)

{
    "type": "page",
    "url": "https://example.com/pricing",
    "finalUrl": "https://example.com/pricing",
    "status": 200,
    "depth": 1,
    "responseMs": 214,
    "score": 82,
    "errors": 0,
    "warnings": 3,
    "notices": 2,
    "issues": [
        { "severity": "warning", "code": "description_missing", "message": "No meta description." },
        { "severity": "warning", "code": "canonical_missing", "message": "No canonical link tag." }
    ],
    "title": "Pricing | Example",
    "titleLength": 17,
    "h1": "Simple pricing",
    "wordCount": 412,
    "structuredDataTypes": ["Organization"],
    "internalLinks": 38,
    "externalLinks": 4
}

The last dataset item has "type": "site-summary" and contains averageScore, topIssues, brokenInternalLinks, brokenExternalLinks, redirectedLinks, duplicateTitles, duplicateDescriptions, duplicateH1 and sites (robots.txt, sitemap, llms.txt and AI crawler access per site). URLs that fail to load are dataset items with "type": "error" and an errorType such as HTTP_ERROR, TIMEOUT, DNS_ERROR, CONNECTION_ERROR, BLOCKED_URL or INVALID_URL.

How the score works

Every page starts at 100. Each error subtracts 10, each warning 4 and each notice 1, down to a minimum of 0. Use the score to sort pages and track progress, and the issue list to know what to fix.

Pricing

Pay per event, no subscription.

Event	Price
Page audited (an HTML page that loaded successfully and was analysed)	US$0.002
Actor start	US$0.001

Pages that fail to load, pages blocked by robots.txt, files such as PDFs and images, and redirects to a page already audited are not charged. Use Maximum pages and Apify's Maximum cost per run to cap spend: the Actor stops cleanly when the budget is used.

Site size	Cost
50 pages	about US$0.10
500 pages	about US$1.00
2,000 pages	about US$4.00

Use it from AI agents, MCP and the API

Apify MCP server: add https://mcp.apify.com/?actors=northledgertools/website-seo-audit to your MCP client and ask your agent to audit a site.

HTTP API

curl -X POST "https://api.apify.com/v2/acts/northledgertools~website-seo-audit/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://example.com"}],"maxPages":25}'

Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_TOKEN")
run = client.actor("northledgertools/website-seo-audit").call(run_input={"startUrls": [{"url": "https://example.com"}], "maxPages": 25})
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
summary = [i for i in items if i["type"] == "site-summary"][0]
print(summary["averageScore"], summary["topIssues"][:5])

JavaScript

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('northledgertools/website-seo-audit').call({ startUrls: [{ url: 'https://example.com' }], maxPages: 25 });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.at(-1).topIssues);

Schedule it weekly from the Apify Console and send the summary to Slack, email or a Google Sheet to catch regressions after every release.

Use cases

• Agencies and freelancers: quick technical audits for prospects and clients.
• In-house SEO: weekly checks for broken links, missing metadata and accidental noindex.
• Site migrations: compare redirects, titles and canonicals before and after launch.
• AI search readiness: see if AI crawlers are blocked and whether you publish an llms.txt file.
• Developers and AI agents: structured findings that a script or LLM can act on.

Limitations

• The crawler reads the HTML the server returns and does not run JavaScript. Pages rendered only in the browser will show little content. Use a browser-based audit for those.
• It does not measure Core Web Vitals, rankings, backlinks or keyword data.
• Pages behind logins and sites that block cloud IP addresses cannot be audited.
• External links are checked with a light request. Some sites answer automated requests with errors, so confirm external findings before acting.
• Heuristics such as title length are guidelines, not Google rules.

FAQ

Is it polite to the site? Yes. It identifies itself as NorthLedgerAuditBot, respects robots.txt by default and uses a small number of parallel requests. Only audit sites you own or have permission to crawl.

Will it audit my whole site? It stops at Maximum pages (up to 2,000 per run). Raise the limit or run it on sub-sections of the site by using different start URLs.

Does it follow subdomains? Only if you turn on Include subdomains.

Need a screenshot of the pages too? Try our Website Screenshot API Actor.

Missing a check? Open an issue in the Issues tab and we will take a look.