Website SEO Audit Crawler – Broken Links, Schema & llms.txt

Website SEO audit crawler that checks a whole site in one run: on-page SEO issues, broken internal and external links, schema.org JSON-LD errors, canonical and hreflang mistakes, sitemap and robots.txt problems, which AI crawlers (GPTBot, ClaudeBot, PerplexityBot…) your robots.txt blocks, and whether the site has a valid llms.txt. Use it instead of a generic web crawler when you need a ready-made technical SEO report with issue codes, not raw HTML.

Price: $3 per 1,000 audited pages + $0.50 per 1,000 checked external links. The site summary row and all error rows are free. No start fee.

What does Website SEO Audit Crawler do?

It crawls each website you enter (plain HTTP, no browser, same host only), starting from your URL and the XML sitemap, and returns:

• 📄 One row per page with status code, indexability and a list of issue codes ranked error → warning → notice
• 🏷 On-page SEO – title and meta description (missing, too long, too short, duplicate), H1/H2, word count (thin content), images without alt text, Open Graph and Twitter tags, lang, viewport, mixed content, response time and HTML size
• 🔗 Broken links – internal pages that answer 4xx/5xx (with the page that links to them) and every external link checked: ok, redirect, broken, blocked (bot protection), timeout
• 🧩 Structured data – JSON-LD, microdata and RDFa types; JSON-LD that does not parse; required properties missing for rich results (Product, Article, Event, FAQPage, JobPosting, Recipe, LocalBusiness, BreadcrumbList, VideoObject…)
• 🌍 Canonical & hreflang – missing/multiple/relative/cross-domain canonicals, canonicals pointing to broken or noindex pages, invalid hreflang codes (en-UK, en_US), missing self-reference or x-default, duplicate codes and missing return links between language versions
• 🤖 robots.txt & AI crawlers – status for 18 AI bots (OpenAI GPTBot / OAI-SearchBot / ChatGPT-User, Anthropic ClaudeBot / Claude-SearchBot / Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot and more): allowed, partial, blocked; plus Googlebot and Bingbot access, noindex / noai meta and X-Robots-Tag
• 📝 llms.txt – found or not, title, summary, sections, link count and format issues; llms-full.txt presence and size
• 🗺 Sitemap – sitemap files from robots.txt (index files and .gz supported), URL count, and pages that are in the sitemap but return errors, are noindex or canonicalised
• 📊 Site summary – health score, AI-readiness score, issue totals, top issues, duplicate titles and descriptions, broken link lists

Why use this SEO audit tool?

• All in one run – on-page audit, broken link checker, schema validator, hreflang checker and AI-crawler / llms.txt check. No need to chain four tools.
• Cheap – $3 per 1,000 pages, about 10× less than the most popular audit crawler, with no start fee.
• Fair – pages blocked by Cloudflare or another firewall, unreachable sites and invalid URLs produce a free error row with an explanation; you never pay for them.
• Fast & polite – plain HTTP requests (no headless browser), low default concurrency, robots.txt respected. 50 pages + 100 external links take about 30 seconds.
• Made for automation – flat JSON with stable issue codes, ready for Google Sheets, Looker Studio, Make, Zapier, n8n or an AI agent.

Use cases

• Technical SEO audits for clients – export the issue list to a spreadsheet and fix the errors first.
• Broken link monitoring – run it weekly on a schedule and get alerts when a page or outbound link breaks.
• AI search readiness (GEO) – see whether ChatGPT, Claude, Perplexity and Google AI can crawl the site and whether llms.txt is valid.
• International SEO – find hreflang codes and return-link errors before Google ignores them.
• Rich results QA – catch broken JSON-LD and missing Product / Article / Event properties after a release.
• Pre-launch and migration checks – compare status codes, canonicals and redirects before and after a site move.
• Competitor research – see which competitors block AI crawlers or publish llms.txt.

How to run an SEO audit

1. Click Try for free / Start.
2. Enter one or more websites in Websites to audit (e.g. example.com).
3. Set Max pages per site (default 50) and, if you like, Max external links to check per site.
4. Click Start. Download the results as JSON, CSV, Excel or HTML, or use the ready-made views: Overview, On-page SEO, Schema & hreflang, External links and Site summary & AI readiness.

Input

Field	Type	Description
startUrls	array of strings	Websites to audit. Each URL is a separate site (same host only). Default ["https://crawlee.dev"].
maxPagesPerSite	integer	Max HTML pages to audit per site. Default 50, max 10000.
checkExternalLinks	boolean	Check outgoing links for broken/redirected targets. Default true.
maxExternalLinksPerSite	integer	Max unique external links to check per site. Default 100. 0 = skip.
useSitemap	boolean	Read the XML sitemap(s), queue sitemap URLs and flag sitemap problems. Default true.
respectRobotsTxt	boolean	Skip URLs disallowed for User-agent: *. Default true.
includeSubdomains	boolean	Crawl subdomains as part of the site. Default false.
maxConcurrency	integer	Parallel requests. Default 5.
proxyConfiguration	object	Optional proxy, only for sites that block datacenter IPs.

Example input:

{
    "startUrls": ["https://www.python.org", "https://crawlee.dev"],
    "maxPagesPerSite": 100,
    "checkExternalLinks": true,
    "maxExternalLinksPerSite": 200
}

Output

Each row has a type: page, link, site or error. Real sample output (shortened):

Page

{
    "type": "page",
    "url": "https://crawlee.dev/js/docs/guides",
    "statusCode": 200,
    "indexable": true,
    "issuesCount": 1,
    "errorsCount": 0,
    "warningsCount": 1,
    "issues": ["TITLE_TOO_LONG"],
    "issueDetails": [{ "code": "TITLE_TOO_LONG", "severity": "warning", "message": "Title longer than 60 characters (may be truncated in search results)." }],
    "title": "Guides | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "titleLength": 64,
    "metaDescriptionLength": 146,
    "h1": ["Guides"],
    "canonical": "https://crawlee.dev/js/docs/guides",
    "canonicalStatus": "self",
    "noindex": false,
    "lang": "en",
    "hreflang": [{ "lang": "en", "url": "https://crawlee.dev/js/docs/guides" }, { "lang": "x-default", "url": "https://crawlee.dev/js/docs/guides" }],
    "jsonLdTypes": ["BreadcrumbList", "ListItem"],
    "jsonLdErrors": 0,
    "structuredDataWarnings": [],
    "wordCount": 234,
    "imagesMissingAlt": 0,
    "internalLinksCount": 50,
    "externalLinksCount": 9,
    "responseTimeMs": 319,
    "inSitemap": true,
    "depth": 1,
    "foundOn": "https://crawlee.dev/"
}

External link

{
    "type": "link",
    "url": "https://sdk.apify.com/docs/guides/getting-started",
    "linkStatus": "redirect",
    "statusCode": 200,
    "finalUrl": "https://docs.apify.com/sdk/js/docs/overview",
    "foundOn": "https://crawlee.dev/js/docs/guides",
    "foundOnCount": 4,
    "anchorText": "2.2"
}

Site summary (free)

{
    "type": "site",
    "host": "crawlee.dev",
    "pagesCrawled": 50,
    "healthScore": 68,
    "aiReadinessScore": 95,
    "topIssues": [{ "code": "TITLE_TOO_LONG", "severity": "warning", "pages": 46 }, { "code": "H1_MISSING", "severity": "warning", "pages": 3 }],
    "duplicateMetaDescriptions": [{ "value": "Crawlee helps you build and maintain your crawlers…", "count": 16, "urls": ["https://crawlee.dev/", "…"] }],
    "brokenInternalLinksCount": 0,
    "brokenExternalLinksCount": 0,
    "hreflangMissingReturnLinks": [],
    "structuredDataTypes": { "BlogPosting": 31, "BreadcrumbList": 8 },
    "robotsTxtFound": true,
    "googlebotAccess": "allowed",
    "aiCrawlers": [{ "bot": "GPTBot", "owner": "OpenAI", "purpose": "training", "status": "allowed", "matchedGroup": "wildcard" }],
    "aiCrawlersBlocked": [],
    "sitemapUrlCount": 4836,
    "llmsTxtFound": true,
    "llmsTxtLinkCount": 361,
    "llmsTxtIssues": ["LLMS_TXT_NO_SUMMARY"],
    "llmsFullTxtFound": true
}

On a news site that blocks AI bots (tested on a large US newspaper), aiCrawlersBlocked lists 16 bots, including GPTBot, ClaudeBot, PerplexityBot and Google-Extended.

Error (free) – errorCode is one of BLOCKED (403/429/Cloudflare challenge), SITE_UNREACHABLE, ROBOTS_DISALLOWED, FETCH_FAILED, INVALID_INPUT.

Issue codes

Severity	Codes
error	HTTP_4XX, HTTP_5XX, TITLE_MISSING, CANONICAL_MULTIPLE, CANONICAL_INVALID, HREFLANG_INVALID_CODE, STRUCTURED_DATA_INVALID, ERROR_IN_SITEMAP
warning	TITLE_TOO_LONG, TITLE_TOO_SHORT, TITLE_MULTIPLE, META_DESCRIPTION_MISSING, H1_MISSING, NOINDEX, CANONICAL_RELATIVE, HREFLANG_NO_SELF_REFERENCE, HREFLANG_DUPLICATE_CODE, HREFLANG_RELATIVE_URL, VIEWPORT_MISSING, IMAGES_MISSING_ALT, STRUCTURED_DATA_WARNINGS, SLOW_RESPONSE, MIXED_CONTENT, NOT_HTTPS, NOINDEX_IN_SITEMAP
notice	REDIRECTED, META_DESCRIPTION_TOO_LONG, META_DESCRIPTION_TOO_SHORT, H1_MULTIPLE, CANONICAL_MISSING, CANONICAL_TO_OTHER_URL, CANONICAL_CROSS_DOMAIN, NOFOLLOW, HREFLANG_NO_X_DEFAULT, HTML_LANG_MISSING, STRUCTURED_DATA_MISSING, OPEN_GRAPH_MISSING, THIN_CONTENT, LARGE_HTML, URL_TOO_LONG, URL_UPPERCASE, URL_UNDERSCORES, CANONICALISED_IN_SITEMAP

Health score = 100 − (pages with errors + 0.35 × pages with only warnings) ÷ pages × 100. AI-readiness score (0–100) = llms.txt (25, or 15 if it has format issues) + AI search/user bots not blocked (35) + share of pages with structured data (20) + share of indexable pages with real text (20).

Pricing

Pay-per-event – you pay only for results. Platform usage is included.

Event	Price	When it is charged
page	$0.003 per page ($3 / 1,000)	Each audited page saved to the dataset (including 404/5xx pages – they are findings)
external-link	$0.0005 per link ($0.50 / 1,000)	Each external link checked (only if Check external links is on)
Site summary	Free	One row per website
Errors	Free	Blocked pages, unreachable sites, invalid input

Examples:

Job	Cost
20 pages + 30 external links (prefilled input)	20 × $0.003 + 30 × $0.0005 = $0.075
50 pages + 100 external links (defaults)	$0.20
1,000 pages + 2,000 external links	$4.00

Tip: set Max cost per run in the run options – the crawler stops cleanly at your budget and still writes the site summary.

Use with AI agents (MCP)

This Actor works as a tool for AI agents through the Apify MCP server (Claude, ChatGPT, Cursor, VS Code, n8n…). Agents find it with search-actors ("website SEO audit", "broken links", "llms.txt"). The input is small and typed; the site row gives the agent a compact answer, the page rows the details. Example tool input:

{
    "startUrls": ["https://example.com"],
    "maxPagesPerSite": 30,
    "maxExternalLinksPerSite": 50
}

Typical prompts: "Audit example.com and list the 10 most important SEO fixes.", "Does our robots.txt block ChatGPT or Claude, and is our llms.txt valid?", "Find broken links on our docs site and the pages that contain them."

FAQ

Does it render JavaScript? No. It reads the HTML the server sends, which is what most search engines and all AI crawlers see first. Pages that build their content only in the browser show up as THIN_CONTENT / H1_MISSING – a useful signal in itself.

Which pages are crawled first? The start URL, then the links on it, then sitemap URLs, then deeper links (breadth-first), until Max pages per site is reached. Only the same host is crawled (add subdomains with Include subdomains).

Are internal links beyond the page limit checked? No – an internal URL is checked when it is crawled. Increase Max pages per site for a complete broken-link check of large sites.

What does blocked mean for an external link? The target answered 401/403/405/429/999 – usually bot protection (LinkedIn, Stack Overflow, Cloudflare). The link probably works in a browser, so it is not counted as broken.

A site returns 403 / Cloudflare challenge. What now? Those pages are free BLOCKED error rows. Enable Proxy configuration (Apify Proxy, ideally the RESIDENTIAL group) and run again. Sites behind an interactive challenge may still refuse plain HTTP crawlers.

How is AI crawler access decided? robots.txt is parsed per RFC 9309: a group that names the bot wins over User-agent: *; the longest matching rule wins and Allow wins ties. blocked = / is disallowed, partial = some paths are disallowed. Firewall-level AI blocking (e.g. Cloudflare's "Block AI bots" setting) cannot be seen from robots.txt.

Can I schedule it? Yes – use Apify Schedules (e.g. weekly) and a webhook or integration to get the results in Slack, email or Google Sheets.

Is it legal to crawl websites?

The Actor only reads publicly available pages and files (HTML, robots.txt, sitemaps, llms.txt) – no login, no forms. It respects robots.txt by default and uses low concurrency. It does not collect personal data; page content is only measured (lengths, counts, tags). Audit sites you own or have permission to audit, and follow each site's terms. If you are unsure, consult a lawyer.

Related actors

Related actors by the same developer:

• Sitemap URL Extractor – All URLs from sitemap.xml & robots.txt – list every URL of a site before or after the audit
• Website Tech Stack Detector – Bulk CMS, Analytics & Framework Lookup – see which CMS, analytics and frameworks the audited sites use
• Bulk WHOIS Lookup – RDAP, DNS Records, SSL & Redirects – domain, DNS, SSL certificate and redirect checks
• PDF to Markdown – DOCX, PPTX, XLSX & HTML for LLM & RAG – turn the audited pages into clean Markdown for AI
• Keyword Suggestions – Google, YouTube, Amazon & Bing Autocomplete – find keywords for the pages you fix
• Google Trends Scraper – Interest Over Time & Related Queries – check demand for your topics

Support & feedback

Found a false positive or need another check? Open an issue in the Issues tab with the URL – we usually respond within a day.