SiteVerdict — SEO Site Audit Crawler

Find out what is stopping your site from ranking. You get a prioritised fix list, a 0–100 site score, a PDF report for your client, and alerts when something new breaks.

The Actor crawls a whole website politely: it obeys robots.txt and starts from your XML sitemap. It runs 94 technical and on-page SEO checks on every page: status codes, redirects, indexability, canonicals, titles, meta descriptions, headings, thin and duplicate content, images, internal links, orphan pages, sitemap hygiene, HTTPS, hreflang, structured data and more. Every issue comes with why it matters, how to fix it, how much effort the fix takes, and example URLs.

Who it is for: SEO agencies running monthly client audits, in-house SEO and web teams checking releases, developers who want an SEO gate in CI, and AI agents that need a structured technical audit (MCP-ready).

Three ways people use it

1. Client audit in 5 minutes. Paste a domain and download REPORT-PDF, which you can white-label in the HTML version. The CSV of every finding goes to the dev team.
2. Release guard / monitoring. Schedule it weekly with monitoring on (the default). Each run compares against the previous one and lists new and fixed issues with the score change. A Slack or webhook alert fires only when a new critical/high issue appears or the score drops.
3. Migration QA. Crawl the staging or new site with include/exclude patterns. Catch noindex left on, canonicals pointing at the old domain, redirect chains, 404s from old links and sitemap URLs that redirect.

Before / after (sample-mode output, abridged)

{ "type": "summary", "site": "https://demo-shop.siteverdict.test/", "score": 41, "grade": "D",
  "htmlPagesAnalysed": 14, "urlsCrawled": 19, "orphanCandidates": 1,
  "bySeverity": { "critical": 1, "high": 9, "medium": 14, "low": 22 },
  "topIssues": [
    { "checkId": "status-5xx", "title": "Page returns a server error (5xx)", "severity": "critical", "affectedUrls": 1 },
    { "checkId": "sitemap-url-noindex", "title": "Sitemap lists noindex pages", "severity": "high", "affectedUrls": 1 },
    { "checkId": "orphan-page", "title": "Orphan page (in sitemap, no internal links)", "severity": "high", "affectedUrls": 1 }
  ] }

After you fix them, the next monitored run reports "scoreDelta": +27, fixedIssues: [...] and no alert.

What it checks (94 rules in 15 categories)

Category	Examples
Status & redirects	5xx, 4xx, fetch errors, redirect chains and loops, temporary redirects, internal links to redirects, meta refresh
Indexability	noindex (meta and X-Robots-Tag), nofollow, missing/multiple/relative/cross-domain/http canonicals, canonical in body, canonical target 4xx/redirect/noindex, robots-blocked pages that are linked
Titles & meta	missing, multiple, too long/short, duplicates across the site, title equals H1
Headings	missing/empty/multiple/long/duplicate H1, skipped heading levels, no subheadings
Content	thin content, low text ratio, exact duplicates (hash), near-duplicates (SimHash), lorem ipsum left in
Links	broken internal links, internal nofollow, too many links, dead ends, orphan pages (in the sitemap, no internal links), deep pages, single-inlink pages, empty and generic anchors, http links on https pages
Images	missing alt, missing width/height (CLS), broken images, very large images (HEAD size)
Technical	lang, viewport, charset, doctype, deprecated HTML, long/uppercase/underscore/parameter/non-ASCII URLs, trailing-slash duplicates, www vs non-www duplicate, favicon
Security	http pages, mixed content, missing HSTS, http not redirected to https
Sitemap & robots	missing robots.txt/sitemap; sitemap URLs that error, redirect, are noindex, non-canonical or blocked by robots.txt
Structured data & social	invalid JSON-LD, no structured data, Open Graph, Twitter card
International	invalid hreflang codes, no self-reference
Performance	slow TTFB, HTML over 1 MB, no compression, heavy inline JS
Site probes	soft 404 (a random URL that returns 200), http→https and www canonical host probes

The full list with severities and fixes is in src/catalog.ts. Each finding row says which rule fired and why.

Input

Field	Default	Notes
startUrls	—	One or more sites (home page or a section). Up to 20 per run.
maxPagesPerSite	200	Up to 5,000.
maxDepth	10	Click depth from the start URL.
useSitemap	true	Seeds the crawl from robots.txt sitemaps / /sitemap.xml. This is required for orphan detection.
includePatterns / excludePatterns	—	Globs (/blog/**) or /regex/.
includeSubdomains, ignoreQueryParameters	false	Crawl scope.
checkImages	true	Checks images with a HEAD request (broken, size).
siteProbes	true	Soft-404, http→https and www probes (3–4 extra requests).
maxConcurrency / requestDelayMs	2 / 500 ms	Polite per-host pacing. robots.txt Crawl-delay is honoured when it is larger.
monitoring, projectName	true	Keeps one snapshot per site in the named store siteverdict-seo-audit-history.
alertWebhookUrl (secret), alertOn, scoreDropThreshold	—	HTTPS Slack or any webhook.
aiFixes, aiProvider, aiApiKey (secret), aiModel, maxAiFixes	off	Optional fix snippets (e.g. a rewritten title tag, a redirect rule) written by your own Anthropic or OpenAI key.
sampleMode	false	Audits a built-in demo shop from memory. No network requests and no charge.

Output

• Dataset, with views Prioritised issues, Pages and Site summary:
  • type: "issue": one row per issue type with priority, severity, category, affectedUrls, sampleUrls, fix, effort and newSinceLastRun.
  • type: "page": one row per crawled URL with status, depth, inSitemap, title, meta description, H1, canonical, word count, TTFB and the issue IDs.
  • type: "summary": score, grade, category scores, severity counts, top issues, the monitoring diff and report links.
  • type: "ai-fix": only when AI fixes are on.
• Key-value store: REPORT (HTML), REPORT-PDF, ISSUES-CSV (every finding: URL, rule, severity, detail) and SUMMARY (agent-ready JSON). With several sites, keys are prefixed 1-, 2-, and so on.

Pricing (pay per event)

Event	Price	When
page-audited	$0.002	Each HTML page that was fetched and analysed
site-report	$0.03	Once per site: score, prioritised issues, HTML/PDF/CSV report, monitoring diff and alert
ai-fix	$0.01	Each AI fix suggestion delivered (your own AI key pays the AI provider)

Examples: a 100-page site ≈ $0.23. A 1,000-page site ≈ $2.03. A weekly 300-page monitor ≈ $2.52/month. Errors, robots-blocked URLs and non-HTML URLs are never charged, and nothing is charged in sample mode. The run stops cleanly at your max-charge limit.

FAQ

Does it render JavaScript? No. It audits the HTML your server returns, which is what most crawlers see first. JS-only content shows up as thin content or missing tags, and that is usually a real SEO risk.

Will it hammer my server? No. It sends 2 requests at a time at most, waits at least 500 ms per host, honours robots.txt Crawl-delay (up to 30 s), backs off on 429/5xx and honours Retry-After.

Can it crawl pages behind a login or bot protection? No, by design. Blocked pages are reported, not bypassed.

Is the score comparable with Ahrefs/Semrush? It is not the same formula. It is a transparent weighted score: severity × share of pages affected, per category. Track it over time rather than across tools.

Can it run on my staging site? Yes, if it is reachable on the public internet. Private/intranet addresses are refused for security.

Limits

• Up to 5,000 pages per site and 20 sites per run. Pages over 5 MB are truncated.
• Orphan detection needs a sitemap. "Orphan" means no link found within the crawled pages.
• Near-duplicate detection compares pages within one run (SimHash, distance ≤ 3).
• Image size checks use Content-Length from HEAD. Some CDNs omit it.
• Core Web Vitals are not measured (use our Speed & Revenue scanner for that).

Sources & compliance

• Data source: the public pages of the site you choose to audit, fetched with the honest user agent SiteVerdictBot/1.0 (+aman.thakran@bharatfare.com). robots.txt is parsed per RFC 9309 and obeyed for every request, including redirect hops. Sitemaps are discovered from robots.txt or /sitemap.xml. No third-party data sources are used. Checked 11 Oct 2026.
• Never bypasses CAPTCHAs, logins, paywalls or bot protection. No proxies or fingerprint evasion.
• SSRF-safe: private, loopback, link-local and cloud-metadata addresses are refused, and every redirect hop is re-checked.
• Personal data: the Actor does not collect personal data. Page text is analysed in memory, and only SEO facts (titles, headings, counts) are stored. If the pages you audit contain personal data, you are the controller of that audit (UK GDPR / PECR). Only audit sites you own or are authorised to audit.
• Secrets (webhook URL and AI key) are marked secret, redacted from logs and output, and used only for the call you enabled.
• Google requirements referenced: Google Search Central documentation (canonicalisation, robots meta, sitemaps, hreflang), read 11 Oct 2026. This is not legal advice.