🔍 SEO Audit Crawler - Real Core Web Vitals & JS Sites

Crawl a website and get a 0-100 SEO score for every page across meta tags, headings, content, technical SEO, structured data, links, images and accessibility. Every issue comes back with a priority and a specific fix — not just a list of what is wrong.

✅ It audits sites that block other SEO tools

Most SEO crawlers give up the moment a site returns a Cloudflare challenge or a 403 — you get an error, an empty report, or a score calculated from the "Just a moment..." interstitial instead of your actual page.

This Actor detects the block and automatically retries through Apify Unblocker, which solves the JavaScript challenge and returns the real page. Measured on w3.org, which serves a Cloudflare challenge to ordinary requests:

	Result
Normal request (what most tools get)	HTTP 403 — blocked, nothing to audit
This Actor's automatic fallback	HTTP 200 — full page audited, 807 words, 8 issues found

The escalation only happens when a site actually blocks us, so ordinary pages cost nothing extra. And if a page is still blocked after the retry, it is reported as an error and never charged — you never pay for an audit of a challenge screen.

Logins are not supported. Unblocker defeats bot detection, not authentication. Pages behind a sign-in form cannot be audited by this Actor, and no SEO crawler that respects your credentials should claim otherwise.

✅ It renders JavaScript — and tells you what that costs you

React, Next.js, Vue and Angular sites often send an almost empty <div> and build the page in the browser. An SEO tool that only reads the served HTML does not merely miss that content — it invents faults that do not exist.

Measured on quotes.toscrape.com/js, a page whose content is built client-side:

	Words found	Score	Critical issues raised
Reading the served HTML (what most tools do)	14	39 — "poor"	viewport-missing, no-content
This Actor, after rendering	227	59 — "needs improvement"	viewport-missing

The no-content critical in the first row is wrong. The page has content; the tool just could not see it. That single phantom fault dragged the score down 20 points and told the owner to fix something that was never broken.

The part no other crawler reports

Because the page is fetched twice, the Actor knows exactly how much of it depends on JavaScript — and says so:

js-dependent-content — 94% of this page's text appears only after JavaScript runs (14 words in the HTML the server sends, 227 once rendered).

That matters because Google renders JavaScript on a delayed second pass with a limited budget, while Bing, AI answer engines and social preview bots largely do not render at all. To them, this page really is 14 words. The same comparison also catches a <title> or a <link rel="canonical"> that only exists after hydration — both silently invisible to a first-pass crawl.

Rendering only happens where it is needed

renderJavaScript defaults to auto: the Actor renders a page only when the served HTML looks hollow — an empty framework mount, a script-heavy document with no structure, or content shipped as data inside a <script> tag. In a live run over python.org, react.dev, apify.com and the client-rendered page above, only the page that needed a browser got one; the three server-rendered sites were audited straight from their HTML at no extra cost or delay.

Set it to always to render every page, or off to audit exactly what the server sends.

✅ Performance from real users, not a lab

Most audit tools load a page once, from one location, on one connection, and call that "performance". This Actor uses the Chrome UX Report — what actual Chrome visitors measured on the site over the last 28 days, including INP (Interaction to Next Paint), the Core Web Vital that replaced FID and that is effectively impossible to reproduce in a synthetic lab run. Lab signals (server response time, document weight, render-blocking scripts) are measured too, so performance always carries a real number.

---

What gets checked

Category	Checks
Meta	Title pixel width (Google truncates by pixels, not characters), meta description, canonical, viewport, robots noindex, Open Graph, H1/title consistency
Headings	H1 presence and uniqueness, skipped heading levels
Content	Word count, thin content, Flesch readability, sentence length, lang declaration
Technical	HTTPS, URL length and casing, DOM size, server response time, content that only exists after JavaScript runs
Performance	LCP, INP, CLS, FCP, TTFB from real Chrome users, plus measured server response time, document weight, render-blocking scripts and lazy-loading
Schema	JSON-LD presence, detected types, invalid blocks
Links	Internal/external counts, empty hrefs, unsafe target="_blank"
Images	Missing alt text, missing width/height (a direct cause of layout shift)
Accessibility	Unlabelled inputs, unnamed buttons, missing <main> landmark, missing skip link

Each category can be switched off individually. A disabled category is excluded from the overall score rather than counted as perfect, so turning checks off cannot inflate your number.

---

Input

Only startUrls is required.

{
  "startUrls": ["https://example.com"],
  "crawlPages": true,
  "maxPages": 25,
  "maxConcurrency": 5,
  "respectRobotsTxt": true,
  "excludeUrlPatterns": ["/tag/", "utm_"],
  "fieldData": true,
  "useUnblocker": true
}

Field	Type	Default	Notes
startUrls	array	—	Required. Include the scheme
crawlPages	boolean	true	Follow internal links
maxPages	integer	25	Your budget cap — billing is per page analyzed
maxConcurrency	integer	5	Lower it if a site rate-limits you
respectRobotsTxt	boolean	true	Disallowed pages are never fetched or charged
excludeUrlPatterns	array	[]	Regex or plain substring
includeSubdomains	boolean	false	Follow links to subdomains too
fieldData	boolean	true	Real-user Core Web Vitals
fieldDataPages	integer	10	How many pages get their own field-data lookup
useUnblocker	boolean	true	Retry blocked pages through Unblocker
renderJavaScript	string	auto	auto renders only pages that need it, always renders every page, off disables it
renderTimeoutSecs	integer	30	Falls back to the served HTML on timeout; a page is never dropped
renderConcurrency	integer	2	Browser tabs open at once — capped separately from maxConcurrency
audit*	boolean	true	Nine individual category toggles

---

Output

page record

{
  "recordType": "page",
  "url": "https://example.com/pricing",
  "statusCode": 200,
  "transport": "direct",
  "overallScore": 74,
  "rating": "good",
  "categoryScores": { "meta": 88, "headings": 100, "content": 100, "technical": 88, "performance": 63, "schema": 100, "links": 100, "images": 75, "accessibility": 100 },
  "title": "Pricing - Example",
  "wordCount": 812,
  "imageCount": 14,
  "imagesMissingAlt": 2,
  "fieldData": {
    "available": true, "scope": "url",
    "lcp": 3120, "lcpRating": "needs-improvement",
    "inp": 187,  "inpRating": "good",
    "cls": 0.04, "clsRating": "good"
  },
  "issueCount": 4,
  "highPriorityCount": 1,
  "issues": [
    {
      "category": "images",
      "priority": "high",
      "code": "img-no-alt",
      "message": "2 of 14 images have no alt attribute.",
      "fix": "Add descriptive alt text. Use alt=\"\" only for decorative images."
    }
  ]
}

site-summary record

Rolls every page up and — importantly — ranks issues by how much score they are actually costing across the whole site, so you know what to fix first:

{
  "recordType": "site-summary",
  "pagesAudited": 25,
  "averageScore": 71,
  "rating": "good",
  "pagesNeedingUnblocker": 3,
  "distribution": { "excellent": 4, "good": 12, "needsImprovement": 7, "poor": 2 },
  "crawlCoverage": { "indexable": 21, "noindex": 1, "canonicalisedAway": 2, "errors": 1, "blocked": 0 },
  "duplicateTitles": [{ "value": "Products", "pages": 6, "urls": ["..."] }],
  "duplicateDescriptions": [],
  "worstPages": [{ "url": "...", "overallScore": 42 }],
  "topIssues": [
    { "code": "img-no-dimensions", "pages": 19, "scoreCost": 228, "priority": "medium", "fix": "..." }
  ]
}

crawlCoverage answers a question page-level scores cannot: how much of the site can actually appear in search at all? A site can score well page by page and still have most of it excluded by noindex or canonicalised away.

duplicateTitles and duplicateDescriptions catch a site-wide fault that no single-page check can see — six pages sharing one title compete with each other in search.

The site summary is free — you already paid per page, and the roll-up is what makes those pages actionable.

---

Scoring

Every page starts at 100 in each category and loses points only for issues found on it: 40 for critical, 25 for high, 12 for medium, 5 for low. Fixing something always moves the number by a predictable amount, which a "points awarded for things present" score does not.

A critical defect caps the whole score. A page missing its viewport tag is not mobile-friendly, and a page with 30 words has nothing to rank — no amount of clean markup elsewhere should let either read as "excellent". One critical issue caps the page at 59; two or more cap it at 39. The scoreCappedBy field tells you when a cap was applied.

Score	Rating
80-100	Excellent
60-79	Good
40-59	Needs improvement
0-39	Poor

---

Cost control

Billing is pay per page analyzed. Pages skipped by robots.txt or by excludeUrlPatterns are never fetched and never charged. Pages that error are reported for visibility, also free. maxPages is a hard cap — a 5-page preview costs a fraction of a 50-page audit.

---

Limitations

• Real-user data is only available for pages with enough traffic. The Chrome UX Report needs a minimum sample size. When a specific URL does not qualify, the Actor falls back to origin-level data and records which scope it used; small sites may get no field data at all. On-page audits always run in full.
• Field-data lookups are capped per run. The Chrome UX Report quota is 150 queries per minute shared across everyone using this Actor, so fieldDataPages limits how many pages get their own lookup rather than risking failures mid-crawl.
• auto rendering is a heuristic, not an oracle. It renders a page when the served HTML looks hollow — an empty framework mount, a script-heavy document with no structure, or content shipped as data inside a script tag. A site that hides content in an unusual way could be missed; set renderJavaScript to always if you would rather not rely on the detector. Rendering never removes checks, so auto can only ever under-render.
• Accessibility checks are structural — labels, accessible names, landmarks, skip links. This is not a full WCAG conformance audit and does not test colour contrast.
• Anti-bot protection is handled; logins are not. Unblocker solves JavaScript challenges, not authentication. Pages behind a sign-in form are out of scope.
• Scoring thresholds follow common SEO guidance, not any single vendor's algorithm. They are set at the lenient end so the report does not raise false alarms.
• One crawl per start URL, confined to that domain unless includeSubdomains is on.