# 🔓 SEO Audit Crawler - Core Web Vitals & Blocked Sites (`acebuilds/seo-audit-crawler`) Actor

Crawl any site, score every page 0-100, get a fix for every issue. REAL Chrome user Core Web Vitals including INP - not lab guesses. Audits meta tags, technical SEO, schema, broken links, images and accessibility. Gets past Cloudflare blocks that stop other SEO crawlers.

- **URL**: https://apify.com/acebuilds/seo-audit-crawler.md
- **Developed by:** [Ace](https://apify.com/acebuilds) (community)
- **Categories:** SEO tools, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $21.00 / 1,000 page analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🔍 SEO Audit Crawler - Real Core Web Vitals & JS Sites

Crawl a website and get a **0-100 SEO score for every page** across meta tags, headings,
content, technical SEO, structured data, links, images and accessibility. Every issue
comes back with a **priority and a specific fix** — not just a list of what is wrong.

### ✅ It audits sites that block other SEO tools

Most SEO crawlers give up the moment a site returns a Cloudflare challenge or a 403 —
you get an error, an empty report, or a score calculated from the "Just a moment..."
interstitial instead of your actual page.

**This Actor detects the block and automatically retries through Apify Unblocker**, which
solves the JavaScript challenge and returns the real page. Measured on `w3.org`, which
serves a Cloudflare challenge to ordinary requests:

| | Result |
|---|---|
| Normal request (what most tools get) | **HTTP 403 — blocked, nothing to audit** |
| This Actor's automatic fallback | **HTTP 200 — full page audited, 807 words, 8 issues found** |

The escalation only happens when a site actually blocks us, so ordinary pages cost
nothing extra. And if a page is still blocked after the retry, it is reported as an error
and **never charged** — you never pay for an audit of a challenge screen.

> **Logins are not supported.** Unblocker defeats bot detection, not authentication. Pages
> behind a sign-in form cannot be audited by this Actor, and no SEO crawler that respects
> your credentials should claim otherwise.

### ✅ It renders JavaScript — and tells you what that costs you

React, Next.js, Vue and Angular sites often send an almost empty `<div>` and build the
page in the browser. An SEO tool that only reads the served HTML does not merely *miss*
that content — it **invents faults that do not exist**.

Measured on `quotes.toscrape.com/js`, a page whose content is built client-side:

| | Words found | Score | Critical issues raised |
|---|---|---|---|
| Reading the served HTML (what most tools do) | **14** | **39 — "poor"** | `viewport-missing`, **`no-content`** |
| This Actor, after rendering | **227** | **59 — "needs improvement"** | `viewport-missing` |

The `no-content` critical in the first row is **wrong**. The page has content; the tool
just could not see it. That single phantom fault dragged the score down 20 points and
told the owner to fix something that was never broken.

#### The part no other crawler reports

Because the page is fetched twice, the Actor knows exactly how much of it depends on
JavaScript — and says so:

> **`js-dependent-content`** — 94% of this page's text appears only after JavaScript runs
> (14 words in the HTML the server sends, 227 once rendered).

That matters because Google renders JavaScript on a **delayed second pass with a limited
budget**, while Bing, AI answer engines and social preview bots largely do not render at
all. To them, this page really is 14 words. The same comparison also catches a `<title>`
or a `<link rel="canonical">` that only exists after hydration — both silently invisible
to a first-pass crawl.

#### Rendering only happens where it is needed

`renderJavaScript` defaults to **`auto`**: the Actor renders a page only when the served
HTML looks hollow — an empty framework mount, a script-heavy document with no structure,
or content shipped as data inside a `<script>` tag. In a live run over `python.org`,
`react.dev`, `apify.com` and the client-rendered page above, **only the page that needed
a browser got one**; the three server-rendered sites were audited straight from their
HTML at no extra cost or delay.

Set it to `always` to render every page, or `off` to audit exactly what the server sends.

### ✅ Performance from real users, not a lab

Most audit tools load a page once, from one location, on one connection, and call that
"performance". This Actor uses the **Chrome UX Report** — what actual Chrome visitors
measured on the site over the last 28 days, including **INP (Interaction to Next Paint)**,
the Core Web Vital that replaced FID and that is effectively impossible to reproduce in a
synthetic lab run. Lab signals (server response time, document weight, render-blocking
scripts) are measured too, so performance always carries a real number.

***

### What gets checked

| Category | Checks |
|---|---|
| **Meta** | Title **pixel width** (Google truncates by pixels, not characters), meta description, canonical, viewport, robots noindex, Open Graph, H1/title consistency |
| **Headings** | H1 presence and uniqueness, skipped heading levels |
| **Content** | Word count, thin content, **Flesch readability**, sentence length, `lang` declaration |
| **Technical** | HTTPS, URL length and casing, DOM size, server response time, **content that only exists after JavaScript runs** |
| **Performance** | LCP, INP, CLS, FCP, TTFB from real Chrome users, plus measured server response time, document weight, render-blocking scripts and lazy-loading |
| **Schema** | JSON-LD presence, detected types, invalid blocks |
| **Links** | Internal/external counts, empty hrefs, unsafe `target="_blank"` |
| **Images** | Missing alt text, missing width/height (a direct cause of layout shift) |
| **Accessibility** | Unlabelled inputs, unnamed buttons, missing `<main>` landmark, missing skip link |

Each category can be switched off individually. A disabled category is **excluded** from
the overall score rather than counted as perfect, so turning checks off cannot inflate
your number.

***

### Input

Only `startUrls` is required.

```json
{
  "startUrls": ["https://example.com"],
  "crawlPages": true,
  "maxPages": 25,
  "maxConcurrency": 5,
  "respectRobotsTxt": true,
  "excludeUrlPatterns": ["/tag/", "utm_"],
  "fieldData": true,
  "useUnblocker": true
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `startUrls` | array | — | **Required.** Include the scheme |
| `crawlPages` | boolean | `true` | Follow internal links |
| `maxPages` | integer | `25` | **Your budget cap** — billing is per page analyzed |
| `maxConcurrency` | integer | `5` | Lower it if a site rate-limits you |
| `respectRobotsTxt` | boolean | `true` | Disallowed pages are never fetched or charged |
| `excludeUrlPatterns` | array | `[]` | Regex or plain substring |
| `includeSubdomains` | boolean | `false` | Follow links to subdomains too |
| `fieldData` | boolean | `true` | Real-user Core Web Vitals |
| `fieldDataPages` | integer | `10` | How many pages get their own field-data lookup |
| `useUnblocker` | boolean | `true` | Retry blocked pages through Unblocker |
| `renderJavaScript` | string | `auto` | `auto` renders only pages that need it, `always` renders every page, `off` disables it |
| `renderTimeoutSecs` | integer | `30` | Falls back to the served HTML on timeout; a page is never dropped |
| `renderConcurrency` | integer | `2` | Browser tabs open at once — capped separately from `maxConcurrency` |
| `audit*` | boolean | `true` | Nine individual category toggles |

***

### Output

#### `page` record

```json
{
  "recordType": "page",
  "url": "https://example.com/pricing",
  "statusCode": 200,
  "transport": "direct",
  "overallScore": 74,
  "rating": "good",
  "categoryScores": { "meta": 88, "headings": 100, "content": 100, "technical": 88, "performance": 63, "schema": 100, "links": 100, "images": 75, "accessibility": 100 },
  "title": "Pricing - Example",
  "wordCount": 812,
  "imageCount": 14,
  "imagesMissingAlt": 2,
  "fieldData": {
    "available": true, "scope": "url",
    "lcp": 3120, "lcpRating": "needs-improvement",
    "inp": 187,  "inpRating": "good",
    "cls": 0.04, "clsRating": "good"
  },
  "issueCount": 4,
  "highPriorityCount": 1,
  "issues": [
    {
      "category": "images",
      "priority": "high",
      "code": "img-no-alt",
      "message": "2 of 14 images have no alt attribute.",
      "fix": "Add descriptive alt text. Use alt=\"\" only for decorative images."
    }
  ]
}
```

#### `site-summary` record

Rolls every page up and — importantly — **ranks issues by how much score they are
actually costing across the whole site**, so you know what to fix first:

```json
{
  "recordType": "site-summary",
  "pagesAudited": 25,
  "averageScore": 71,
  "rating": "good",
  "pagesNeedingUnblocker": 3,
  "distribution": { "excellent": 4, "good": 12, "needsImprovement": 7, "poor": 2 },
  "crawlCoverage": { "indexable": 21, "noindex": 1, "canonicalisedAway": 2, "errors": 1, "blocked": 0 },
  "duplicateTitles": [{ "value": "Products", "pages": 6, "urls": ["..."] }],
  "duplicateDescriptions": [],
  "worstPages": [{ "url": "...", "overallScore": 42 }],
  "topIssues": [
    { "code": "img-no-dimensions", "pages": 19, "scoreCost": 228, "priority": "medium", "fix": "..." }
  ]
}
```

`crawlCoverage` answers a question page-level scores cannot: **how much of the site can
actually appear in search at all?** A site can score well page by page and still have most
of it excluded by `noindex` or canonicalised away.

`duplicateTitles` and `duplicateDescriptions` catch a site-wide fault that no single-page
check can see — six pages sharing one title compete with each other in search.

The site summary is **free** — you already paid per page, and the roll-up is what makes
those pages actionable.

***

### Scoring

Every page starts at 100 in each category and loses points only for issues **found on
it**: 40 for critical, 25 for high, 12 for medium, 5 for low. Fixing something always
moves the number by a predictable amount, which a "points awarded for things present"
score does not.

**A critical defect caps the whole score.** A page missing its viewport tag is not
mobile-friendly, and a page with 30 words has nothing to rank — no amount of clean markup
elsewhere should let either read as "excellent". One critical issue caps the page at 59;
two or more cap it at 39. The `scoreCappedBy` field tells you when a cap was applied.

| Score | Rating |
|---|---|
| 80-100 | Excellent |
| 60-79 | Good |
| 40-59 | Needs improvement |
| 0-39 | Poor |

***

### Cost control

Billing is **pay per page analyzed**. Pages skipped by `robots.txt` or by
`excludeUrlPatterns` are never fetched and never charged. Pages that error are reported
for visibility, also free. `maxPages` is a hard cap — a 5-page preview costs a fraction
of a 50-page audit.

***

### Limitations

- **Real-user data is only available for pages with enough traffic.** The Chrome UX
  Report needs a minimum sample size. When a specific URL does not qualify, the Actor
  falls back to origin-level data and records which scope it used; small sites may get
  no field data at all. On-page audits always run in full.
- **Field-data lookups are capped per run.** The Chrome UX Report quota is 150 queries
  per minute shared across everyone using this Actor, so `fieldDataPages` limits how many
  pages get their own lookup rather than risking failures mid-crawl.
- **`auto` rendering is a heuristic, not an oracle.** It renders a page when the served
  HTML looks hollow — an empty framework mount, a script-heavy document with no structure,
  or content shipped as data inside a script tag. A site that hides content in an unusual
  way could be missed; set `renderJavaScript` to `always` if you would rather not rely on
  the detector. Rendering never *removes* checks, so `auto` can only ever under-render.
- **Accessibility checks are structural** — labels, accessible names, landmarks, skip
  links. This is not a full WCAG conformance audit and does not test colour contrast.
- **Anti-bot protection is handled; logins are not.** Unblocker solves JavaScript
  challenges, not authentication. Pages behind a sign-in form are out of scope.
- **Scoring thresholds follow common SEO guidance**, not any single vendor's algorithm.
  They are set at the lenient end so the report does not raise false alarms.
- One crawl per start URL, confined to that domain unless `includeSubdomains` is on.

# Actor input Schema

## `startUrls` (type: `array`):

One or more URLs to audit. Include the scheme, e.g. https://example.com

## `crawlPages` (type: `boolean`):

Follow internal links from the start URLs. Turn off to audit only the URLs you provide.

## `maxPages` (type: `integer`):

Hard cap on pages analyzed. You are charged per page, so this is your budget control.

## `maxConcurrency` (type: `integer`):

Pages fetched in parallel. Lower this if a site rate-limits you.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by the site's robots.txt. Skipped pages are never fetched and never charged.

## `excludeUrlPatterns` (type: `array`):

Regular expressions or plain substrings. Matching URLs are never crawled or charged.

## `includeSubdomains` (type: `boolean`):

Also follow links to subdomains of the start URL's domain.

## `fieldData` (type: `boolean`):

Look up what real Chrome users actually measured on this site over the last 28 days — LCP, INP, CLS, FCP and TTFB. This is field data, not a lab estimate from a single crawl.

## `fieldDataPages` (type: `integer`):

How many pages get their own real-user lookup. The quota is shared across everyone using this Actor, so it is capped rather than run on every page.

## `useUnblocker` (type: `boolean`):

When a site returns a Cloudflare challenge or a 403, retry through Apify Unblocker, which solves the JavaScript challenge and returns the real page. Most SEO crawlers simply fail here. Only used when a site actually blocks us, so ordinary pages cost nothing extra. Note: this defeats bot detection, NOT logins - pages behind a sign-in form cannot be audited.

## `renderJavaScript` (type: `string`):

Auto (recommended) renders only pages whose served HTML is an empty app shell, so server-rendered sites cost nothing extra. Always renders every page. Off audits the HTML exactly as the server sends it.

## `renderTimeoutSecs` (type: `integer`):

How long a single page may take to render before we fall back to auditing its served HTML. A page is never dropped for timing out.

## `renderConcurrency` (type: `integer`):

Browser tabs open at once. Each one costs memory, so this is capped separately from the crawl's concurrency. Raise it only if you have also raised the run's memory.

## `auditMetaTags` (type: `boolean`):

Title, meta description, canonical, viewport, robots and Open Graph.

## `auditHeadings` (type: `boolean`):

H1 presence and uniqueness, and skipped heading levels.

## `auditContent` (type: `boolean`):

Word count, thin content and language declaration.

## `auditTechnical` (type: `boolean`):

HTTPS, URL hygiene, DOM size and server response time.

## `auditPerformance` (type: `boolean`):

Core Web Vitals from real user data.

## `auditSchema` (type: `boolean`):

JSON-LD presence, detected types and invalid blocks.

## `auditLinks` (type: `boolean`):

Internal/external counts, empty hrefs and unsafe target=\_blank.

## `auditImages` (type: `boolean`):

Missing alt text and missing width/height, which causes layout shift.

## `auditAccessibility` (type: `boolean`):

Unlabelled form inputs, unnamed buttons and missing main landmark.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com"
  ],
  "crawlPages": true,
  "maxPages": 25,
  "maxConcurrency": 5,
  "respectRobotsTxt": true,
  "excludeUrlPatterns": [
    "/tag/",
    "/category/",
    "utm_"
  ],
  "includeSubdomains": false,
  "fieldData": true,
  "fieldDataPages": 10,
  "useUnblocker": true,
  "renderJavaScript": "auto",
  "renderTimeoutSecs": 30,
  "renderConcurrency": 2,
  "auditMetaTags": true,
  "auditHeadings": true,
  "auditContent": true,
  "auditTechnical": true,
  "auditPerformance": true,
  "auditSchema": true,
  "auditLinks": true,
  "auditImages": true,
  "auditAccessibility": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("acebuilds/seo-audit-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://example.com"] }

# Run the Actor and wait for it to finish
run = client.actor("acebuilds/seo-audit-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://example.com"
  ]
}' |
apify call acebuilds/seo-audit-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,acebuilds/seo-audit-crawler"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/68txvUIWc66jyJmiW/builds/JiHsgmOQHjrMgdFnq/openapi.json