# SEO Site Audit Crawler - Title, Meta, Canonical, H1, Robots (`neverempty/seo-site-audit`) Actor

Crawls a site and returns measured facts per page: title, meta description, canonical, meta robots and X-Robots-Tag, headings, image alt, links, JSON-LD types, hreflang, redirect chain and duplicate titles. No invented 0-100 score. Every page lists what was not checked and why.

- **URL**: https://apify.com/neverempty/seo-site-audit.md
- **Developed by:** [NeverEmpty](https://apify.com/neverempty) (community)
- **Categories:** SEO tools, Developer tools, MCP servers
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$25.00 / 1,000 page analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## SEO Site Audit Crawler - Title, Meta, Canonical, H1, Robots

Crawls a site and returns one row per page: what the page actually says to a search engine, what evidence that claim rests on, and what this Actor did not look at.

There is no 0–100 score. A score is a weighting somebody chose; it is not a measurement. Every finding here points at the tag, the header or the status code it came from, so you can check it yourself.

### What it will not tell you, and why

This list is attached to every row as `notChecked`, so nobody has to guess whether a blank cell means "fine" or "not looked at".

| Not checked | Why |
|---|---|
| JavaScript-rendered content | The HTML is read as the server sends it. Scripts are not executed. |
| Core Web Vitals | Those are field measurements collected from real visitors (CrUX). They cannot be derived from one server-side fetch, so they are not reported. |
| Ranking positions | No search result data is queried. |
| Image file sizes and dimensions | Image files are not downloaded. |

`responseMs` is the time from sending the request to receiving the response headers, over a connection that may already be open, and the response may have come from a CDN cache — `cacheAgeSeconds` tells you how old it was. It is not a visitor-perceived load time and is not presented as one.

### What each page row contains

**Indexability** — `httpStatus`, `noindex`, `metaRobots`, `xRobotsTag`, `canonical`, `canonicalIsSelf`, `canonicalCount`, `redirectCount`, `redirectChain`.
Both the `robots` meta tag and the `X-Robots-Tag` response header are read; a page can be excluded by either.

**On-page** — `title`, `titleLength`, `metaDescription`, `metaDescriptionLength`, `h1`, `h1Count`, `headingJumps`, `wordCount`, `lang`, `hasViewport`, `charset`.

**Images** — `imagesTotal`, `imagesMissingAltAttr`, `imagesEmptyAlt`.
`alt=""` is counted separately from a missing `alt` attribute, because an empty alt is the correct way to mark a decorative image, not a defect.

**Links** — `internalLinks`, `externalLinks`, `nofollowLinks`.

**Structured data and sharing** — `jsonLdBlocks`, `jsonLdTypes`, `jsonLdInvalidBlocks`, `openGraphTitle`, `openGraphDescription`, `openGraphImage`, `twitterCard`, `hreflang`.
A JSON-LD block that does not parse is counted as invalid rather than silently ignored — search engines will drop it, so you should know it is there.

**Request and crawl context** — `url` (the final URL after redirects), `requestedUrl` (what was asked for), `foundOn` (the page the link was found on), `analyzed`, `contentType`, `htmlBytes`, `responseMs`, `totalMs`, `cacheAgeSeconds`, `nofollow` (the page-level directive), `checkedAt`.

**Across the crawl** — `duplicateTitleWith`, `duplicateDescriptionWith`. Each lists the other URLs in the same crawl that share the exact value.

**Findings** — `issues`, `issueCount`, `blockingIssueCount`. Each issue carries `severity`, `code`, `message` and `evidence`:

```json
{
  "severity": "info",
  "code": "duplicate-description",
  "message": "18 pages in this crawl share this exact meta description.",
  "evidence": "https://www.python.org/psf-landing/ https://www.python.org/jobs/ https://www.python.org/community/"
}
```

`severity` is `blocking`, `defect` or `info` — three named categories, not a number:

- **blocking** — the page will not appear in search results as it stands: an HTTP error, a `noindex` directive, or a canonical pointing at a different URL.
- **defect** — something is missing or broken that has a real effect: no `<title>`, no `<h1>`, images with no `alt` attribute, unparseable JSON-LD, duplicate titles.
- **info** — a measured fact worth knowing: title length, several `<h1>` elements, skipped heading levels, redirect chains, and a server response that arrived nearly empty.

**A page that arrives empty is reported as such, not left blank.** When the HTML the server sends contains almost no text and no links to the same site, the row carries an `empty-server-html` finding saying so, and pointing at the usual cause: the page builds its content with JavaScript, which this Actor does not run. Measured example: `nhk.or.jp` returns 6 KB of HTML with 4 words and no links, while `gov.uk` returns 798 words and 54 links from the same kind of request.

### Pages it did not analyse are still returned

A crawler that quietly drops what it could not read is indistinguishable from a crawler that found nothing. Every skipped URL comes back with `analyzed: false` and a `reason`:

| Situation | What the row says |
|---|---|
| robots.txt disallows the path | Not fetched, and the row says so. Verified against `wordpress.org/wp-admin/`, which is disallowed for all crawlers. |
| The URL is not HTML | `Not analysed because it is not HTML (Content-Type: image/png)` |
| The request failed | The transport error, after the configured retries |

Skipped pages are not charged.

### robots.txt is obeyed by default, and obeyed exactly

The crawler requests each URL exactly as it was found, including the trailing slash. That sounds trivial and is not: stripping the trailing slash from `/wp-admin/` makes it stop matching `Disallow: /wp-admin/`, and the crawler would fetch a path the site asked it not to. It also manufactures redirects — on a site whose links all end in a slash, requesting the slashless form produces a 301 on every page, which then looks like a site defect that does not exist. On a 20-page crawl of python.org, fixing this took the redirect findings from 19 down to 3. The remaining 3 are real.

`Allow` beats `Disallow` when it is more specific, `*` and `$` in patterns are honoured, and comments are stripped — including trailing comments on a rule line, which otherwise corrupt the path being matched.

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `startUrls` | array | `https://www.python.org/` | Where the crawl begins. Internal links found on these pages are followed up to the page limit. Links to other sites are counted, not crawled. |
| `maxPages` | number | `20` | How many pages to analyse before stopping. Range 1–1000. |
| `respectRobots` | boolean | `true` | When on, URLs that robots.txt disallows are reported, not fetched. |
| `timeoutSecs` | number | `20` | Timeout for one page. Range 3–120. |
| `maxRetries` | number | `3` | Retries for temporary failures — timeout, connection reset, HTTP 429, HTTP 5xx. A 404 is never retried, because that is already an answer. Range 1–10. |

### Speed and cost

Measured: **20 pages of python.org in 1.25 s**, 2.01 MB of HTML. No proxies and no third-party API, so the only cost is compute time.

You are charged per page analysed. Pages that were skipped — disallowed by robots.txt, not HTML, or unreachable — are returned with a reason and are not charged.

### Typical uses

- **Before a migration.** Record canonical, redirect chains and `noindex` across the site, then run it again afterwards and compare.
- **Finding duplicate titles and descriptions.** The crawl reports which other URLs share each exact value, so the fix is obvious.
- **Checking a client site you did not build.** Blocking issues first: HTTP errors, `noindex`, canonicals pointing elsewhere.
- **Structured data review.** Which JSON-LD types exist per page, and which blocks are invalid.

### FAQ

**Why is there no SEO score?**
Because a score is not a measurement. Two tools will give the same page different scores, and neither number can be checked. Facts and their evidence can be.

**Why does it report Core Web Vitals as not checked instead of estimating them?**
Core Web Vitals come from real visitors' browsers. A number produced from one server-side fetch would not be that measurement, whatever it was called.

**Does it run JavaScript?**
No. If a site renders its content client-side, that content is not in the HTML this reads — which is why `notChecked` says so on every row rather than letting a low `wordCount` look like a finding.

**Can it crawl pages robots.txt disallows?**
`respectRobots` can be turned off, but leaving it on is the default for a reason: those URLs are returned with a reason instead, so nothing disappears silently.

### Other tools by NeverEmpty

Every NeverEmpty Actor follows the same rule: it never returns an empty result to mean two different things, and it only charges for rows that actually carry an answer.

- **[link-checker](https://apify.com/neverempty/link-checker)** - broken links on a site, with blocked told apart from dead
- **[domain-lookup](https://apify.com/neverempty/domain-lookup)** - WHOIS, DNS, MX, SPF, DMARC and SSL expiry for a list of domains
- **[website-contacts](https://apify.com/neverempty/website-contacts)** - emails, phones and social profiles from any website, no invented addresses

### Support

Found a value you believe is wrong? Open an issue on the Issues tab with the URL and the row you got. Reports about incorrect data are fixed first.

# Actor input Schema

## `startUrls` (type: `array`):

Where the crawl begins. Internal links found on these pages are followed, up to the page limit. Links to other sites are counted but not crawled.

## `maxPages` (type: `number`):

How many pages to analyse before stopping. You are charged per page that was actually analysed; pages that were skipped are reported with a reason and are not charged.

## `respectRobots` (type: `boolean`):

When on, a URL that robots.txt disallows for this crawler is not fetched; it is returned as a row saying so. Turning this off means fetching pages the site asked crawlers not to fetch.

## `timeoutSecs` (type: `number`):

How long to wait for one page before giving up on it.

## `maxRetries` (type: `number`):

How many times to retry a request that failed for a temporary reason - timeout, connection reset, HTTP 429 or HTTP 5xx. A page that returns 404 is not retried, because that is already an answer.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.python.org/"
    }
  ],
  "maxPages": 20,
  "respectRobots": true,
  "timeoutSecs": 20,
  "maxRetries": 3
}
```

# Actor output Schema

## `results` (type: `string`):

All pages produced by this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.python.org/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("neverempty/seo-site-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://www.python.org/" }] }

# Run the Actor and wait for it to finish
run = client.actor("neverempty/seo-site-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.python.org/"
    }
  ]
}' |
apify call neverempty/seo-site-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neverempty/seo-site-audit"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bcSbiV34FkGeRDmub/builds/ecgCkKnIXBhrPKOnm/openapi.json