# Website Technical Health Auditor (`automation-lab/website-technical-health-auditor`) Actor

Crawl public websites and export page-level HTTP, redirect, indexability, metadata, link, performance, security-header, and severity-coded issue records.

- **URL**: https://apify.com/automation-lab/website-technical-health-auditor.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.68 / 1,000 item extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Technical Health Auditor

Crawl a bounded public website and export one technical health record per page. The Actor captures HTTP status, redirect chains, indexability, titles, descriptions, canonicals, internal and external link counts, response performance, security headers, and severity-coded issues.

Use this website technical audit when you need machine-readable page evidence rather than a single domain score. It follows same-site links from each supplied root while enforcing page, depth, concurrency, and discovery limits.

### What does Website Technical Health Auditor do?

The Actor requests public HTTP(S) pages, follows redirects safely, parses server-rendered HTML, discovers internal pages, and writes a normalized row for every audited page.

Each row includes:

- requested and final URLs;
- final HTTP status and redirect hops;
- response duration and body size;
- title and meta description values and lengths;
- canonical URL and robots directives;
- a derived indexability value;
- internal and external link counts;
- six common security headers;
- deterministic findings with category, code, severity, and message.

It does not run Lighthouse, execute page JavaScript, grade SSL certificates, resolve DNS health, or claim Core Web Vitals. Response time is an observed HTTP request duration, not a lab or field Web Vitals score.

### Who is it for?

- **SEO teams** audit indexability, canonicals, titles, and descriptions across a site.
- **Web agencies** create evidence-backed launch and migration checklists.
- **Developers** identify HTTP failures, redirect chains, oversized HTML, and missing headers.
- **Content teams** locate metadata problems before publishing campaigns.
- **Operations teams** schedule the same bounded audit and compare exported datasets in their own workflow.
- **Data teams** load stable page rows into spreadsheets, warehouses, or BI tools.

### Why use this Actor?

A homepage-only check can miss problems deeper in a site. This Actor turns internal-link discovery into page-level records while keeping the run bounded.

Unlike `Website Health Report`, which produces domain-oriented health scores and DNS/SSL checks, this Actor focuses on crawling multiple internal pages and exporting the direct evidence behind each finding. It intentionally does not invent an aggregate score.

Compared with an on-page SEO crawler, it adds explicit redirect chains, response-size and timing warnings, selected HTTP security headers, and severity-coded findings in a single page record.

### What technical checks are included?

| Category | Evidence and findings |
| --- | --- |
| HTTP | Final status, failed requests, server errors, redirect chain |
| Indexability | HTML response, successful status, meta robots and X-Robots-Tag `noindex` |
| Metadata | Missing or unusually sized title and meta description |
| Canonical | Missing, invalid, or resolved canonical URL |
| Links | Unique internal and external link counts, no-internal-links warning |
| Performance | HTTP response duration and downloaded body size |
| Security | HTTPS plus HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, and Permissions-Policy |

Findings use `critical`, `high`, `medium`, `low`, or `info` severity. Thresholds are deterministic diagnostics, not legal, accessibility, ranking, or vulnerability guarantees.

### Input parameters

| Field | Type | Default | Description |
| --- | --- | ---: | --- |
| `startUrls` | array | required | One to twenty public HTTP(S) roots or pages |
| `maxPages` | integer | `10` | Maximum total page rows, from 1 to 1,000 |
| `maxDepth` | integer | `2` | Internal-link hops from each supplied URL; `0` audits only supplied pages |
| `maxConcurrency` | integer | `5` | Simultaneous page requests, from 1 to 20 |
| `maxLinksPerPage` | integer | `20` | Internal links accepted for discovery from each page |
| `requestTimeoutSecs` | integer | `20` | Timeout for each bounded page request |
| `includeSubdomains` | boolean | `false` | Whether subdomains belong to crawl scope |
| `followQueryParameters` | boolean | `false` | Whether distinct non-tracking query strings become separate pages |

URLs must resolve exclusively to public network addresses. Localhost, credentials in URLs, private IP ranges, non-HTTP protocols, and redirects to private destinations are rejected.

### Get started

1. Open the Actor input page.
2. Add one or more website roots to **Website URLs**.
3. Keep `maxPages` small for a first audit.
4. Set `maxDepth` to `0` for only the submitted pages, or increase it for internal discovery.
5. Start the run.
6. Open the default dataset and sort by `highestSeverity` or `issueCount`.
7. Export JSON, CSV, Excel, XML, or RSS using Apify dataset tools.

A practical first input is:

```json
{
  "startUrls": [{ "url": "https://apify.com" }],
  "maxPages": 10,
  "maxDepth": 1,
  "maxConcurrency": 3
}
```

### Output fields

| Field | Meaning |
| --- | --- |
| `url` | Requested normalized page URL |
| `finalUrl` | Final URL after the redirect chain |
| `startUrl` | Supplied root responsible for this crawl |
| `depth` | Link distance from the root |
| `statusCode` | Final HTTP status, or null after a request failure |
| `contentType` | Response Content-Type |
| `redirectChain` | Ordered source, status, and destination for each redirect |
| `responseTimeMs` | Total request time including redirects |
| `responseBytes` | Downloaded response body bytes |
| `title` | Page title, when available |
| `metaDescription` | Meta description, when available |
| `canonicalUrl` | Absolute canonical URL, when valid |
| `robotsDirectives` | Combined normalized robots directives |
| `indexable` | Derived status/HTML/noindex assessment |
| `internalLinkCount` | Unique internal HTTP(S) links |
| `externalLinkCount` | Unique external HTTP(S) links |
| `securityHeaders` | Selected response header values |
| `highestSeverity` | Highest finding severity or `none` |
| `issueCount` | Number of findings |
| `issues` | Structured finding objects |
| `error` | Bounded fetch error when a request failed |
| `auditedAt` | UTC audit timestamp |

All website-derived fields can be absent or null. Consumers should use `issues[].code` for automation and `message` for display.

### Output example

```json
{
  "url": "https://apify.com/",
  "finalUrl": "https://apify.com/",
  "startUrl": "https://apify.com/",
  "depth": 0,
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "redirectChain": [],
  "redirectCount": 0,
  "responseTimeMs": 312,
  "responseBytes": 184220,
  "title": "Apify: Full-stack web scraping and data extraction platform",
  "titleLength": 60,
  "metaDescription": "Cloud platform for web scraping, browser automation, and data extraction.",
  "metaDescriptionLength": 76,
  "canonicalUrl": "https://apify.com/",
  "robotsDirectives": [],
  "indexable": true,
  "internalLinkCount": 42,
  "externalLinkCount": 6,
  "securityHeaders": {
    "strict-transport-security": "max-age=31536000",
    "content-security-policy": null,
    "x-content-type-options": "nosniff",
    "x-frame-options": null,
    "referrer-policy": "strict-origin-when-cross-origin",
    "permissions-policy": null
  },
  "issueCount": 3,
  "highestSeverity": "low",
  "issues": [
    {
      "code": "MISSING_CONTENT_SECURITY_POLICY",
      "severity": "low",
      "category": "security",
      "message": "Response is missing the content-security-policy header."
    }
  ],
  "error": null,
  "auditedAt": "2026-09-07T20:00:00.000Z"
}
```

Values above illustrate the output shape. Live site values can change between runs.

### How much does it cost to audit website technical health?

This Actor uses pay-per-event pricing:

- a **$0.005 audit start** fee per run;
- **$0.0028 per page audited** on the BRONZE tier;
- lower per-page rates on higher tiers.

At BRONZE, an audit that returns 10 pages costs about **$0.033**. A 100-page audit costs about **$0.285**. Failed page requests still produce a useful, severity-coded page record and count as an audited page. Pages never scheduled because a limit was reached are not charged.

Always check the current Apify pricing panel before relying on an estimate. User platform-usage terms and tier eligibility may also apply.

### Crawl scope and limits

The Actor discovers links from HTML anchor elements only. It does not execute JavaScript to reveal client-rendered navigation.

Scope is based on the hostname of each supplied root, with a normalized `www` prefix. Subdomains are excluded unless `includeSubdomains` is enabled. Media, archive, script, stylesheet, document, and common binary extensions are not scheduled as pages.

The page limit applies across all supplied roots. If you supply many websites with a small `maxPages`, earlier roots can consume the available rows. Run separate Tasks when each website needs an independent quota.

Tracking parameters such as `utm_*`, `fbclid`, and `gclid` are removed. All query strings are removed by default to avoid crawl traps. Enable `followQueryParameters` only when query pages are meaningful and bounded.

### Severity and automation

Use stable codes instead of parsing messages. For example:

- fail a deployment when `highestSeverity` is `critical`;
- create a ticket for `HTTP_ERROR`, `NOINDEX`, or `INVALID_CANONICAL`;
- group low-severity missing-header findings for later hardening;
- compare code counts between scheduled run datasets;
- filter non-indexable rows before an SEO review.

Severity prioritizes likely impact but cannot know your business intent. A deliberate `noindex` or missing CSP can be correct for a specific page.

### Redirect and request behavior

The Actor follows up to eight redirect hops and records every hop. Each destination is rechecked against public-address restrictions before it is requested.

Transient timeouts, HTTP 429, and server failures receive up to three bounded attempts with backoff and jitter. Parser and input failures are not blindly retried. A page that exhausts requests becomes a `REQUEST_FAILED` row so scheduled audits retain evidence of the failure.

### Performance interpretation

`responseTimeMs` measures this Actor's HTTP transaction from Apify's environment. It includes redirect time but excludes browser rendering, layout, JavaScript execution, and user-network latency.

The Actor reports:

- a medium warning above 1.5 seconds;
- a high warning above 3 seconds;
- a medium large-HTML warning above 2 MB.

Use Lighthouse Website Audit when you need browser-based performance, accessibility, best-practice, or SEO scores.

### Security interpretation

Security-header checks report presence, not policy correctness. A present CSP can still be permissive. HSTS only protects HTTPS deployments after compatible clients receive the header. This Actor is not a penetration test and does not probe vulnerabilities.

Only audit sites you are authorized to access, respect site terms and crawl capacity, and keep concurrency conservative for smaller servers.

### Schedule recurring website audits

Create an Apify Task with a stable input, then attach a schedule. Each run creates a separate default dataset. Export or compare those datasets in your own automation to detect new issue codes, status changes, or slower responses.

The Actor itself does not store a historical baseline or send alerts. Apify webhooks, Make, Zapier, a warehouse, or your own job can process successful run datasets.

### Export to spreadsheets and data pipelines

Dataset rows are flat for common page fields and retain nested arrays for redirect and issue detail. CSV and Excel exports serialize nested values; JSON is best when preserving full structured findings.

Common workflows include:

1. scheduled crawl to dataset;
2. filter rows by severity;
3. send high-impact findings to a ticketing system;
4. archive all page rows in a warehouse;
5. chart issue counts by run date.

### Use the Apify API

Replace `APIFY_TOKEN` with your token.

#### cURL

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-technical-health-auditor/runs?token=APIFY_TOKEN&waitForFinish=120" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com"}],"maxPages":10,"maxDepth":1}'
```

#### JavaScript

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-technical-health-auditor').call({
  startUrls: [{ url: 'https://apify.com' }],
  maxPages: 10,
  maxDepth: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("automation-lab/website-technical-health-auditor").call(run_input={
    "startUrls": [{"url": "https://apify.com"}],
    "maxPages": 10,
    "maxDepth": 1,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)
```

### Use with Apify MCP

The same Actor-scoped MCP URL works in Claude Desktop, Cursor, and VS Code.

Add the Actor to Claude Code:

```bash
claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"
```

#### Claude Desktop setup

Add this server object to Claude Desktop's MCP configuration:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"
    }
  }
}
```

#### Cursor setup

Add the same Actor-scoped server to `.cursor/mcp.json` or Cursor's MCP settings:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"
    }
  }
}
```

#### VS Code setup

Use the same remote server object in VS Code's MCP settings:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"
    }
  }
}
```

Example prompts:

- “Audit up to 20 pages of https://apify.com and summarize high-severity findings.”
- “Check these three launch pages without following links and return missing security headers.”
- “Run a bounded technical audit and group non-indexable pages by robots directive.”

### Tips for reliable audits

- Begin with 5–10 pages before increasing scope.
- Use `maxDepth: 0` for a list of exact landing pages.
- Keep query parameters disabled unless essential.
- Reduce concurrency when a server responds with rate limits.
- Split unrelated sites into separate runs for independent quotas.
- Treat response-time changes as signals, not synthetic performance benchmarks.
- Review critical request failures before interpreting an empty content field.

### Limitations

- Public anonymous HTTP(S) pages only.
- Server-rendered HTML only; no browser execution.
- No authenticated pages, forms, sitemaps, or JavaScript navigation.
- No robots.txt policy evaluation or sitemap completeness analysis.
- No DNS, SSL certificate, accessibility, visual, or Core Web Vitals audit.
- No historical baseline, diff, alert delivery, or aggregate health score.
- Link counts describe links on the page; external destinations are not separately requested.
- Website changes, bot controls, and geographic delivery can affect results.

### Legality and responsible use

Audit public sites lawfully and only at a rate appropriate for the operator. Do not use the Actor to bypass authentication, probe private networks, or overload a service. The SSRF guard blocks non-public destinations, including redirects.

Website content and headers remain owned by their respective publishers. Apply your own retention, privacy, and compliance requirements to exported datasets.

### Troubleshooting

#### Why did I receive a `REQUEST_FAILED` record?

The page could not be fetched after bounded retries. Inspect `error`, reduce concurrency, confirm that the URL is public, and try the exact page in a browser. Persistent authentication or bot challenges are outside this HTTP-only Actor's scope.

#### Why were fewer pages audited than expected?

The site may expose few server-rendered internal links, links may use excluded file extensions, query variants may be deduplicated, or `maxDepth` may have been reached. Increase limits deliberately rather than enabling unbounded URL variants.

#### Why is a page marked non-indexable?

A page is indexable only when it returns successful HTML and has no detected `noindex` directive. Review `statusCode`, `contentType`, and `robotsDirectives`. The result does not test search-engine canonicals, robots.txt, or actual index membership.

#### Why are security issues low severity?

Most missing headers are context-dependent hardening opportunities. Insecure HTTP is high severity; missing HSTS is medium; other absent headers default to low. Review the page's purpose and full policies before remediation.

### FAQ

#### Can I audit several websites in one run?

Yes. Supply up to twenty roots. The total `maxPages` limit is shared across them.

#### Does it check broken external links?

No. It counts external links but does not request every external destination. Use Sitewide Broken Link Checker for destination-level link records.

#### Does it execute JavaScript?

No. The HTTP-first implementation is fast and economical for server-rendered pages. Client-only links and metadata are not visible.

#### Can I schedule it?

Yes. Save the input as an Apify Task and attach a schedule. Use webhooks or integrations to compare and route each new dataset.

#### Are failed requests charged?

A failed request that produces a structured page audit record is charged as one page. A URL that is never scheduled because a limit was reached is not charged.

### Related automation-lab Actors

- [Website Health Report](https://apify.com/automation-lab/website-health-report) for one-row URL health scores, DNS, and SSL evidence.
- [Lighthouse Website Audit](https://apify.com/automation-lab/website-lighthouse-seo-audit) for browser-based Lighthouse scores.
- [Multi-page On-page SEO Audit Crawler](https://apify.com/automation-lab/multi-page-on-page-seo-audit) for deeper content and heading signals.
- [Sitewide Broken Link Checker](https://apify.com/automation-lab/sitewide-broken-link-checker) for destination-level broken-link checks.

### Support

For reproducible questions, include a redacted input, run URL, affected page URL, and the relevant issue code. Do not include credentials or private URLs.

# Actor input Schema

## `startUrls` (type: `array`):

One to twenty public HTTP(S) pages or website roots to audit.

## `maxPages` (type: `integer`):

Maximum total page audit records across all supplied websites.

## `maxDepth` (type: `integer`):

Maximum number of internal-link hops from each supplied URL. Use 0 to audit only supplied pages.

## `maxConcurrency` (type: `integer`):

Maximum pages requested at once. Lower this for small or rate-limited websites.

## `maxLinksPerPage` (type: `integer`):

Maximum internal page links accepted for crawl discovery from each page.

## `requestTimeoutSecs` (type: `integer`):

Seconds allowed for each page request before bounded retries are exhausted.

## `includeSubdomains` (type: `boolean`):

Follow links to subdomains of each supplied website root.

## `followQueryParameters` (type: `boolean`):

Treat URLs with different non-tracking query parameters as separate pages.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxPages": 10,
  "maxDepth": 2,
  "maxConcurrency": 5,
  "maxLinksPerPage": 20,
  "requestTimeoutSecs": 20,
  "includeSubdomains": false,
  "followQueryParameters": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

Default dataset containing every audited page.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/website-technical-health-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("automation-lab/website-technical-health-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call automation-lab/website-technical-health-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,automation-lab/website-technical-health-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3AtGYPQOOxb8NxUk8/builds/2Yzy5R4ewbvBuVBRO/openapi.json