Website Technical Health Auditor

Crawl a bounded public website and export one technical health record per page. The Actor captures HTTP status, redirect chains, indexability, titles, descriptions, canonicals, internal and external link counts, response performance, security headers, and severity-coded issues.

Use this website technical audit when you need machine-readable page evidence rather than a single domain score. It follows same-site links from each supplied root while enforcing page, depth, concurrency, and discovery limits.

What does Website Technical Health Auditor do?

The Actor requests public HTTP(S) pages, follows redirects safely, parses server-rendered HTML, discovers internal pages, and writes a normalized row for every audited page.

Each row includes:

• requested and final URLs;
• final HTTP status and redirect hops;
• response duration and body size;
• title and meta description values and lengths;
• canonical URL and robots directives;
• a derived indexability value;
• internal and external link counts;
• six common security headers;
• deterministic findings with category, code, severity, and message.

It does not run Lighthouse, execute page JavaScript, grade SSL certificates, resolve DNS health, or claim Core Web Vitals. Response time is an observed HTTP request duration, not a lab or field Web Vitals score.

Who is it for?

• SEO teams audit indexability, canonicals, titles, and descriptions across a site.
• Web agencies create evidence-backed launch and migration checklists.
• Developers identify HTTP failures, redirect chains, oversized HTML, and missing headers.
• Content teams locate metadata problems before publishing campaigns.
• Operations teams schedule the same bounded audit and compare exported datasets in their own workflow.
• Data teams load stable page rows into spreadsheets, warehouses, or BI tools.

Why use this Actor?

A homepage-only check can miss problems deeper in a site. This Actor turns internal-link discovery into page-level records while keeping the run bounded.

Unlike Website Health Report, which produces domain-oriented health scores and DNS/SSL checks, this Actor focuses on crawling multiple internal pages and exporting the direct evidence behind each finding. It intentionally does not invent an aggregate score.

Compared with an on-page SEO crawler, it adds explicit redirect chains, response-size and timing warnings, selected HTTP security headers, and severity-coded findings in a single page record.

What technical checks are included?

Category	Evidence and findings
HTTP	Final status, failed requests, server errors, redirect chain
Indexability	HTML response, successful status, meta robots and X-Robots-Tag noindex
Metadata	Missing or unusually sized title and meta description
Canonical	Missing, invalid, or resolved canonical URL
Links	Unique internal and external link counts, no-internal-links warning
Performance	HTTP response duration and downloaded body size
Security	HTTPS plus HSTS, CSP, X-Content-Type-Options, X-Frame-Options, Referrer-Policy, and Permissions-Policy

Findings use critical, high, medium, low, or info severity. Thresholds are deterministic diagnostics, not legal, accessibility, ranking, or vulnerability guarantees.

Input parameters

Field	Type	Default	Description
startUrls	array	required	One to twenty public HTTP(S) roots or pages
maxPages	integer	10	Maximum total page rows, from 1 to 1,000
maxDepth	integer	2	Internal-link hops from each supplied URL; 0 audits only supplied pages
maxConcurrency	integer	5	Simultaneous page requests, from 1 to 20
maxLinksPerPage	integer	20	Internal links accepted for discovery from each page
requestTimeoutSecs	integer	20	Timeout for each bounded page request
includeSubdomains	boolean	false	Whether subdomains belong to crawl scope
followQueryParameters	boolean	false	Whether distinct non-tracking query strings become separate pages

URLs must resolve exclusively to public network addresses. Localhost, credentials in URLs, private IP ranges, non-HTTP protocols, and redirects to private destinations are rejected.

Get started

1. Open the Actor input page.
2. Add one or more website roots to Website URLs.
3. Keep maxPages small for a first audit.
4. Set maxDepth to 0 for only the submitted pages, or increase it for internal discovery.
5. Start the run.
6. Open the default dataset and sort by highestSeverity or issueCount.
7. Export JSON, CSV, Excel, XML, or RSS using Apify dataset tools.

A practical first input is:

{
  "startUrls": [{ "url": "https://apify.com" }],
  "maxPages": 10,
  "maxDepth": 1,
  "maxConcurrency": 3
}

Output fields

Field	Meaning
url	Requested normalized page URL
finalUrl	Final URL after the redirect chain
startUrl	Supplied root responsible for this crawl
depth	Link distance from the root
statusCode	Final HTTP status, or null after a request failure
contentType	Response Content-Type
redirectChain	Ordered source, status, and destination for each redirect
responseTimeMs	Total request time including redirects
responseBytes	Downloaded response body bytes
title	Page title, when available
metaDescription	Meta description, when available
canonicalUrl	Absolute canonical URL, when valid
robotsDirectives	Combined normalized robots directives
indexable	Derived status/HTML/noindex assessment
internalLinkCount	Unique internal HTTP(S) links
externalLinkCount	Unique external HTTP(S) links
securityHeaders	Selected response header values
highestSeverity	Highest finding severity or none
issueCount	Number of findings
issues	Structured finding objects
error	Bounded fetch error when a request failed
auditedAt	UTC audit timestamp

All website-derived fields can be absent or null. Consumers should use issues[].code for automation and message for display.

Output example

{
  "url": "https://apify.com/",
  "finalUrl": "https://apify.com/",
  "startUrl": "https://apify.com/",
  "depth": 0,
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "redirectChain": [],
  "redirectCount": 0,
  "responseTimeMs": 312,
  "responseBytes": 184220,
  "title": "Apify: Full-stack web scraping and data extraction platform",
  "titleLength": 60,
  "metaDescription": "Cloud platform for web scraping, browser automation, and data extraction.",
  "metaDescriptionLength": 76,
  "canonicalUrl": "https://apify.com/",
  "robotsDirectives": [],
  "indexable": true,
  "internalLinkCount": 42,
  "externalLinkCount": 6,
  "securityHeaders": {
    "strict-transport-security": "max-age=31536000",
    "content-security-policy": null,
    "x-content-type-options": "nosniff",
    "x-frame-options": null,
    "referrer-policy": "strict-origin-when-cross-origin",
    "permissions-policy": null
  },
  "issueCount": 3,
  "highestSeverity": "low",
  "issues": [
    {
      "code": "MISSING_CONTENT_SECURITY_POLICY",
      "severity": "low",
      "category": "security",
      "message": "Response is missing the content-security-policy header."
    }
  ],
  "error": null,
  "auditedAt": "2026-09-07T20:00:00.000Z"
}

Values above illustrate the output shape. Live site values can change between runs.

How much does it cost to audit website technical health?

This Actor uses pay-per-event pricing:

• a $0.005 audit start fee per run;
• $0.0028 per page audited on the BRONZE tier;
• lower per-page rates on higher tiers.

At BRONZE, an audit that returns 10 pages costs about $0.033. A 100-page audit costs about $0.285. Failed page requests still produce a useful, severity-coded page record and count as an audited page. Pages never scheduled because a limit was reached are not charged.

Always check the current Apify pricing panel before relying on an estimate. User platform-usage terms and tier eligibility may also apply.

Crawl scope and limits

The Actor discovers links from HTML anchor elements only. It does not execute JavaScript to reveal client-rendered navigation.

Scope is based on the hostname of each supplied root, with a normalized www prefix. Subdomains are excluded unless includeSubdomains is enabled. Media, archive, script, stylesheet, document, and common binary extensions are not scheduled as pages.

The page limit applies across all supplied roots. If you supply many websites with a small maxPages, earlier roots can consume the available rows. Run separate Tasks when each website needs an independent quota.

Tracking parameters such as utm_*, fbclid, and gclid are removed. All query strings are removed by default to avoid crawl traps. Enable followQueryParameters only when query pages are meaningful and bounded.

Severity and automation

Use stable codes instead of parsing messages. For example:

• fail a deployment when highestSeverity is critical;
• create a ticket for HTTP_ERROR, NOINDEX, or INVALID_CANONICAL;
• group low-severity missing-header findings for later hardening;
• compare code counts between scheduled run datasets;
• filter non-indexable rows before an SEO review.

Severity prioritizes likely impact but cannot know your business intent. A deliberate noindex or missing CSP can be correct for a specific page.

Redirect and request behavior

The Actor follows up to eight redirect hops and records every hop. Each destination is rechecked against public-address restrictions before it is requested.

Transient timeouts, HTTP 429, and server failures receive up to three bounded attempts with backoff and jitter. Parser and input failures are not blindly retried. A page that exhausts requests becomes a REQUEST_FAILED row so scheduled audits retain evidence of the failure.

Performance interpretation

responseTimeMs measures this Actor's HTTP transaction from Apify's environment. It includes redirect time but excludes browser rendering, layout, JavaScript execution, and user-network latency.

The Actor reports:

• a medium warning above 1.5 seconds;
• a high warning above 3 seconds;
• a medium large-HTML warning above 2 MB.

Use Lighthouse Website Audit when you need browser-based performance, accessibility, best-practice, or SEO scores.

Security interpretation

Security-header checks report presence, not policy correctness. A present CSP can still be permissive. HSTS only protects HTTPS deployments after compatible clients receive the header. This Actor is not a penetration test and does not probe vulnerabilities.

Only audit sites you are authorized to access, respect site terms and crawl capacity, and keep concurrency conservative for smaller servers.

Schedule recurring website audits

Create an Apify Task with a stable input, then attach a schedule. Each run creates a separate default dataset. Export or compare those datasets in your own automation to detect new issue codes, status changes, or slower responses.

The Actor itself does not store a historical baseline or send alerts. Apify webhooks, Make, Zapier, a warehouse, or your own job can process successful run datasets.

Export to spreadsheets and data pipelines

Dataset rows are flat for common page fields and retain nested arrays for redirect and issue detail. CSV and Excel exports serialize nested values; JSON is best when preserving full structured findings.

Common workflows include:

1. scheduled crawl to dataset;
2. filter rows by severity;
3. send high-impact findings to a ticketing system;
4. archive all page rows in a warehouse;
5. chart issue counts by run date.

Use the Apify API

Replace APIFY_TOKEN with your token.

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~website-technical-health-auditor/runs?token=APIFY_TOKEN&waitForFinish=120" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://apify.com"}],"maxPages":10,"maxDepth":1}'

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/website-technical-health-auditor').call({
  startUrls: [{ url: 'https://apify.com' }],
  maxPages: 10,
  maxDepth: 1,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("automation-lab/website-technical-health-auditor").call(run_input={
    "startUrls": [{"url": "https://apify.com"}],
    "maxPages": 10,
    "maxDepth": 1,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Use with Apify MCP

The same Actor-scoped MCP URL works in Claude Desktop, Cursor, and VS Code.

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"

Claude Desktop setup

Add this server object to Claude Desktop's MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"
    }
  }
}

Cursor setup

Add the same Actor-scoped server to .cursor/mcp.json or Cursor's MCP settings:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"
    }
  }
}

VS Code setup

Use the same remote server object in VS Code's MCP settings:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/website-technical-health-auditor"
    }
  }
}

Example prompts:

• “Audit up to 20 pages of https://apify.com and summarize high-severity findings.”
• “Check these three launch pages without following links and return missing security headers.”
• “Run a bounded technical audit and group non-indexable pages by robots directive.”

Tips for reliable audits

• Begin with 5–10 pages before increasing scope.
• Use maxDepth: 0 for a list of exact landing pages.
• Keep query parameters disabled unless essential.
• Reduce concurrency when a server responds with rate limits.
• Split unrelated sites into separate runs for independent quotas.
• Treat response-time changes as signals, not synthetic performance benchmarks.
• Review critical request failures before interpreting an empty content field.

Limitations

• Public anonymous HTTP(S) pages only.
• Server-rendered HTML only; no browser execution.
• No authenticated pages, forms, sitemaps, or JavaScript navigation.
• No robots.txt policy evaluation or sitemap completeness analysis.
• No DNS, SSL certificate, accessibility, visual, or Core Web Vitals audit.
• No historical baseline, diff, alert delivery, or aggregate health score.
• Link counts describe links on the page; external destinations are not separately requested.
• Website changes, bot controls, and geographic delivery can affect results.

Legality and responsible use

Audit public sites lawfully and only at a rate appropriate for the operator. Do not use the Actor to bypass authentication, probe private networks, or overload a service. The SSRF guard blocks non-public destinations, including redirects.

Website content and headers remain owned by their respective publishers. Apply your own retention, privacy, and compliance requirements to exported datasets.

Troubleshooting

Why did I receive a REQUEST_FAILED record?

The page could not be fetched after bounded retries. Inspect error, reduce concurrency, confirm that the URL is public, and try the exact page in a browser. Persistent authentication or bot challenges are outside this HTTP-only Actor's scope.

Why were fewer pages audited than expected?

The site may expose few server-rendered internal links, links may use excluded file extensions, query variants may be deduplicated, or maxDepth may have been reached. Increase limits deliberately rather than enabling unbounded URL variants.

Why is a page marked non-indexable?

A page is indexable only when it returns successful HTML and has no detected noindex directive. Review statusCode, contentType, and robotsDirectives. The result does not test search-engine canonicals, robots.txt, or actual index membership.

Why are security issues low severity?

Most missing headers are context-dependent hardening opportunities. Insecure HTTP is high severity; missing HSTS is medium; other absent headers default to low. Review the page's purpose and full policies before remediation.

FAQ

Can I audit several websites in one run?

Yes. Supply up to twenty roots. The total maxPages limit is shared across them.

Does it check broken external links?

No. It counts external links but does not request every external destination. Use Sitewide Broken Link Checker for destination-level link records.

Does it execute JavaScript?

No. The HTTP-first implementation is fast and economical for server-rendered pages. Client-only links and metadata are not visible.

Can I schedule it?

Yes. Save the input as an Apify Task and attach a schedule. Use webhooks or integrations to compare and route each new dataset.

Are failed requests charged?

A failed request that produces a structured page audit record is charged as one page. A URL that is never scheduled because a limit was reached is not charged.

Related automation-lab Actors

• Website Health Report for one-row URL health scores, DNS, and SSL evidence.
• Lighthouse Website Audit for browser-based Lighthouse scores.
• Multi-page On-page SEO Audit Crawler for deeper content and heading signals.
• Sitewide Broken Link Checker for destination-level broken-link checks.

Support

For reproducible questions, include a redacted input, run URL, affected page URL, and the relevant issue code. Do not include credentials or private URLs.