Site Health Auditor — Broken Links, Meta, Vitals, Weekly Diffs

One crawl. One report. And a diff against last week's run.

Most site auditors hand you the same 400-line list of problems every time you run them, and you have no idea which ones are new. This Actor crawls a site you own, finds the problems, and then tells you the only thing that actually changes your Monday:

12 new issues, 5 resolved since 2026-08-08.

Put it on a weekly Schedule, point it at Slack, and it becomes a standing early-warning system for your site.

This Actor is for auditing sites you own or operate. It respects robots.txt, caps concurrency at 10, and checks third-party domains one request at a time. It is designed to be polite; please keep it pointed at your own properties.

---

What it checks

Area	What you get
Broken links	Internal links verified by the crawl itself; outbound links verified with HEAD (falling back to GET). Records source page, target, and status.
Redirect chains	Every redirect captured from the response history. One hop is a warning, two or more is an error.
Metadata & Open Graph	Missing <title>, duplicate <title> across pages, missing meta description, missing canonical, missing og:image.
Accessibility & weight	Images with no alt attribute, pages with multiple <h1>, HTML payloads over 2 MB.
Core Web Vitals	Lighthouse performance / SEO / accessibility scores plus LCP, CLS, INP and TBT on a smart sample of pages.
Diff vs. last run	New issues, resolved issues, and score movements per page — the reason to run this on a schedule.

Who it's for

• Solo founders and small teams who own a marketing site or docs site and have no SEO tooling budget.
• Agencies managing a handful of client sites who want a weekly "what broke" digest per domain.
• Engineering teams who want a post-deploy check that content edits did not quietly break links or tank LCP.

---

Three concrete use cases

1. Weekly Monday-morning digest to Slack. Schedule the Actor for 07:00 every Monday with slackWebhookUrl set. You get one message per site: "3 new issues, 8 resolved since last Monday", with the new ones listed. If nothing broke, the message says so and you move on.

2. Post-migration link sweep. You moved a CMS, changed a URL structure, or merged two sites. Run once before the switch to capture a baseline, then again after. The diff is exactly your regression list — every internal link that used to work and now 404s, and every new redirect chain the migration introduced.

3. Content-team guardrail. Writers publish weekly. Point the Actor at /blog and schedule it nightly. Missing meta descriptions, duplicate titles, and images without alt text surface within a day of publication instead of at the next quarterly audit.

---

Input

Field	Type	Default	Description
startUrl	string	required	The site to audit. The crawl stays on this origin. If the URL redirects (e.g. apex → www), the crawl follows it and audits the origin it lands on.
maxPages	integer	300	Hard cap on pages fetched. Max 2000.
maxOutboundChecks	integer	500	Distinct off-site links to verify. Set to 0 to skip outbound checking. Max 5000.
runLighthouse	boolean	true	Measure Core Web Vitals. Turn off for a much faster, cheaper link-and-metadata audit.
lighthousePages	integer	5	Pages to measure: homepage + most-linked pages, one per URL template. Max 20.
slackWebhookUrl	string (secret)	–	Incoming-webhook URL. Receives the run summary and what changed.
reportWebhookUrl	string (secret)	–	Receives a JSON POST with the full summary and diff.
proxyConfiguration	proxy	–	Route traffic through Apify Proxy. Useful when a CDN rate-limits datacenter IPs.

Minimal input

{
  "startUrl": "https://your-site.com"
}

Fast link-only audit (no browser)

{
  "startUrl": "https://your-site.com",
  "maxPages": 300,
  "maxOutboundChecks": 500,
  "runLighthouse": false
}

---

Output

Every run produces:

1. Dataset — one record per issue, plus a final summary record.
2. REPORT.md in the key-value store — the human-readable report, leading with the diff.
3. REPORT.json in the key-value store — the same data, machine-readable.
4. PAGES.json in the key-value store — the raw per-page crawl data, if you want to do your own analysis.

All four are linked from the Output tab of every run, and the dataset ships three prebuilt views:

View	Shows
Issues overview	Severity, type, the page to fix, the target, and whether the issue is new since the last run.
Technical detail	The same issues with the details payload flattened into columns — HTTP statuses, redirect hop counts, and fingerprints.
Run summary	Run-level totals: pages crawled, issue counts, new vs. resolved, and a direct link to REPORT.md.

Every field is documented with a title, description and example in the dataset schema, so the column meanings show up in the Console rather than needing this README.

Sample output

All samples below are copied verbatim from a real local run against https://quotes.toscrape.com (maxPages: 30).

Dataset — an issue record:

{
  "recordType": "issue",
  "type": "redirect_chain",
  "severity": "warning",
  "sourceUrl": "https://quotes.toscrape.com/author/Albert-Einstein",
  "targetUrl": "http://quotes.toscrape.com/author/Albert-Einstein/",
  "message": "Redirect: https://quotes.toscrape.com/author/Albert-Einstein → http://quotes.toscrape.com/author/Albert-Einstein/",
  "details": {
    "hops": 1,
    "chain": [
      "https://quotes.toscrape.com/author/Albert-Einstein",
      "http://quotes.toscrape.com/author/Albert-Einstein/"
    ],
    "finalStatus": 200
  },
  "fingerprint": "c2630c2595cac16a",
  "isNewSinceLastRun": true,
  "hostname": "quotes.toscrape.com",
  "runAt": "2026-08-15T16:31:14.498Z"
}

Dataset — the final summary record:

{
  "recordType": "summary",
  "startUrl": "https://quotes.toscrape.com/",
  "origin": "https://quotes.toscrape.com",
  "hostname": "quotes.toscrape.com",
  "runAt": "2026-08-15T16:31:14.498Z",
  "pagesCrawled": 30,
  "pagesFailed": 0,
  "outboundLinksChecked": 4,
  "totalIssues": 126,
  "errorCount": 0,
  "warningCount": 126,
  "issueCountsByType": {
    "redirect_chain": 6,
    "duplicate_title": 30,
    "missing_meta_description": 30,
    "missing_canonical": 30,
    "missing_og_image": 30
  },
  "brokenLinkCount": 0,
  "lighthousePagesAudited": 0,
  "newIssueCount": 1,
  "resolvedIssueCount": 0,
  "isBaseline": false,
  "snapshotSaved": true,
  "robotsBlockedCount": 0,
  "newIssues": ["c2630c2595cac16a"],
  "resolvedIssues": []
}

REPORT.md — the diff section on a repeat run:

## Since last run

Previous run: **2026-08-15 16:30:38 UTC**

### New issues (1)

| Severity | Type | Page | Detail |
| --- | --- | --- | --- |
| 🟠 warning | Redirects | https://quotes.toscrape.com/author/Albert-Einstein | 1 hop → http://quotes.toscrape.com/author/Albert-Einstein/ |
### Resolved since last run (0)

REPORT.md — the Core Web Vitals table (from a run with runLighthouse: true):

| Page | Perf | SEO | A11y | LCP | CLS | INP | TBT |
| --- | --- | --- | --- | --- | --- | --- | --- |
| https://quotes.toscrape.com/ | 🟢 90 | 🟢 90 | 🟠 83 | 2.88 s | 0.000 | 16 ms | 0 ms |
| https://quotes.toscrape.com/login | 🟢 92 | 🟢 90 | 🟠 73 | 3.04 s | 0.000 | 16 ms | 0 ms |

---

The weekly schedule pattern

This is what the Actor is built for.

1. Run it once against your site. That run is your baseline — no diff yet.
2. In the Apify Console, open the run and click Schedule (or Schedules → Create new).
3. Set a weekly cron, e.g. 0 7 * * 1 for Mondays at 07:00.
4. Add your slackWebhookUrl to the schedule's input.

From the second run on, every report opens with what changed. History is stored per hostname in a named key-value store called site-health-history, so you can schedule the same Actor against several sites and each keeps its own timeline.

To reset a site's history and start a fresh baseline, delete its key (named after the hostname) from that store.

---

Pricing — pay per event

Mirror this table exactly in Actor → Monetization → Pay per event in the Apify Console.

Event name	Title	Description	Price
audit-page	Page audited	One page of your site crawled and analysed — status code, redirect chain, title, meta description, canonical, Open Graph tags, headings, images missing alt text, page weight, and every internal and outbound link found on it. Charged once per page.	$0.004
lighthouse-page	Core Web Vitals page	A full Lighthouse audit of one page in a real Chromium browser, returning performance, SEO and accessibility scores plus LCP, CLS, INP and TBT. Charged once per page successfully measured.	$0.05
report	Health report	The combined report for the run — every issue found, grouped by severity, with the diff against your previous run and the Core Web Vitals table. Delivered as REPORT.md and REPORT.json. Charged once per completed run.	$0.05

When each event fires

Event	Fires
audit-page	Once per crawled page, after the crawl finishes and page data is durably stored. Pages that returned an error status are included — fetching them is the work, and their status is what the broken-link report is built from.
lighthouse-page	After each individual page audit succeeds.
report	After REPORT.md is written to the key-value store.

Nothing is charged before the work it pays for is durably persisted. A run that fails mid-crawl bills nothing, and a Lighthouse page that errors is not charged.

Every charge bills exactly one event — the count parameter is never used. The platform can partially fulfil a bulk charge, which would silently put billing and delivered work out of step; charging one unit at a time means each call either bills one event or bills nothing, and the run stops charging the moment a limit is reached.

Typical cost: a 300-page site with 5 Lighthouse pages is 300 × $0.004 + 5 × $0.05 + $0.05 = $1.50 per run, or about $6.50/month on a weekly schedule.

Because pages are billed individually, you pay for exactly what was crawled — a 40-page site costs 40 × $0.004 = $0.16, not a rounded-up block.

---

Running it yourself

$npm install

$npm test

$npm start

npm start compiles and runs against storage/key_value_stores/default/INPUT.json, which ships with a working sample. Results land in storage/datasets/default/ and storage/key_value_stores/default/.

With the Apify CLI:

$npx apify run

Everything at once — lint, type-check, tests, build:

$npm run verify

---

FAQ

Does it respect robots.txt? Yes, always, and it cannot be turned off. URLs disallowed for * are skipped and counted in the report under "URLs skipped by robots.txt".

Why does it report a redirect on a link that works fine? Because a redirect costs a round trip on every visit and dilutes link equity. One hop is reported as a warning — informational, fix it when convenient. Two or more hops is an error, because chains are usually accidental.

Why do internal links to http:// show up as outbound? "Same origin" means same scheme, host, and port. A link from https:// to http:// on the same host is genuinely cross-origin, and it is usually a bug worth seeing.

A link works in my browser but is reported as broken. Check the details.method field on the issue. Some servers reject or ignore HEAD; the Actor already retries those with GET (including when HEAD hangs), and only reports the link broken if GET fails too, three attempts apart with exponential backoff. If GET genuinely fails from Apify's network but works from yours, the target is probably geo-blocking or rate-limiting datacenter IPs — try enabling Apify Proxy.

Why did Lighthouse pick those pages? The homepage first, then the most-linked internal pages, taking only one page per URL template — /blog/a and /blog/b count as one template, so a 200-post blog contributes one representative post rather than crowding out the rest of the site.

Why is INP always small or identical? A cold page load records no interactions, so lab INP is usually unavailable. The Actor falls back to Lighthouse's Max Potential FID, which is its stand-in for responsiveness. For real INP, use field data from CrUX.

Can I audit a site I don't own? Please don't. The tool is deliberately polite rather than stealthy, and crawling someone else's site at volume is their call to make, not yours.

Can it crawl a staging site behind a private IP or on localhost? No. URLs that resolve to loopback, private, or link-local addresses are refused — that guard is what stops a page full of http://169.254.169.254/... links from turning the crawler into an internal network probe. Audit staging over a public hostname instead.

What happens on the very first run? It is the baseline. The report says so and skips the diff section. Every run after that compares against it.

---

How it works

resolve startUrl ─▶ crawl (CheerioCrawler, robots.txt, maxConcurrency 10)
                        │
                        ├─▶ outbound link check (HEAD → GET, 1 req/domain at a time)
                        │
                        ├─▶ issue engine (pure functions, fully unit-tested)
                        │
                        ├─▶ Lighthouse (bundled Chromium over CDP, sequential)
                        │
                        └─▶ diff vs. site-health-history ─▶ dataset + REPORT.md/json ─▶ Slack/webhook

Issues are identified across runs by a stable fingerprint, sha256(type|sourceUrl|targetUrl). It deliberately excludes counts and messages, so an img_missing_alt issue whose count goes from 3 to 4 stays the same issue rather than showing up as one resolved plus one new.

Plain HTTP is used for everything except Lighthouse — no headless browser is launched when runLighthouse is false.

---

Changelog

0.1.0 — 2026-08-15

• Initial release.
• Same-origin crawl with robots.txt support and a configurable page budget.
• Outbound link checking with HEAD→GET fallback, per-domain serialisation, and Retry-After-aware backoff.
• Ten issue types across links, redirects, metadata, Open Graph, accessibility, and page weight.
• Lighthouse Core Web Vitals on a template-deduplicated page sample.
• Run-to-run diff with new/resolved issues and per-page score deltas.
• REPORT.md and REPORT.json artifacts, Slack and generic webhook delivery.

---

License

Apache-2.0. See SECURITY.md for the security model and RUNBOOK.md for operational procedures.