Site Health Scanner 🔍

Find broken links before your users do. The scanner crawls your site, checks every link, and captures screenshots of broken internal pages as client-ready proof.

⚠️ Bot Protection Notice: Some websites use aggressive bot detection that blocks automated requests. You can enable the Residential Proxy setting to bypass basic protection, but heavily protected sites (Cloudflare, advanced WAFs) may still block scans. Links on those sites are marked as BLOCKED rather than broken — verify them manually in a browser.

What does it do?

Site Health Scanner uses a real (Playwright) browser to crawl your website and check every link it finds — internal pages, external URLs, images, scripts, and stylesheets. When it finds a broken internal page, it can take a screenshot so you have visual proof for clients or stakeholders.

Perfect for:

• SEO professionals auditing client sites
• Web agencies delivering health reports
• Site owners maintaining link integrity
• QA teams catching issues before launch

Features

Feature	Description
🔗 Broken Link Detection	Flags 4xx and 5xx responses, timeouts, and connection errors across your site
🚫 Smart Bot-Block Detection	Heuristically separates truly broken links from bot-blocked sites (external, and widespread internal 403 patterns)
📸 Screenshot Proof	Captures screenshots of broken internal pages (see limitations below)
↪️ Redirect Chain Tracking	Follows and records redirect chains (up to 10 hops), flags long chains
⚠️ Mixed Content Warnings	Flags HTTP resources linked from HTTPS start URLs
⏱️ Slow Response Flagging	Records response time per link and warns when it exceeds 3 seconds
🌐 External Link Checking	Optionally verifies the status code of outbound links (not crawled)
🔁 Regression Monitoring	Optional mode that compares each run to the prior run and can POST a webhook alert when new broken links appear

Pricing

This actor runs on the Apify platform, so you pay for the platform compute it consumes plus any optional residential-proxy traffic you enable. Cost scales with:

• Number of pages crawled (maxPages) and crawl depth (maxDepth)
• Whether external link checking is enabled (adds many more requests)
• Whether screenshots are enabled (adds browser navigation + storage)
• Whether the residential proxy is enabled (you pay for proxy traffic)

For current rates, see the pricing shown on this actor's Apify Store page. Use maxPages to cap costs on large sites.

Input

Field	Type	Description	Default
startUrls	array	URLs to start crawling (homepage, sitemap, or specific pages)	Required
maxDepth	integer	How many clicks deep to crawl (0–10). 0 = only start URLs	3
maxPages	integer	Maximum pages to crawl (1–10,000)	100
checkExternalLinks	boolean	Also check the status of links to other domains	true
screenshotBrokenPages	boolean	Screenshot broken internal pages	true
followRedirects	boolean	Follow and record redirect chains	true
timeout	integer	Per-request timeout in seconds (5–120)	30
includeWarnings	boolean	Populate the warning field (mixed content, slow, long redirects, bot-protection notices)	true
userAgent	string	Custom user-agent string. Leave empty to rotate realistic browser UAs	""
useProxy	boolean	Route requests through Apify residential proxy (adds proxy traffic cost)	false
requestDelay	integer	Delay between requests in ms (0–10000) to avoid rate limiting. External links use 3× this value	0
monitoringMode	boolean	Persist broken-link state per start URL and report new vs. fixed links against the prior run	false
alertWebhookUrl	string	Optional URL to POST a JSON alert to when new broken links are detected (Slack, Discord, Zapier, custom)	""

About Bot Protection (403 Errors)

Many websites block automated requests and return 403 Forbidden. That does not necessarily mean the link is broken — the site may just be blocking bots.

Without proxy (default): External links may show a 403 BLOCKED status. These are recorded with isBroken: false and confidence: low because the link likely works for real users. If more than half of a sample of internal links return 403 (5+ samples), the scanner treats it as site-wide bot protection and also marks those as BLOCKED / isBroken: false.

With residential proxy: Enable useProxy to route requests through residential IPs, which are less likely to be blocked. This adds residential-proxy traffic cost.

About Rate Limiting (429 Errors)

If a site rate-limits requests (429 responses), increase requestDelay:

• 0 (default): No delay — fastest, but may trigger rate limits
• 500–1000: Light throttling — good for most sites
• 1500–2000: Heavy throttling — for aggressive sites
• External links: Automatically use 3× the configured delay

Regression Monitoring

Set monitoringMode: true to store the set of broken URLs per start URL in the actor's key-value store. On the next run the scanner reports how many links are newly broken and how many are now fixed versus the prior run. If alertWebhookUrl is set, a JSON payload is POSTed only when new broken links appear — ideal for scheduled runs.

Example Input

{
  "startUrls": [{ "url": "https://example.com" }],
  "maxDepth": 3,
  "maxPages": 500,
  "checkExternalLinks": true,
  "screenshotBrokenPages": true
}

Output

Each checked link produces one dataset record:

Field	Description
url	The URL that was checked
statusCode	HTTP status code (200, 404, 500, etc.). 0 = request failed before a response
status	Category: OK, REDIRECT, BROKEN, BLOCKED, TIMEOUT, ERROR, SERVER_ERROR, CLIENT_ERROR, CRAWL_FAILED, UNKNOWN
confidence	Confidence in the status: high, medium, low
isBroken	Definitive broken flag (true only for links judged actually broken)
type	Link type: internal, external
foundOnPage	The page the link was found on
anchorText	The link's anchor text, image alt text, or resource marker
responseTime	Response time in milliseconds (null if the page could not be crawled)
redirectChain	Full redirect path joined by → if redirected, otherwise null
screenshotUrl	Key-value store URL of the broken-page screenshot (if captured)
error	Error message if the request failed
warning	Warnings (mixed content, slow response, long redirect chain, bot-protection notice)
checkedAt	ISO-8601 timestamp of when the URL was checked

Status Categories

Status	Meaning	Is Broken?
OK	Link works (200–299)	No
REDIRECT	Link redirects (300–399)	No
BROKEN	Link is dead (404, 410)	Yes
BLOCKED	Access denied (401/403) — often bot protection	No (external / widespread internal) · Yes (isolated internal)
TIMEOUT	Request timed out	Yes
ERROR	Connection / DNS failure	Yes
SERVER_ERROR	Server error (500–599)	Yes
CLIENT_ERROR	Other 4xx errors	Yes
CRAWL_FAILED	A start/queued page could not be crawled after retries	Yes
UNKNOWN	Status could not be classified	No

Confidence Levels

Confidence	Meaning
high	Status is definitive (404, 200, server error, etc.)
medium	Status may vary (a timeout can be temporary)
low	Status uncertain — external 403s often block bots but work in a browser

Example Output

{
  "url": "https://example.com/old-page",
  "statusCode": 404,
  "status": "BROKEN",
  "confidence": "high",
  "isBroken": true,
  "type": "internal",
  "foundOnPage": "https://example.com/blog",
  "anchorText": "Read our old article",
  "responseTime": 245,
  "redirectChain": null,
  "screenshotUrl": "https://api.apify.com/v2/key-value-stores/abc123/records/screenshot-xyz",
  "error": null,
  "warning": null,
  "checkedAt": "2025-12-13T10:30:00.000Z"
}

External Blocked Link Example

{
  "url": "https://www.viator.com/tours",
  "statusCode": 403,
  "status": "BLOCKED",
  "confidence": "low",
  "isBroken": false,
  "type": "external",
  "foundOnPage": "https://yoursite.com/travel",
  "warning": "External site may be blocking automated requests. Verify manually.",
  "checkedAt": "2025-12-13T10:30:00.000Z"
}

Summary Statistics

After each run, a summary object is saved to the Key-Value Store under the key summary:

{
  "totalLinksChecked": 847,
  "brokenLinks": 12,
  "blockedLinks": 5,
  "redirects": 45,
  "warnings": 8,
  "pagesProcessed": 100,
  "scanCompletedAt": "2025-12-13T10:45:00.000Z"
}

The end-of-run log also prints detailed lists of warnings, blocked links, and broken links for quick review.

How to Use the Results

1. Export to CSV — download the dataset for spreadsheets or client reports.
2. Filter by status — use dataset filters to show only broken links or only redirects.
3. Use the isBroken field — filter isBroken: true to get only definitely broken links and ignore bot-blocked externals.
4. Use screenshots — broken internal-page screenshots are stored in the Key-Value Store; their URLs appear in screenshotUrl.
5. Automate with schedules — schedule runs, optionally with monitoringMode, to track site health over time.

Integrate via API

const { ApifyClient } = require('apify-client');
const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('YOUR_USERNAME/site-health-scanner').call({
    startUrls: [{ url: 'https://example.com' }],
    maxPages: 500
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();

// Only definitely broken links (ignores bot-blocked externals)
const brokenLinks = items.filter(item => item.isBroken === true);
console.log(`Found ${brokenLinks.length} broken links`);

Limitations

• Crawl scope: Crawling is bounded by maxPages (hard cap 10,000) and maxDepth (max 10). Larger sites are truncated. Only pages on the start URLs' domains are crawled; external links are status-checked but never crawled.
• Link status checks are HTTP HEAD/GET, not full page loads. Crawled pages are rendered in a real browser (so JavaScript-injected links are found), but each discovered link's status is verified with a HEAD request (GET fallback) — no JavaScript execution or cookies on the link target itself. Very complex SPAs may still hide some links from extraction.
• Screenshots are narrow by design. A screenshot is only taken for an internal page link that returns a 4xx/5xx error, excluding 401/403 (access-blocked) and excluding non-page resources (images, scripts, stylesheets). External broken links are not screenshotted.
• Bot-block detection is heuristic. External 401/403s and widespread internal 403 patterns are treated as bot protection (isBroken: false), which can occasionally misclassify a genuinely broken or genuinely blocked page. Verify BLOCKED links manually.
• Anti-bot / WAF blocking. Sites behind Cloudflare or advanced WAFs may block the scanner entirely even with the residential proxy enabled.
• No robots.txt enforcement. The crawler does not read or obey robots.txt or crawl-delay directives — only scan sites you own or have permission to scan, and use requestDelay to be polite.
• No performance/SEO scoring. This actor does not run Lighthouse, does not score performance, and does not analyze SSL certificates, Core Web Vitals, or on-page SEO. It records raw response time and flags responses slower than 3 seconds only.
• Rate limits. Fast scans of aggressive sites can trigger 429s; raise requestDelay if needed.
• Redirect depth. Redirect chains are followed up to 10 hops; longer chains are truncated.

Use Cases

• SEO audit — run before and after migrations to catch links that could hurt rankings.
• Client reporting — use screenshots to show clients exactly what's broken.
• Continuous monitoring — schedule runs with monitoringMode to catch newly broken links.
• Pre-launch QA — verify links before going live.

Troubleshooting

Problem	Solution
Timeout errors	Increase the timeout setting
Many 403 "blocked" on external sites	Expected — large sites block bots. Verify manually if needed
Missing pages	Increase maxDepth or maxPages
Slow scans	Disable checkExternalLinks or screenshotBrokenPages
Some links not found	Complex JavaScript navigation may hide links from extraction
Everything blocked	Try enabling useProxy; the site may have a strong WAF

Changelog

v1.2 (2026)

• NEW: Regression monitoring mode (monitoringMode) with per-start-URL state and optional webhook alerts (alertWebhookUrl) when new broken links appear.

v1.1.0

• IMPROVED: Better status classification — BLOCKED (401/403) is separated from BROKEN; external 403s marked isBroken: false, isolated internal 403s marked isBroken: true.
• NEW: confidence field (high/medium/low) and isBroken field.
• NEW: Verbose end-of-run log output for warnings, blocked links, and broken links.
• NEW: blockedLinks count in the summary.

v1.0.0

• Initial release: broken-link detection with screenshot proof, redirect chain tracking, mixed content warnings, external link checking, response-time flagging.

---

Find broken links before your users do.