Broken Link Checker + On-Page SEO Audit - find 404s, duplicate titles, missing meta tags

Crawl a website (politely, HTTP only, no browser) and get, for every page: broken internal and external links (404/410/5xx/DNS/timeouts) and a basic on-page SEO check, plus a site summary with duplicate titles/descriptions. Fast and cheap, built for agencies, SEO freelancers, site owners and AI agents.

What is checked

• Links: every <a href> on each crawled page (HEAD, GET fallback, redirects followed). 401/403/429 are reported as "blocked", not broken. Links disallowed by robots.txt are skipped, not reported.
• On-page SEO: title (missing/too long/short), meta description, H1 (missing/multiple), canonical, lang, viewport, noindex (meta and X-Robots-Tag), images without alt, thin content, Open Graph tags, mixed content on https pages, HTTP errors and redirects, structured data and hreflang presence.
• Site level: average score, pages with broken links, duplicate titles and meta descriptions, issue counts.

Input

Field	Default	Description
urls	required	Up to 100 start URLs or domains.
maxPagesPerSite	20	Pages audited per site (max 200).
maxDepth	2	Link depth from the start page.
checkExternalLinks	true	Also test links to other sites.
maxLinksPerPage	100	Distinct links tested per page.
delayMs	500	Spacing between requests to one host (max 2 in flight).
concurrency	3	Sites crawled in parallel.
respectRobots	true	Obey robots.txt for crawling and link testing.

{ "urls": ["https://example.com"], "maxPagesPerSite": 50, "maxDepth": 2 }

Output

Items with "type": "page" (one per crawled page) and one "type": "site-summary" per site.

{ "type": "page", "url": "https://www.w3.org/WAI/demos/bad/before/reports/home.html", "status": 200, "score": 58,
  "issues": [{ "severity": "high", "id": "broken-links", "message": "1 broken link(s)" }, { "severity": "low", "id": "og-missing", "message": "..." }],
  "meta": { "title": "...", "titleLength": 41, "wordCount": 320, "imagesWithoutAlt": 0 },
  "brokenLinks": [{ "url": "https://www.w3.org/WAI/WCAG20/quickref/G153", "status": 404, "anchorText": "Technique G153", "internal": true }] }

{ "type": "site-summary", "siteOrigin": "https://www.w3.org", "pagesAudited": 5, "averageScore": 77,
  "brokenLinkOccurrences": 1, "duplicateTitles": [], "issueCounts": { "broken-links": 1, "viewport-missing": 3 } }

The page score starts at 100 and subtracts 20/8/3 per high/medium/low issue (a ranking heuristic).

Pricing (pay per event)

Event	When charged	Price
apify-actor-start	once per run	$0.005
page-audited	each page crawled and analysed (its links included)	$0.004 (= $4 per 1,000 pages)

Why this price: simple link checkers in the Store cost $0.25-$5 per 1,000 results, and SEO crawlers more. Here one billed page includes up to 100 tested links plus a full on-page SEO check and a site summary. Example: a 200-page site = $0.80; 50 client sites x 50 pages = $10.

Politeness and safety

• Identifies as LinkSeoAudit; obeys robots.txt; max 2 requests in flight per host with delayMs spacing; pages of one site are fetched sequentially; hard caps on pages and links.
• Only http/https; localhost, internal hostnames and private/reserved IPs are refused (connect time and every redirect hop).
• Does not execute JavaScript: client-rendered content is not seen (use a rendering crawler for SPAs). Public pages only, no logins.

FAQ

Does it crawl JavaScript-rendered sites?
No. It is HTTP-only for speed and cost. Client-rendered content is not seen.

Are 403 and 429 responses counted as broken?
No. 401/403/429 are reported as blocked, not broken, to avoid false alarms.

Can it check external links?
Yes, checkExternalLinks (default true) tests outbound links too.

How many pages per site?
Up to 200 pages per site and 100 start URLs per run.

Will it overload my server?
No. It obeys robots.txt, keeps at most 2 requests in flight per host and spaces them with delayMs.

Part of the Site Audit Suite

Run the same list of domains through every audit:

• Website Tech Stack Detector
• Website Accessibility Audit (WCAG 2.2 + EAA)
• Security Headers + TLS Audit
• llms.txt Generator and Sitemap + robots.txt Analyzer (coming soon)

Tip: pipe one domain list into several actors with an Apify Task or the API and join the results by url.

Develop

npm install && npm test
mkdir -p storage/key_value_stores/default
echo '{"urls":["https://example.com"],"maxPagesPerSite":5}' > storage/key_value_stores/default/INPUT.json
npm start          # results in storage/datasets/default/
docker build -t broken-link-seo-audit .

Deploy (needs Apify account): npx apify-cli login && npx apify-cli push.

Verified (2026-10-05, local)

w3.org BAD demo (5 pages, ~190 links, found a real 404), example.com, httpbin.org. Docker: docker run --user 996 -v <dir>:/usr/src/app/storage broken-link-seo-audit.