Broken Link Checker: 404 Finder, Redirect Chain Audit & SEO Audit CSV

Broken link checker and 404 finder for SEO folks and site owners. Crawls one site, runs a redirect chain audit on every internal and external link, and hands you a ready-made SEO audit CSV (OUTPUT.csv) plus a SUMMARY record. No post-processing.

What it does

• Crawls one site (same domain only) with Crawlee CheerioCrawler, up to maxPages and maxDepth.
• Checks every unique internal and external link once (HEAD, GET fallback on 403/405/501 or network error).
• Follows redirects manually (max 10 hops) and records every hop URL + status.
• Writes one dataset row per (source page, link) pair, a SUMMARY JSON and OUTPUT.csv to the key-value store.

Why this one

• Redirect chains: the full chain per link (301 -> 302 -> 200), not just the final status.
• Summary: counts of 2xx, 404, 4xx, 5xx, redirects, chains longer than 1 hop, timeouts.
• CSV: open OUTPUT.csv in Excel/Sheets straight away. Quotes, commas and newlines are escaped.

Input

field	default	note
startUrl (required)	https://crawlee.dev
maxPages	500	1-10000
maxDepth	3	0 = start page only
checkExternal	true	false: external links listed with error skipped, never requested
respectRobots	true	disallowed pages are not crawled, links to them are still checked
maxConcurrency	10	max 20, also caps link-check concurrency
requestTimeoutSecs	15	per link check / page

Output

Dataset row:

{"source_page":"https://quotes.toscrape.com","link":"https://quotes.toscrape.com/author/Albert-Einstein","status":200,"redirect_chain":[{"url":"https://quotes.toscrape.com/author/Albert-Einstein","status":308}],"final_url":"http://quotes.toscrape.com/author/Albert-Einstein/","type":"internal","anchor":"(about)","error":null}

error is null, timeout, too_many_redirects, skipped or the network error message (status is then null).

SUMMARY (key-value store): see the example run below.

OUTPUT.csv:

source_page,link,status,redirect_chain,final_url,type,anchor,error
https://quotes.toscrape.com,https://quotes.toscrape.com/login,200,,https://quotes.toscrape.com/login,internal,Login,
https://quotes.toscrape.com,https://quotes.toscrape.com/author/Albert-Einstein,200,https://quotes.toscrape.com/author/Albert-Einstein (308),http://quotes.toscrape.com/author/Albert-Einstein/,internal,(about),

Chains are written url1 (301) -> url2 (302).

Pricing

Pay per event: $0.005 per run start + $0.003 per page crawled. Link checks are free. 500 pages is about $1.50 + start fee.

Example run

Smoke run, {"startUrl":"https://quotes.toscrape.com","maxPages":20,"maxDepth":2,"requestTimeoutSecs":10} against https://quotes.toscrape.com (2026-10-15):

{"startUrl":"https://quotes.toscrape.com","pages_crawled":20,"links_total":458,"unique_links":121,"ok_2xx":121,"broken_404":0,"errors_4xx":0,"errors_5xx":0,"redirects":31,"redirect_chains_gt1":0,"timeouts":0,"other_errors":0,"finished_at":"2026-10-15T03:21:57.150Z"}

FAQ

• robots.txt? Respected by default for crawling. Disallowed pages are not fetched as pages, but links pointing to them are still checked (HEAD).
• External links? Checked by default. Set checkExternal false to skip them (they stay in the output with skipped).
• Limits? 10000 pages, 20 concurrent requests, 300 page requests/min, 10 redirect hops per link. Internal links beyond maxPages/maxDepth get a status but are not crawled.
• JS-rendered links? Not seen: plain HTML only.
• Summary counts are per unique link; links_total counts dataset rows.

Run and test locally

npm ci
echo '{"startUrl":"https://example.com"}' > storage/key_value_stores/default/INPUT.json   # mkdir -p first
npm start
npm test   # local fixture servers, no internet, ~30-60 s