Broken Link Checker - Bulk URL Status Code Scraper

Checks every link on a page and reports which state each one is actually in. A link that answers 403 to an automated checker is reported as blocked to us, not as broken, because a person clicking it in a browser will very likely reach the page.

Why that distinction is the whole product

Measured on 26 August 2026 against fifteen well-known sites, using a bot user-agent:

What happened	How many	Examples
Answered 403 to the checker — not broken, just refusing automated requests	6 of 15	stackoverflow.com, yelp.com, tripadvisor.com, etsy.com, quora.com, glassdoor.com
Refused HEAD but answered GET	2 of 15	amazon.com (HEAD 405 → GET 202), medium.com (HEAD 403 → GET 200)
Answered with a 2xx that is not 200	4 of 15	amazon.com and imdb.com and booking.com (202), github.com (206)
Genuinely gone	1 of 15	a 404 URL used as a control

A checker that sends HEAD and treats "not 200" as broken would have reported twelve of those fifteen links as broken. One of them actually was.

This checker sends HEAD, retries with a short ranged GET when the server rejects HEAD, accepts every 2xx, and keeps 403 in a category of its own.

The states

state	Meaning
ok	Any 2xx, after redirects. 202 and 206 count.
not-found	404 or 410. The link is genuinely gone.
blocked-to-us	401, 403, 429, 451. The server refused this checker. This is not a broken link.
server-error	5xx. Often temporary; retried, and the state reflects the last attempt.
client-error	Another 4xx, a 3xx with no Location, or a redirect loop.
unreachable	No HTTP answer at all — DNS has no record, connection refused, TLS failed, or the request timed out. The reason says which.

Every row also carries naiveCheckWouldSayBroken: true when a HEAD-and-200-only check would have called this link broken, while this checker found it alive or merely blocked. That column is the difference between the two tools, stated per link.

What it does not claim

Attached to every row as notClaimed:

• A blocked-to-us link is not a broken link. The server refused this checker specifically.
• Page content is not inspected. A URL that returns 200 with the words "page not found" written in the body is reported as ok.
• JavaScript is not executed, so links created by scripts after load are not seen.
• A server-error can be temporary; it is retried, and the state reflects the last attempt.

A real run, with the default input

The default input is https://www.w3.org/History.html — a page about the early history of the web, whose outbound links have had thirty years to rot. Measured on Apify: 72 links in 27 seconds — 58 ok, 10 not-found, 4 unreachable, peak memory 73 MB.

The same run from a home connection took 50 seconds and reported one more link as unreachable. That is not a discrepancy to hide: reachability is measured from wherever the check runs, and a host that times out from one network can answer from another. The row says unreachable with the transport reason rather than not-found, precisely so the difference is visible.

Among them: http://europa.eu.int/comm/index.htm (the domain no longer resolves), http://www.cern.ch/WWW94/ (same), and several W3C pages that now return 404. Two links would have been misreported by a naive check.

Input

Field	Type	Default	What it does
startUrls	array	https://www.w3.org/History.html	Pages whose links should be checked. Each distinct link is checked once, no matter how many pages point at it.
urls	array	empty	Check these URLs directly, without reading a page first. Can be combined with startUrls.
checkInternal	boolean	true	Include links back to the same site. The www prefix is ignored when deciding.
checkExternal	boolean	true	Include links to other sites.
maxLinks	number	100	How many links to check. Taken in page order; the run reports how many were left unchecked rather than hiding the truncation.
concurrency	number	5	Links checked in parallel across all servers.
perHostConcurrency	number	2	Never more than this many at once to one server.
respectRobots	boolean	true	Applies to fetching the pages links are read from.
timeoutSecs	number	20	Timeout for one link. Range 3–120.
maxRetries	number	3	Retries for 429 and 5xx. Range 1–10.

Two deliberate choices about speed

Requests to one server are limited. Sending everything at once makes a server answer 429, and 429 is blocked-to-us — the checker would be manufacturing the very result it is measuring. Default is two at a time per server, five in total.

Timeouts are retried at most twice, not maxRetries times. A timeout costs the entire timeout period, and a server that did not answer in twenty seconds is rarely fixed by asking again six hundred milliseconds later. On the default page one dead host was taking 21 seconds per attempt; three attempts made the whole run 74 seconds. Capping timeout attempts at two brought it to 50 seconds with identical results. 429 and 5xx still get the full maxRetries, because those genuinely do clear.

What each row contains

url, finalUrl (after redirects), state, httpStatus, reason, naiveCheckWouldSayBroken, method (whether HEAD was enough or GET was needed), headRejectedStatus, redirectCount, redirectChain (every hop with its status), elapsedMs, transportError, isInternal, linkedFrom (which pages link to it), anchorTexts, notClaimed, checkedAt.

Pages that could not be read for links are returned too, with state: null and a reason — robots.txt disallowed them, they were not HTML, or the fetch failed. Those rows are not charged.

Typical uses

• Link rot on old content. Long-lived pages accumulate dead outbound links; this separates the genuinely dead from the merely unfriendly-to-bots.
• Before a site migration. Record the redirect chain of every internal link, then compare afterwards.
• Cleaning a directory or resource list. not-found and unreachable are actionable; blocked-to-us usually is not, and should not be deleted by mistake.

FAQ

Why is 403 not treated as broken? Because six of the fifteen sites measured answer 403 to any automated checker while working perfectly in a browser. Deleting those links would remove working references.

Why send HEAD first? It avoids downloading page bodies. When a server rejects HEAD — amazon.com answers 405, medium.com answers 403 — the check is repeated with a GET limited to the first 2 KB.

Does it follow redirects? Yes, and it records them. Redirects are followed manually rather than by the HTTP client, because an automatically-followed redirect is invisible afterwards and the chain is often the thing you need to see.

Is a 200 guaranteed to mean the page is fine? No, and this does not claim so. Some sites answer 200 with a "not found" message in the body. Checking that would mean reading and interpreting page content, which this deliberately does not do — notClaimed says so on every row.

Other tools by NeverEmpty

Every NeverEmpty Actor follows the same rule: it never returns an empty result to mean two different things, and it only charges for rows that actually carry an answer.

• seo-site-audit - technical SEO audit: meta, canonical, robots, headings
• domain-lookup - WHOIS, DNS, MX, SPF, DMARC and SSL expiry for a list of domains
• website-contacts - emails, phones and social profiles from any website, no invented addresses

Support

Found a link classified wrongly? Open an issue on the Issues tab with the URL and the row you got. Reports about incorrect data are fixed first.