Broken Link Checker: crawl a site, find dead links

Give it the address of a site and it crawls the site to the depth and page limit you set, checks every link it finds, and tells you which ones are broken, where each one appears and what anchor text points to it.

Each unique link is checked once, however many pages carry it. You get:

• The HTTP status and a category: ok, redirected, clientError, forbidden (401 and 403), serverError, rateLimited (429), redirectError (a 3xx with no usable Location), redirectLoop, timeout, dnsError, connectionError, tlsError, invalidUrl or robotsDisallowed
• A broken flag, true for 4xx (other than 429), 5xx, redirect faults, unknown hosts, refused connections, certificate failures and malformed URLs. Timeouts and 429 answers get their own category and are not flagged broken, so a slow or busy server does not read as a dead link
• The redirect chain with each hop's URL and status, the final URL and the redirect count
• Response time in milliseconds, a slow flag past your threshold, and the content type
• The first page that links to it with its anchor text, how many times it is linked, and up to 25 source pages, each with its anchor text, element and whether the link is nofollow
• Whether the link is internal or external, and the element it came from (a, area, img, script, iframe or link)

For an image link with no text, the anchor text is the image's alt text, marked [image].

How links are checked

Links are checked with a HEAD request. When HEAD fails or answers 400 or above, the link is checked again with GET, since some servers refuse HEAD. A request that times out on GET is tried once more. Pages the crawl opens are fetched with GET, and that answer is the page's result.

Pages on the start URL's host, with or without www., are internal. Internal HTML pages are opened and their links collected, level by level, up to the crawl depth and page limit. External links are checked but never crawled. Links starting with mailto:, tel:, javascript: and data: are skipped, and the part after # is dropped.

Input

• Sites to check (required). Each start URL is crawled as its own site.
• Maximum crawl depth (default 2). 0 checks only the links on the start page.
• Maximum pages to crawl (default 50) per start URL
• Maximum links per site (default 1,000)
• Check external links (default on)
• Check images, scripts, stylesheets and frames (default off)
• Only output problem links (default off). Leaves out links that answered 2xx, directly or after redirects.
• Follow robots.txt (default on). Internal URLs the site's robots.txt disallows are not requested and are listed with the category robotsDisallowed.
• Slow response threshold (default 3,000 ms), request timeout (default 20 seconds), parallel requests (default 10)
• User agent (optional). The default is Mozilla/5.0 (compatible; BrokenLinkChecker/1.0).

Output

Dataset: one item per unique link with url, status, category, broken, linkType, element, sourcePage, anchorText, sourceCount, sources, finalUrl, redirectCount, redirectChain, responseTimeMs, slow, contentType, method, error and startUrl. The start URL itself is an item with the link type start. Export as JSON, CSV or Excel, or read it over the API.

The key value store holds a SUMMARY record with, for each start URL, the pages crawled, links found, links checked, links left unchecked because a limit was reached, broken links and a count per category.

Pricing

Pay per event: one event per link checked, including the pages the crawl opens. A link that "Only output problem links" leaves out of the dataset is still checked and counted. Links disallowed by robots.txt are not requested and not counted.

Scope and conduct

It requests only public pages, sends no forms and does not log in. Email and phone links are skipped and never stored. Lower the parallel requests for small sites.

Support

Questions and problems: the Issues tab on this page. Replies are written by an AI assistant on behalf of the developer and reviewed by the developer.