Broken Link Checker

What it does

This Actor reads the pages of a website, collects every link on them, and checks each link once. You get one row per link, with its final HTTP status, whether it is broken, its full redirect chain, the page it was found on and its link text. A link counts as broken when it ends in a 4xx or 5xx answer, a redirect loop, too many redirects or a redirect that points nowhere usable.

Give it start URLs, a sitemap, or both. From start URLs it follows links on the same site to more pages, up to Maximum pages. From a sitemap it reads the pages the sitemap lists.

It reads each site's robots.txt before anything else and obeys it. It never logs in, never sends a cookie and never tries to get past a CAPTCHA.

How to use it

1. Open the Actor in Apify Console.
2. Put the pages to start from in Start URLs, or a sitemap address in Sitemap URL.
3. Set Maximum pages and Maximum links, and choose whether to Check external links.
4. Click Start.
5. When the run ends, open the Output tab. The Links view shows every checked link; filter on Broken to see only the broken ones. Export as CSV, JSON or Excel.

Input

Field	What it does	Default	Limits
Start URLs (startUrls)	Pages to start from. Links on the same site are followed to more pages. If a start URL redirects to another host name, such as example.com to www.example.com, that host counts as the same site.	None. Prefilled with one example site	Up to 100. http and https only
Sitemap URL (sitemapUrl)	A sitemap.xml, sitemap index or text sitemap. Its pages are read. Without start URLs, no other links are followed to new pages.	None	Up to 20 child sitemaps of an index
Maximum pages (maxPages)	The most pages read to collect links.	10	1 to 1,000
Maximum links (maxLinks)	The most unique links checked.	100	1 to 10,000
Check external links (checkExternalLinks)	On: links to other sites are checked too. Off: they are counted in REPORT only.	On
Request timeout (requestTimeoutSecs)	How long one request may take to answer.	15 seconds	3 to 60 seconds
Maximum redirects (maxHops)	The most redirects followed for one link or page.	10	0 to 20

A site is a host name, plus its port when that is not 80 or 443. So http and https addresses of the same host are one site.

Output

One dataset item per checked link:

Field	What it holds
linkUrl	The link, made absolute, with any #fragment removed
broken	true or false
finalStatus, finalUrl	The last answer in the chain and its address
hops, hopCount, loop	Every answer in the redirect chain, how many were redirects, and whether they loop
stopReason, redirectNotFollowed	Why a chain ended early, and where its last redirect pointed
isExternal	true when the link points to another site
foundOnPage, anchorText	The first page the link was found on, and its link text
timesFound, foundOnPages	How often the link appears across the pages read, and up to 10 of those pages
responseTimeMs, checkedAt	How long the answers took, and when the last one arrived

Example row from a local test run:

{"linkUrl": "http://127.0.0.1:18950/deep404", "broken": true, "finalStatus": 404, "finalUrl": "http://127.0.0.1:18950/deep404", "hopCount": 0, "isExternal": false, "foundOnPage": "http://127.0.0.1:18950/page2", "anchorText": "Deep", "timesFound": 1}

Only <a href> links are collected. mailto:, tel:, javascript: and other non-web links are counted in REPORT and never stored, so no email address or phone number on a page is written anywhere.

Pricing

This Actor is priced per event, with two charges:

• Actor start: Apify's automatic start event, $0.00005 per start (one event per GB of run memory, minimum one).
• Link checked: one charge for each link written to the dataset.

Price per checked link: $1.00 per 1,000 links, which is $0.001 each.

Reading pages is never charged. These are never charged either: links robots.txt disallows, links that time out or whose site cannot be reached, links that answer 401, 403 or 429, links that lead to a login or CAPTCHA page, links past Maximum links, and the REPORT and SKIPPED records.

If you set a maximum cost for the run and it is reached, the Actor checks no more links, lists the rest in SKIPPED as spending_limit_reached, and ends the run without an error.

Records

Record	What it holds
REPORT	Why the run stopped, the settings used, counts (pages read, unique links found, links checked, broken, unreachable, skipped by reason), every page read and why any page was not read, robots.txt results per site and requests per host
SKIPPED	Every link that was not checked or not saved, with the reason and the page it was found on. Links that timed out or whose site could not be reached are marked unreachable. Up to 5,000 are listed, then a count of the rest

Rules it follows

• robots.txt is read first on every site, including every site a redirect leads to, and obeyed. A robots.txt that answers a server error, 401, 403 or 429 skips that whole site.
• One request at a time to any one site, with the site's Crawl-delay. A Crawl-delay over 30 seconds skips the site.
• Links are checked with HEAD. GET is used only when HEAD is refused, and then the body is not read.
• A page is read only when it answers with HTML, and at most 5 MB of it. PDFs, images and other files are checked as links but never downloaded.
• On Apify, private and local network addresses are refused.
• There is no cookie, header, login, proxy or user agent input. The user agent starts with broken-link-checker/0.1.

Your responsibility

You are responsible for having the right to check the sites you give this Actor and for following their terms of use. The Actor obeys robots.txt for you, but it cannot read or judge a site's terms.

Limits

Limit	Value
Pages per run	1 to 1,000
Links checked per run	1 to 10,000
Page size read	5 MB of HTML per page
Sitemaps	20 MB per sitemap file, 20 child sitemaps per index
Memory	256 MB by default