Broken Link Checker and Bulk URL Status Checker

Give it a list of URLs, or a website to crawl, and it tells you which links work and which do not. For every URL you get the HTTP status code, the full redirect chain, the final URL, the response time and a plain error type (DNS, SSL, timeout, 4xx, 5xx). In crawl mode each row also lists the pages where the link was found, so you know where to fix it.

It uses HEAD requests and falls back to GET when a server answers HEAD badly, which many do. It reads no JavaScript, opens no browser and needs no login.

What you can use it for

Use	How
Find dead links on your site	Crawl mode on your home page; filter the dataset to broken
Check a redirect migration	List mode with your old URLs; compare finalUrl with what you expected
Audit redirect chains	Look at redirectCount and redirects (301 then 302 then 200 is easy to spot)
Clean a backlink or outreach list	List mode with the URLs; drop the rows with DNS, 404 or 410
Monitor a handful of pages	Schedule the Actor on a list of key URLs and watch for status changes
Feed an AI agent or workflow	Call it through the API or MCP and read the JSON rows

How to use it

1. Paste URLs, one per line. A missing https:// is added for you.
2. Pick the mode. List checks exactly what you pasted. Crawl opens each page you pasted, reads its links and checks them.
3. Start the run and read the dataset. The "Error details" view shows the reason for each failure.

Example input:

{
    "mode": "crawl",
    "urls": ["https://www.sitemaps.org/"],
    "maxDepth": 1,
    "maxPagesPerSite": 4,
    "checkExternal": true
}

Field	Default	What it does
urls	required	URLs to check (list mode) or pages to start from (crawl mode)
mode	list	list or crawl
maxDepth	1	Crawl only. 0 opens just the page you gave. 1 also opens the internal pages it links to. Deeper links are still checked, just not opened
maxPagesPerSite	50	Crawl only. Pages opened per site
checkExternal	true	Crawl only. Also check links to other sites (status only, never opened)
respectRobotsTxt	true	Crawl only. See Limits
maxConcurrency	10	Requests in flight in total
perDomainConcurrency	2	Requests in flight against one domain
perDomainDelayMs	200	Minimum gap between two requests to one domain
requestTimeoutSecs	20	Wait time per answer
maxRetries	2	Retries for timeouts, resets, 429 and 5xx
userAgent	empty	Empty uses BrokenLinkCheckerBot/1.0, which says what it is

Output

One row per URL. This is a real row from a crawl of sitemaps.org, shortened:

{
    "url": "http://asg.web.cmu.edu/rfc/rfc3986.php",
    "status": "broken",
    "statusCode": null,
    "finalUrl": "http://asg.web.cmu.edu/rfc/rfc3986.php",
    "redirectCount": 0,
    "redirects": [],
    "errorType": "dns",
    "error": "Domain does not exist or has no address (ENOTFOUND).",
    "responseTimeMs": null,
    "totalTimeMs": 170,
    "method": "HEAD",
    "isExternal": true,
    "foundOn": ["https://www.sitemaps.org/faq.php"],
    "foundOnCount": 1,
    "charged": true
}

status is one of:

status	Meaning
ok	Answered 2xx with no redirect
redirected	Ended on a 2xx after one or more redirects. Check redirects
broken	A real failure: 404, 410, other 4xx, 5xx, DNS, SSL, refused connection, redirect loop, too many redirects
unverified	No reliable answer: timeout, 429, connection reset, 401/403/999 (often bot protection)
skipped	Not requested because robots.txt disallows it
invalid-input	The line was not a checkable URL

errorType is dns, ssl, timeout, connection-refused, connection-reset, http-4xx, http-5xx, access-denied, rate-limited, redirect-loop, too-many-redirects, invalid-redirect, robots-blocked and a few others. A 5xx is reported only after retries, so a one-off 503 does not show up as broken.

A run summary (rows, billed URLs, URLs not checked, time) is stored under SUMMARY in the key-value store.

How much does it cost?

You pay per checked URL (the url-checked event; the price is on the Pricing tab). A URL is billed when it gets a real answer: an HTTP response of any status except 429, a DNS failure, an SSL error, a refused connection, a redirect loop. These are not billed:

• timeouts, 429 answers, connection resets and temporary DNS failures (we could not get an answer, so we do not charge for the attempt)
• invalid lines and pages skipped because of robots.txt
• a URL that appears twice in one run

The Actor reads your maximum cost per run and stops by itself when it is reached. It never starts a check it could not bill within that limit. The summary says how many URLs were left unchecked. In crawl mode, pages that get opened count as checked URLs too.

How it works

Each URL is requested with HEAD. If the answer is an error, or the server hangs, the same URL is requested with GET and the GET answer wins. Redirects are followed one by one, up to 10, so the whole chain is recorded and loops are caught. Retries use a short backoff and honor Retry-After. After a 429 the domain is slowed down for the rest of the run.

In crawl mode the Actor opens the start page with GET, reads the <a> links (the <base> tag is honored), opens internal HTML pages down to maxDepth, then checks every link it found. Links to files (PDF, images and so on) are checked, never downloaded. mailto:, tel: and javascript: links are ignored.

Limits, read before you run

• Public pages only. No login, no cookies, no captcha solving. Pages that need a session will report 401 or 403.
• Private, local and internal addresses (localhost, 10.x, 192.168.x, 169.254.x) are rejected on purpose.
• Links are read from the HTML the server sends. Links added later by JavaScript are not seen.
• Only <a href> and <area href> links are checked. Images, scripts and stylesheets are not.
• robots.txt: with respectRobotsTxt on, pages and internal links it disallows are not requested. External links are checked with a single HEAD or GET to the URL itself, without reading the other site's robots.txt, the same way a browser or link checker does when you click a link.
• Some sites block anything that is not a browser. They show up as unverified with 403 or 999 (LinkedIn does this), not as broken. We do not try to get around that.
• A timeout means "no answer in time from where the Actor runs". The site may still be fine for you. Try a longer timeout before you trust it.
• There is no proxy support. Sites that block data center IPs may answer differently than they would for a visitor.
• Rows come out in the order checks finish, not in input order.
• One URL is checked once per run, even if several sites link to it. In crawl mode foundOn keeps the first 25 pages.

Is it legal?

It requests public URLs the way a browser does when you click a link, at a low rate: two parallel requests per domain by default, with a pause between them. It sends an honest User-Agent. It stores no personal data, only URLs and what the server answered. You are responsible for checking only sites you are allowed to test, and for the terms of any site you point it at.

FAQ

Why is a working link marked unverified? The server answered 401, 403 or 999, or took too long. Open the link yourself. If it works in a browser, the site is filtering robots.

Why GET sometimes, if HEAD is lighter? The method column shows what the final answer came from. GET means HEAD failed or returned an error, and we asked again to be sure. The body is not downloaded, except for pages opened in crawl mode.

Can it check 100,000 URLs? Yes in list mode. Speed depends on how many domains you have: with 2 requests and a 200 ms gap per domain, 100,000 URLs on one domain takes hours, 100,000 spread over many domains takes much less. Raise perDomainConcurrency only for sites you own.

Does it check anchors (#section)? No. The fragment is removed before the request, and the page is not searched for the anchor.

Can I use it from the API or an AI agent? Yes. Run it through the Apify API, a webhook or the Apify MCP server and read the dataset items.

Not affiliated with any of the websites you check.