Bulk URL Status & Broken Link Checker

Check thousands of URLs in one run and get a clear verdict for each: ok or broken, with the exact reason — HTTP status, full redirect chain, final URL, latency and a machine-readable error category (404, timeout, DNS, SSL, redirect loop). Built to avoid false alarms: HEAD-first with GET verification, automatic retries on flaky errors. Checks only the URLs you provide — it never crawls or follows links on pages. No credentials needed.

What it does

Give it a list of URLs (duplicates are removed). For each URL you get one dataset item with:

• result / ok — "ok" (2xx after redirects) or "broken", also as a boolean
• httpStatus — final status code (null when nothing responded)
• finalUrl, redirectCount, redirectChain[] — every hop with its status and Location
• latencyMs — total time including redirects
• errorCategory — http_client_error, http_server_error, dns, timeout, ssl, connection, redirect_loop, too_many_redirects, invalid_url
• error — human-readable reason (null when ok)
• contentType, contentLengthBytes, method, attempts, checkedAt

Accuracy features (why fewer false alarms)

• HEAD-first, GET-verified: many servers mishandle HEAD (405/403/404 on pages that load fine). A URL is only reported broken after a GET confirms it.
• Retries: network errors and 5xx are retried (configurable) before the verdict, so transient blips don't show up as broken links.
• Bodies are never downloaded — GET responses are closed after the headers, so large files check as fast as small pages.
• Polite: never more than 2 concurrent requests to the same host, no matter how high you set the overall concurrency.

Input

Field	Type	Default	Description
urls	array	—	URLs to check (only these are requested; no crawling)
timeoutSecs	integer	15	Per-request timeout; slower URLs are timeout
retries	integer	1	Retries on network errors and HTTP 5xx
maxRedirects	integer	10	Hop limit before too_many_redirects
concurrency	integer	10	Parallel checks (per-host always capped at 2)

Example output (abridged)

{
    "url": "http://github.com",
    "result": "ok",
    "ok": true,
    "httpStatus": 200,
    "finalUrl": "https://github.com/",
    "redirectCount": 1,
    "redirectChain": [{"url": "http://github.com", "status": 301, "location": "https://github.com/"}],
    "latencyMs": 142,
    "contentType": "text/html; charset=utf-8",
    "method": "HEAD",
    "attempts": 1,
    "error": null,
    "errorCategory": null,
    "checkedAt": "2026-08-18T09:30:00Z"
}

Typical uses

• Find dead links from a sitemap, CMS export or backlink list
• Audit redirect chains after a site migration (every hop is recorded)
• Scheduled uptime/health checks of a URL list, with latency numbers
• Clean stale URLs out of datasets and bookmark collections

Limitations

• Checks the exact URLs given — it does not discover links on pages (pair it with a sitemap extractor if you need URL discovery)
• Sites that block automated clients may report broken here yet load in a browser
• JavaScript-rendered soft-404s (page says "not found" but returns 200) are reported as ok, because the HTTP layer says ok

Development (local)

cd actors/url-status-checker
uv venv --python 3.13 .venv && uv pip install -p .venv/bin/python -r requirements.txt

.venv/bin/python tests/run_local_test.py           # end-to-end test (apify run equivalent), exit 0 = ALL PASS
../../node_modules/.bin/apify run                  # real apify CLI local run (input: storage/key_value_stores/default/INPUT.json)

The test spins up a local HTTP server simulating ok / redirect chains / 404 / persistent 500 / recovers-on-retry / HEAD-hostile / redirect loop / timeout; expected results live in tests/expected_output.json. Publishing → ../../docs/publishing.md.