Broken Link & Redirect Checker

Developer: Luxe_tasks.

Give it one or more pages of a site you own or are authorized to audit. It crawls that site (same registered domain only, within your page and depth limits), checks every link it finds, internal and external, and reports for each link: the HTTP status, the redirect chain (every hop with its status and Location resolved to a full URL (when valid), and the final URL), whether it is broken (4xx, 5xx, redirect loop, DNS failure, timeout), the pages where it was found (up to 3) and its anchor text. Links it could not verify (bot walls, rate limits, robots.txt) are reported as such, never as broken, and have no per-link or per-page charge (the per-run start fee still applies).

How it checks

• Your site (same registered domain as the start URL, for example www.example.org and blog.example.org for example.org): pages are opened with GET and their <a href> links collected, breadth-first, up to Max link depth clicks from the start URL. Same-site links beyond the depth limit get one lightweight check (below), with redirects followed inside the site.
• Other sites (external links): one lightweight request per link: HEAD, or a 1-byte ranged GET if the server does not support HEAD (405/501). A redirect is reported (status, Location) and not followed. External sites are never crawled.
• Each unique link is checked once per start URL (fragments #... removed), however many pages link to it. mailto:, tel:, javascript: and similar links are ignored.
• Compressed pages (gzip, deflate, brotli) are decoded. Pages are read up to 5 MB; links after that point are not seen (the report flags the page with pageTruncated).
• No JavaScript is executed: links that only exist after client-side rendering are not seen. Sitemaps are not read in this version.

Input

• startUrls (required, up to 100): one report per start URL. URLs with a user name or password (user:pass@host) are refused; query strings are never logged.
• maxPages (default 500, 1-5,000): most same-site URLs requested in the whole run (pages crawled plus same-site links checked without crawling; the redirects of one URL count as that URL). Links left over are listed as not_checked_page_cap and have no per-link charge.
• maxDepth (default 5, 0-10): clicks from the start URL to crawl. 0 opens only the start URLs and checks their links.
• checkExternalLinks (default on): check links to other sites (one lightweight request each, plus one robots.txt request per external site). Off: they are listed as not_checked_external_off, not requested and have no per-link charge.
• timeoutSecs (default 15, 2-60): how long to wait for one response; a link that does not answer in time is reported as timeout (broken, no per-link charge).
• maxConcurrency (default 3, at most 3): start URLs processed in parallel. Each site still gets at most about 1 request per second.

Each run has a default timeout of 1 hour (3600 seconds); for very large lists you can raise the timeout in the run options. If the run is about to reach its timeout, the Actor stops starting new requests, saves what it has, and lists the rest as not_checked_time_limit (no per-link charge).

Output

• Dataset: one record per start URL (query strings shown as ?[redacted]): index, url, status, ok, charged, chargedEvents, chargedByEvent, httpStatus, error, robots, timingMs plus startVerdict, pagesCrawled, uniqueLinks, linksChecked (charged link-checked events), pagesCharged (charged page-crawled events), brokenCount, redirectedCount, notCheckedCount, internalLinks, externalLinks, pageCapReached, budgetReached, timeLimitReached, stoppedSites, verdicts (count per verdict), and brokenLinks (first 100: url, status, verdict, foundOn, anchorText). The Broken links view lists them one per row.
• Full report (key-value store, link-report-<index>-<host>-<hash>-json, linked from the record): every unique link with

  url, internal, crawled, verdict, broken, billed, billedEvent, status, method, redirectChain [{url, status, location}], finalUrl, redirectNotFollowed, foundOn, occurrences, anchorText, error
  . The report keeps full URLs, including query strings. Above 8 MB, entries of links that are simply OK (200, no redirect) are left out and counted in okLinksOmitted.
• RUN_SUMMARY (key-value store): counts, charged events (a start URL charged both events is listed as page-crawled*N+link-checked*M), stopped domains. To compare with Apify's billing view, chargedByEvent shows the charges this Actor recorded per event, for example {"page-crawled": 10, "link-checked": 9}: in each dataset record (that start URL), in RUN_SUMMARY's itemCharges (one entry per record saved since the last start or resume) and as RUN_SUMMARY's total (charges made since the last start or resume, so after a restart it can be lower than what Apify billed for the whole run; chargeUncertainByEvent holds events whose charge result is unknown). STOPPED_SITES: sites stopped during the run. Apify's billing is the official record.

Verdicts

Verdict	Meaning	Broken	Charge (link or crawled page)
ok	2xx (after any redirects inside your site; see redirectChain)	no	yes
redirect	external link answered with a redirect (not followed; finalUrl = its target)	no	yes
redirect_offsite	same-site link redirects to another site (not followed)	no	yes
auth_required	401 / 407	no	yes
broken	other 4xx (404, 410, ...)	yes	yes
server_error	5xx	yes	yes
redirect_loop	more than 10 redirects	yes	yes
bad_redirect	redirect without a valid Location	yes	yes
dns_failure, timeout, network_error	no HTTP answer	yes	no
blocked_403, rate_limited_429, blocked_captcha	the site refused us: we cannot tell whether the link works	unknown	no
ssrf_blocked, credential_url_refused	private / metadata address, or links with a user name or password in the URL's user-info part: never requested	unknown	no
not_checked_robots, not_checked_robots_unreachable	robots.txt disallows it / robots.txt failed	unknown	no
not_checked_site_stopped, not_checked_page_cap, not_checked_budget, not_checked_time_limit, not_checked_external_off	not requested	unknown	no

Record statuses with no per-link or per-page charge (the per-run start fee still applies): blocked_or_captcha (the start page refused us), start_unreachable, budget_reached, robots_disallowed, rate_limited, skipped_blocked, ssrf_blocked, invalid_input.

Pricing

Pricing: $0.001 per link checked ($1 per 1,000 links) plus $0.0005 per crawled same-site page (each unique page once; a crawled page does not also count as a link), plus the start fee of $0.005 per run. Links and pages that could not be verified have no charge; the per-run start fee still applies.

• Pay-per-event page-crawled ($0.0005): one event per unique page of your site (per start URL) that the Actor fetched to crawl it, the start page included, and that got an HTTP answer we can report: the verdicts marked "Charge: yes" above (so a crawled page that answers 404 or redirects counts). These are the report entries with crawled: true and billedEvent: "page-crawled"; crawled: true is also set on pages of your site the crawl reached but did not fetch (robots.txt, refused address, timeout), which have no billedEvent and no charge (the per-run start fee still applies).
• Pay-per-event link-checked ($0.001): one event per unique link (per start URL) checked without being crawled (external links, and links of your site beyond the depth limit) that got an HTTP answer we can report: the verdicts marked "Charge: yes" above.
• Each URL is charged at most once per start URL, as one of the two events (billedEvent in the report). Verdicts marked "Charge: no" have no charge, for crawled pages and links alike, and neither do the record statuses listed above; the per-run start fee still applies.
• Example: a run that crawls 200 pages of your site and checks 3,000 other links costs 200 x $0.0005 + 3,000 x $0.001 + $0.005 = $3.105.
• Events are charged only after the report is saved. A maximum total charge set on the run is respected: the price of the event is reserved before each request, and when the budget is used up the remaining links and pages are listed as not_checked_budget.
• A run may stop a fraction of a cent before your maximum charge, because the next link or page would not fit in the remaining budget.
• What you pay even when every link fails: the start fee of $0.005 per run. Apify charges it through its synthetic apify-actor-start event on every run: "Up to and including 1 GB of RAM, the event is charged once. Then, it's charged once for each extra GB of memory" (Apify docs). This Actor's memory is fixed at 1 GB, so the start fee is charged once per run.
• The synthetic apify-default-dataset-item event is not used (the Actor refuses to run if it is priced).

Politeness and safety (kit defaults; may only be tightened)

• robots.txt is honoured for every request: crawled pages, link checks and every redirect hop, on your site and on external sites; Crawl-delay is respected.
• A robots.txt that returns 4xx (except 429) means no restrictions; a 429 is treated as a rate limit and stops the domain; 5xx or an unreachable robots.txt means nothing is fetched.
• If a site's host name does not resolve, its links are reported as dns_failure, not as robots.txt failures.
• At most about 1 request per second per site (scheme, host and port); fixed honest User-Agent BrokenLinkCheckerActor/0.1.0 (+https://apify.com/luxe_tasks/broken-link-checker; honours robots.txt); no proxy rotation, no evasion.
• A 403, 429 (Retry-After over 60 s, or still 429 after one retry), a CAPTCHA/bot-wall page, or three 5xx answers stop that site for the rest of the run (and after a restart); its remaining links are listed as not checked and have no per-link charge.
• Private, loopback, link-local and cloud-metadata addresses are refused, re-checked after DNS and on every redirect. Links with a user name or password in the URL's user-info part are never requested.
• No logins, cookies, headers, proxies or session tokens can be supplied. robots.txt cannot be overridden in this version.

Acceptable use

Acceptable use. Only crawl sites you own or are authorized to audit; external links are checked with a single lightweight request (plus that site's robots.txt). By running this Actor you confirm that:

1. You will submit only start URLs of sites you own or are authorized to audit, and you will use the output lawfully.
2. You will not submit pages behind a login, paywall, or other access control, and the Actor does not accept passwords, cookies or session tokens.
3. You will not use it to get around CAPTCHAs, rate limits, IP blocks, or any other technical measure a site uses to restrict access.
4. You will comply with the target site's terms of use and robots.txt, and with all applicable law, including copyright and data-protection law (GDPR, CCPA and similar). You are the controller of any personal data in your results.
5. You will not use the Actor to harass, stalk, or profile individuals, or for vulnerability scanning or attack reconnaissance. The Actor reports link status, not vulnerabilities.
6. You will keep request volumes reasonable. The Actor enforces per-site rate limits and identifies itself in its User-Agent. The Actor refuses private-network addresses and URLs with a user name or password, and skips URLs that robots.txt disallows. Report problems through the Actor's Issues tab.

Privacy

This Actor stores no page content except link anchor text (up to 200 characters); reports hold URLs, HTTP statuses, redirect targets and that anchor text. The dataset and the report's start URL redact query strings; the report's link list keeps full URLs, including query strings. User names and passwords in a URL's user-info part (user:pass@host) are removed from every stored URL, including redirect targets. Secrets placed in a path or query string are not removed, except where query strings are redacted as described above. Do not submit start URLs that carry secrets or personal data in the query string.

"Results are written only to your run's default Apify storage. The developer does not copy, view or retain them. If your results contain personal data, you are the controller and the developer acts only as your processor under Apify's Standard Actor Contract. Apify's retention settings apply."

Third-party software

Runtime dependencies (apify, robots-parser, tldts) are listed in NOTICE with their licences.

Runtime

Node 24 (engines, .nvmrc, Dockerfile apify/actor-node:24). Memory: fixed at 1024 MB (the work is network-bound and rate-limited to about 1 request per second per site, so more memory does not make a run faster; the local peak was about 400 MB with three 5 MB pages holding 20,000 links each). Tests: npm test; end-to-end: sh scripts/e2e-netns.sh (local fixture sites only).

Contact: https://apify.com/luxe_tasks · Issues tab on the actor page.