Broken link checker and 404 checker: crawl a website and find broken links and images (404s, 410s, 5xx errors, DNS failures, timeouts) with the page each one is on, the link text and the status code. Internal and external links. Healthy links are free. Respects robots.txt.
Version stays "1.0" in .actor/actor.json: Apify accepts only MAJOR.MINOR there and assigns the patch (build) number at the next build, which is the patch bump for this release.
2026-11-02 (actor.json version unchanged: 1.0): private-address (SSRF) guard
On the Apify platform, every request (start pages, crawled pages, checked links and images, each redirect hop, and
robots.txt) is refused when the host resolves to a private, loopback, link-local, carrier-grade NAT, reserved,
multicast or unspecified address (including IPv4-mapped IPv6), or when any of the host's addresses is one. Same
rules as Bulk HTTP Request.
Such a link is "not checked" (ok unknown), like a link robots.txt disallows: it is never charged, is counted in
RUN_SUMMARYnotChecked, and is listed only when onlyBroken is off, as isBroken: false,
error: "not checked: private, loopback and link-local addresses are not allowed".
A start URL with a private address is not crawled. The dataset gets one free row: recordType: "status",
status: "error", message: "Not charged: <url> was not crawled (...)".
A host the guard cannot resolve fails as a DNS error, as before (it is still a broken link, still charged as
before); only the error text now reads "DNS or connection error: DNS lookup failed for ".
A robots.txt that redirects to a private address counts as unavailable (no robots.txt rules).
Local runs (APIFY_IS_AT_HOME not set) are unchanged.
Files: my_actor/common.py (guard helpers; Http._robots_for checks the site first), my_actor/main.py (hook on
the httpx client, Blocked verdict, start-URL status row), tests/test_ssrf_guard_1102t.py (new), README Output
error row and Limits.
Unreleased (next 1.0.x build): per-link timestamps, robots.txt on redirects, free robots rows, fail-soft sample run (round 1030)
BREAKING (output values):checkedAt is now the time each link was checked, not the run start time.
BREAKING (fewer requests and charges): redirects are followed one hop at a time (at most 10) and every redirect target is checked against its own site's robots.txt. A link that redirects to a disallowed page is not followed; its row says not checked: redirects to <url>, which robots.txt disallows and is not counted as broken.
BREAKING (charges): rows for links not checked because of robots.txt (shown only with Only broken links off) are now output free. Before, each was charged as a link event.
Empty input ({}) and the prefilled sample page: the sample check stops after 15 seconds. If the sample site is down or slow, the run still succeeds and saves one free row with a warning field.
New title: "Broken Link Checker: Find 404s & Dead Links on Any Website". New Store description (only claims covered by tests or the code).
Charges (lower): in the default mode (Only broken links on) each unique broken URL is charged once per run as a broken-link event. You still get one row per page that links to it; later rows for the same URL are free. Before, every broken-link row was charged. With Only broken links off, charging is unchanged (one link event per checked row).
Unchanged: startUrls accepts plain URL strings and {"url": ...} objects, as before; per-event prices.
