# Changelog of Broken Link Checker & Redirect Chain Audit for Your Site (`hardwoodlogic/broken-link-checker`) Actor - **URL**: https://apify.com/hardwoodlogic/broken-link-checker/changelog.md - **Full Actor documentation**: https://apify.com/hardwoodlogic/broken-link-checker.md ## Changelog All notable changes to the **Broken Link & Redirect Chain Checker** Actor. Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html). The Apify build version (`.actor/actor.json`) tracks `MAJOR.MINOR`. ### \[0.1.1] - 2026-10-27 #### Security - An off-site redirect is a status check only. The body is discarded at the headers, before it is read, and nothing on that page is extracted or billed. robots.txt is checked on every hop, including a same-host path change and an outbound target. A disallowed hop is `not checked (robots.txt)` and is not billed, even when an earlier hop was already requested. - Ownership verification requests the exact `https:///.well-known/...` URL and does not follow redirects. Another host, a port, or userinfo fails the check. The token input is secret and is not written to the log or to OUTPUT. - A `` that is not http(s) is ignored. A stored value that is not http(s) is `[invalid URL]`. - Query strings are never stored. A discovered link's query is sent only when Keep query strings is on. A redirect `Location` query is always requested, and loop detection uses that full URL. - Status checks strip `authorization`, `proxy-authorization`, `cookie`, `x-goog-api-key`, and caller-marked sensitive headers on a cross-origin redirect, including 307 and 308. The redirect cap stays 5. #### Fixed - A robots.txt larger than 500 KiB is truncated and treated as unreachable, so the host is disallowed. A cut such as `Allow: /` is not parsed. A complete file of at most 500 KiB is parsed. A file that cannot be read disallows the host. - The per-attempt timeout starts after the host slot is acquired. - The default dataset is the charge ledger. A resumed run rebuilds the charged set from pushed rows, so a landed push is not charged again. Crawl progress is appended as deltas about every 30 seconds and on migrating, aborting, and persistState. A save snapshots array lengths before the key-value write, so entries appended during that write land in the next part. Parts are `CRAWL_G_PART_`. A v2 head with no `gen` is generation 0 and still uses `CRAWL_PART_`. Compaction writes the next generation as several parts, each at most 1 MiB of compact JSON, then switches `gen` and `parts` in one head write. The generation holds at most 64 parts (64 MiB), enough for 50,000 links and 50,000 pages. A head or part that is the wrong shape, or over those caps, fails the run before any request. A failed save is logged and does not fail the run; the dataset stays the charge ledger and OUTPUT is still written. A hard kill can re-charge at most 4 links, the pushes still in flight. An `onlyNewBreaks` charge is written into one ledger record for its in-flight batch before that slot is released, so it is inside that cap. - A resumed link uses the dataset row's `firstSeenDepth` as a floor, clamped to 0 through the hard depth cap, so a hard kill cannot put that link back at depth 0 and crawl past `maxDepth`. A part URL is kept only when it is an http(s) origin and path on the start host (or its www twin when Include www is on). Private addresses are not restored into the crawl. A query is not stored. Anchor text is scrubbed again on load. An outbound link that was already pushed stays in the dataset and is not crawled again. - An empty crawl head is written before the first charge. `CRAWL_UNPUSHED` is merged on resume even when that head was never written. A kill in the first 30 seconds re-charges at most the 4 checks still in flight. The ledger keeps only charges not yet in a committed part, and stays within 64 KiB. If that record still cannot be saved after a short retry, the run stops before another charge. A corrupt ledger, including a value that is not JSON, stops the resume before any request. Ledger URLs merged on resume stay uncommitted until a crawl part saves them, so a later kill does not bill them again and does not drop a charge that was still in flight. A user's charge limit still ends `SUCCEEDED` with exactly those charges. Storage errors are redacted before they are logged. 256 URLs of about 48 characters is about 12 KB, and the 64 KiB cap still bounds the record. A failed crawl-state write still publishes `OUTPUT` and then fails the run. A failed final save still writes `OUTPUT`. - The daily-test host root `https://hardwoodlogic.github.io/robots.txt` returns 404. The fixture file `https://hardwoodlogic.github.io/broken-link-checker-fixture/robots.txt` returns 200 with `Allow: /`. - A restarted run re-reads a charged own-host page that was not parsed, without charging it again, and keeps the same depth a fresh discovery would have used (`parent + 1`). `OUTPUT` counts every segment. `resumed` is true when the run continued. `linksChecked` stays equal to `linksCharged`. - With Keep query strings on, links that differ only by query collapse into one stored row. - A robots.txt 4xx, including 404, allows the host. Only a 5xx, an unreachable file, or a truncated file disallows it. An oversized robots.txt (over 500 KiB) disallows the whole host. - The daily-test prefill confirms ownership and starts at our fixture site, https://hardwoodlogic.github.io/broken-link-checker-fixture/ , with `maxPages` 10. - The per-host gap is 550 ms, so a sliding one-second window does not see three starts. - A resumed link keeps `foundOn` and the depth it was first seen at. - A robots.txt `Disallow` rule that contains a query is matched against the path and query of the URL that would be requested. - `linksChecked` counts successful `link-checked` events, the same number as `linksCharged`. - `onlyNewBreaks` without monitor mode is rejected before any request or charge. - A DNS failure (`ENOTFOUND`, `EAI_AGAIN`, and the same family) is an `error` row with error `dns` and is not billed. It is not reported as `not checked (robots.txt)`. #### Changed - Customers pay only the `link-checked` event, $1.00 per 1,000 links. Platform usage is included. `maxPages` still defaults to 100. Redirect chains still stop after 5. - Monitor mode, the robots override, and scheduled runs require the start URL on the default port. The ownership file is still checked at `https:///.well-known/...` with no port in the URL. - **Ignore robots.txt on my site** applies only to the verified start host. The www twin keeps its own robots.txt when Include www is on. ### \[0.1.0] - 2026-10-27 #### Added - First build. Crawl one host you own, status-check outbound links once, and report the redirect chain (at most 5 redirects). Monitor mode compares broken links with the previous run after the ownership file is verified. - Pay per event: `link-checked` at $0.001. No start fee and no `apify-actor-start` or `apify-default-dataset-item` event. - Ownership file `/.well-known/broken-link-checker-verify-.txt` is required for monitor mode, for ignoring robots.txt on your own site, and for scheduled runs (`APIFY_META_ORIGIN=SCHEDULER`).