# Changelog of Backlink & Referring Domains Checker · SiteVerdict (`siteverdict/siteverdict-backlink-explorer-commoncrawl`) Actor

- **URL**: https://apify.com/siteverdict/siteverdict-backlink-explorer-commoncrawl/changelog.md
- **Full Actor documentation**: https://apify.com/siteverdict/siteverdict-backlink-explorer-commoncrawl.md

## Changelog

### 1.2 (5 Nov 2026)

- Fix (Apify daily test / "under maintenance"): Common Crawl's archive index (index.commoncrawl.org) is often slow or returns 502/504. Homepage checks are now bounded: 15 s per request (was 60 s), 2 minutes for all checks together, the run deadline is checked before every request, and after 3 failures in a row the remaining checks are skipped. Unchecked rows show `homepageLink.status: INDEX_UNAVAILABLE` (`found: null`), never "no link", and are not charged; the summary counts them.
- Fix: the graph release list (graphinfo.json) is retried up to 3 times (5 s, 10 s) on 5xx, network errors and unexpected responses instead of failing the run on the first error.
- Fix: the small graph files (properties, size probe) on data.commoncrawl.org are checked and retried up to 4 times (3 s, 6 s, 10 s) on 5xx/429/network errors. Before, a non-200 answer was parsed as the file and the run failed with 'BVGraph properties: missing or invalid "nodes"' (nothing was charged).
- Billing safety: `graph-scan` and `target-analysed` are charged only after the first referring-domain row of that domain is saved (before, a run cut by a short timeout could be charged $0.80 with 0 rows). Domains with 0 referring domains, unresolved `#id` rows and incomplete homepage checks are free; an anchor check is charged only when a homepage capture was actually read. A run that produces nothing useful now ends as FAILED with the reason (nothing charged). Prices, events, inputs, example and memory are unchanged.

### 1.0.7 (15 Oct 2026)

- Fix: the referring-domain name lookup now also respects the run time limit. On a slow Common Crawl download it could run past the platform timeout and the run ended with no rows (seen once on the 300 s example run). It now stops about 45 s before the limit, writes every row (unresolved names shown as #id), marks the run partial and says why. No change to inputs, outputs or pricing.

### 1.0.x — 2026-10-15

- Fix: runs that hit the platform timeout lost everything. The Actor now reads its run time limit, stops the graph scan with 2 minutes in hand, saves results for domains already reached, reports unreached domains with "Skipped (run time limit)" and an estimated timeout, and charges `graph-scan` only if at least one domain was reached. Partial runs say "Partial (run time limit)".
- Speed: the graph decoder reads bit fields 32 bits at a time; profiling showed bit reading was ~70% of CPU. Decoding is about 2.5x faster (measured 92 MB to 248 MB of graph in the same 40 s).
- Speed: name and rank lookups now read the gzipped vertex and rank lists in batches instead of one awaited line at a time (133 million lines). Measured on aph.gov.au: name lookup 235 s -> 32 s under the Apify SDK.
- Pre-filled example (also used by Apify's daily test) changed from python.org (~15-20 min) to aph.gov.au with 100 rows and 2 homepage checks (~1-2 min). python.org stays documented as an extended performance test.
- README: run-time guidance replaced with measured figures (the old "5–15 minutes" was too optimistic for `.org`/`.com` targets).

### 1.0.0 (2026-10-11)

- First release.
- Finds referring domains in Common Crawl's domain-level web graph. Releases are chosen from graphinfo.json, either "latest" or a named release.
- Includes a native streaming decoder for WebGraph BVGraph files: gamma, zeta and unary codes, reference copy blocks, intervals and residuals. Downloads resume with HTTP Range after an interruption.
- Resolves the vertex list to domain names and host counts, and adds harmonic-centrality authority ranks from the top of the ranks file.
- Verifies homepage links through the CDX index and WARC range reads, reporting anchor text, nofollow/ugc/sponsored and the first crawl seen within the release.
- Adds a coverage statement to every summary. One graph scan is shared by up to 20 targets.
- Sample mode runs on a synthetic graph in Common Crawl's exact formats. A live cross-check script is included (scripts/verify-graph.ts).