Find the domains that link to any website using Common Crawl's open web graph (billions of pages): referring domains with authority rank, TLD mix and verified homepage anchors with first-seen crawl. No scraping, honest coverage labels.
Fix (Apify daily test / "under maintenance"): Common Crawl's archive index (index.commoncrawl.org) is often slow or returns 502/504. Homepage checks are now bounded: 15 s per request (was 60 s), 2 minutes for all checks together, the run deadline is checked before every request, and after 3 failures in a row the remaining checks are skipped. Unchecked rows show homepageLink.status: INDEX_UNAVAILABLE (found: null), never "no link", and are not charged; the summary counts them.
Fix: the graph release list (graphinfo.json) is retried up to 3 times (5 s, 10 s) on 5xx, network errors and unexpected responses instead of failing the run on the first error.
Fix: the small graph files (properties, size probe) on data.commoncrawl.org are checked and retried up to 4 times (3 s, 6 s, 10 s) on 5xx/429/network errors. Before, a non-200 answer was parsed as the file and the run failed with 'BVGraph properties: missing or invalid "nodes"' (nothing was charged).
Billing safety: graph-scan and target-analysed are charged only after the first referring-domain row of that domain is saved (before, a run cut by a short timeout could be charged $0.80 with 0 rows). Domains with 0 referring domains, unresolved #id rows and incomplete homepage checks are free; an anchor check is charged only when a homepage capture was actually read. A run that produces nothing useful now ends as FAILED with the reason (nothing charged). Prices, events, inputs, example and memory are unchanged.
1.0.7 (15 Oct 2026)
Fix: the referring-domain name lookup now also respects the run time limit. On a slow Common Crawl download it could run past the platform timeout and the run ended with no rows (seen once on the 300 s example run). It now stops about 45 s before the limit, writes every row (unresolved names shown as #id), marks the run partial and says why. No change to inputs, outputs or pricing.
1.0.x — 2026-10-15
Fix: runs that hit the platform timeout lost everything. The Actor now reads its run time limit, stops the graph scan with 2 minutes in hand, saves results for domains already reached, reports unreached domains with "Skipped (run time limit)" and an estimated timeout, and charges graph-scan only if at least one domain was reached. Partial runs say "Partial (run time limit)".
Speed: the graph decoder reads bit fields 32 bits at a time; profiling showed bit reading was ~70% of CPU. Decoding is about 2.5x faster (measured 92 MB to 248 MB of graph in the same 40 s).
Speed: name and rank lookups now read the gzipped vertex and rank lists in batches instead of one awaited line at a time (133 million lines). Measured on aph.gov.au: name lookup 235 s -> 32 s under the Apify SDK.
Pre-filled example (also used by Apify's daily test) changed from python.org (~15-20 min) to aph.gov.au with 100 rows and 2 homepage checks (~1-2 min). python.org stays documented as an extended performance test.
README: run-time guidance replaced with measured figures (the old "5–15 minutes" was too optimistic for .org/.com targets).
1.0.0 (2026-10-11)
First release.
Finds referring domains in Common Crawl's domain-level web graph. Releases are chosen from graphinfo.json, either "latest" or a named release.
Includes a native streaming decoder for WebGraph BVGraph files: gamma, zeta and unary codes, reference copy blocks, intervals and residuals. Downloads resume with HTTP Range after an interruption.
Resolves the vertex list to domain names and host counts, and adds harmonic-centrality authority ranks from the top of the ranks file.
Verifies homepage links through the CDX index and WARC range reads, reporting anchor text, nofollow/ugc/sponsored and the first crawl seen within the release.
Adds a coverage statement to every summary. One graph scan is shared by up to 20 targets.
Sample mode runs on a synthetic graph in Common Crawl's exact formats. A live cross-check script is included (scripts/verify-graph.ts).
![Moz Scraper [$6+💰] Domain Authority, Spam Score & Keywords avatar](https://images.apifyusercontent.com/S28ueebCNAWUxuWkbuJE9MqzniMSG8dijtmI_mJvgMc/rs:fill:76:76/cb:1/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vbmduOHZSRVNUa3NleVdFNnAtYWN0b3ItZGNUeDFmRUl1YktYa2prNTQtNU80VVNMVDVENi1tb3oucG5n.webp){kind=avatar}
