SiteVerdict — Backlink Explorer (Common Crawl)

See which websites link to any domain, using the open web graph built by Common Crawl from billions of crawled pages. No scraping and no paid SEO-tool subscription.

Enter up to 20 domains (yours, your competitors' or your prospects'). For each one you get:

• every referring domain in Common Crawl's latest domain-level web graph (each release covers three monthly crawls);
• an authority rank for each referrer (Common Crawl's harmonic-centrality rank), its size (hosts crawled) and its domain suffix;
• for the top referrers, verified homepage links from Common Crawl's own page archive: anchor text, nofollow/ugc/sponsored, and the first crawl in the release where the link appears;
• a summary with the total count, the suffix mix (.com/.co.uk/.edu…), the top 20 referrers and plain coverage notes.

Who it is for

• SEO and digital PR teams: compare referring-domain profiles against competitors, and find sites that link to them but not to you.
• Agencies: do cheap first-pass backlink discovery for prospects and audits without a subscription.
• Researchers: run reproducible link-graph lookups on open data.

How it works (and why it is honest)

1. Common Crawl publishes a domain-level web graph for each release. It is a compressed WebGraph (BVGraph) file of about 133 million domains and 2 billion links, and it includes the transposed graph, which holds the links pointing to each domain.
2. The Actor finds your domain's node id, then streams the transposed graph from the start and decodes it until it reaches your domain. Common Crawl does not publish an offsets file, so the scan always starts at the beginning. It resolves the referring ids to names and, if enabled, adds authority ranks.
3. For the top N referrers it looks up their homepage captures in the Common Crawl URL index and reads only those WARC records by byte range, to show real anchors.

All of this uses Common Crawl's public files. Nothing is crawled live.

Coverage: please read

• Common Crawl samples the web; it does not crawl every page. Treat counts as a lower bound of real referring domains. They are typically far below what commercial crawlers (Ahrefs, Majestic, Semrush) report, but they come from open data and can be reproduced.
• Links are counted at domain level: a domain counts if any of its crawled pages links to any host of your domain. Page-level URLs, anchors for every link, and nofollow status are not in the graph. Anchors are verified for homepage links only.
• The data is as fresh as the release (each release covers about three months of crawls).

Speed and cost

Domains sort by reversed name, so the TLD decides how far into the graph file the Actor must read: au.* sits about 2% in, com.* roughly 10–50% (for example bharatfare.com 18%, siteverdict.com 49%) and org.* about 80%. Reading further means more GB and more minutes.

• Batch your domains. Up to 20 targets share one pass, at no extra download.
• Run time depends on where your domain sorts. Measured 15 Oct 2026 on the Jul–Sep 2026 release (4 GB memory):
  • Pre-filled example / daily test: aph.gov.au (Parliament of Australia, 2% into the graph), 100 rows and 2 homepage checks: about 1–2 minutes, with 11,000+ referring domains found.
  • abc.net.au with 200 rows and 3 homepage checks: about 6.5 minutes in the cloud before the 15 Oct speed fixes.
  • Extended performance test (not part of the daily test): python.org (84% into the graph) needs roughly 15–20 minutes; deep .com targets can take 10+ minutes. Set the run timeout to 30 minutes or more for these.
  • Homepage link checks (anchorChecks) each take several seconds because they query Common Crawl's archive politely; keep the number low for fast runs.
• Run time limit: the Actor watches its own run timeout. If the graph scan cannot reach a domain in time, it stops cleanly, saves results for any domain it did reach, lists the others with a "Skipped (run time limit)" note and an estimate of the timeout needed, and does not charge for them (the graph-scan event is charged only when at least one domain is reached). The status message then starts with "Partial (run time limit)". Keep the run timeout at 30 minutes or more for .com/.org targets.
• Stalled downloads: if Common Crawl sends no data for 60 seconds, the download is cut and resumed from the last byte received (up to 5 times). Other requests are limited to 60 seconds each, so a slow server cannot hang the run.
• Homepage checks when Common Crawl's archive index is down: each archive request is limited to 15 seconds, all homepage checks together to 2 minutes (and never past your run timeout), and after 3 index failures in a row the remaining checks are skipped. Rows whose check could not be completed show homepageLink.status = INDEX_UNAVAILABLE (with found: null, which does not mean "no link"), are not charged for the check, and the target summary says how many. The referring-domain results are unaffected.

Event	Price
graph-scan: one shared pass per run	$0.75
target-analysed: per domain	$0.05
referring-domain: per row output	$0.0005
anchor-check: per referrer homepage verified	$0.005

Example: 5 competitors, 1,000 referring domains each, 10 anchor checks each, costs $0.75 + $0.25 + $2.50 + $0.25 = $3.75 in one run.

Sample mode is free. Domains that are not in the graph are free, and so are targets skipped by the size cap. The $0.75 graph scan is only charged if your max total charge can also pay for at least one analysed domain; any domain the budget cannot cover is listed as "Skipped (maximum total charge reached)" and is not charged. graph-scan and target-analysed are charged only after the domain's first referring-domain row has been saved, so a run that stops or times out before writing a row costs nothing, and a domain with no referring domains is free. Unresolved #id rows and homepage checks that could not be completed are free.

Output

• Referring domains view: one row per referrer, with authority rank, hosts, suffix, and homepage-link verification (anchors, nofollow, first-seen crawl).
• Target summaries view: totals, suffix mix, top referrers and coverage notes.
• SUMMARY in the key-value store: release, GB downloaded, duration and events charged.

Run with Mode = Sample to see the full pipeline on a tiny synthetic graph in Common Crawl's exact formats.

FAQ

Why not just call an API? Common Crawl has no "backlinks" API. The graph files are the official way to use the link data, and this Actor does the heavy decoding for you.

Is it accurate? The decoder implements the WebGraph BVGraph format: gamma, zeta and unary codes, reference copy blocks, intervals and residuals. It is checked against Common Crawl's real files. Decoding the first 200,000 nodes of both the forward and the transposed graph must give exactly mirror-image link sets. Use scripts/verify-graph.ts to rerun that check.

Can I get page-level backlinks? No. That would mean processing the WAT files (hundreds of TB). This Actor is deliberately domain-level.

Limits

• 20 domains per run. Default cap of 6 GB graph download (configurable up to 50 GB).
• The domain level is the registered domain, as defined by Common Crawl's public-suffix handling.
• The ranks lookup reads the top 2 million ranks by default, so referrers ranked lower show no rank.
• The Common Crawl index server is rate-limited. We pace requests at about 1 per 1.5 s and back off on 503 responses.

Sources & compliance

• Data: Common Crawl web graphs (https://commoncrawl.org/web-graphs; release index https://index.commoncrawl.org/graphinfo.json), the CC URL index (https://index.commoncrawl.org/) and WARC files on https://data.commoncrawl.org/, all checked 2026-10-11.
• Terms: Common Crawl Terms of Use (https://commoncrawl.org/terms-of-use, last updated 2024-03-07, read 2026-10-11). The terms grant a limited licence to use the service, prohibit disrupting it and require you to respect third parties' rights in crawled content. The terms neither explicitly permit nor forbid commercial use; Common Crawl recommends legal advice for commercial use. This Actor outputs only link facts: domain names, ranks and short anchor texts. It does not output page content.
• Polite access: honest User-Agent SiteVerdict/backlink-explorer-commoncrawl (+aman.thakran@bharatfare.com). Requests are sequential streams, interrupted downloads resume with HTTP Range so nothing is fetched twice, and index queries are paced. Nothing is crawled on third-party sites.
• UK GDPR: domain-level link data, with no personal data collected.