# Backlink & Referring Domains Checker · SiteVerdict (`siteverdict/siteverdict-backlink-explorer-commoncrawl`) Actor

Find the domains that link to any website using Common Crawl's open web graph (billions of pages): referring domains with authority rank, TLD mix and verified homepage anchors with first-seen crawl. No scraping, honest coverage labels.

- **URL**: https://apify.com/siteverdict/siteverdict-backlink-explorer-commoncrawl.md
- **Developed by:** [Aman Thakran](https://apify.com/siteverdict) (community)
- **Categories:** SEO tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 80.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SiteVerdict — Backlink Explorer (Common Crawl)

**See which websites link to any domain, using the open web graph built by Common Crawl from billions of crawled pages. No scraping and no paid SEO-tool subscription.**

Enter up to 20 domains (yours, your competitors' or your prospects'). For each one you get:

- **every referring domain** in Common Crawl's latest domain-level web graph (each release covers three monthly crawls);
- an **authority rank** for each referrer (Common Crawl's harmonic-centrality rank), its size (hosts crawled) and its domain suffix;
- for the top referrers, **verified homepage links** from Common Crawl's own page archive: anchor text, nofollow/ugc/sponsored, and the **first crawl** in the release where the link appears;
- a **summary** with the total count, the suffix mix (.com/.co.uk/.edu…), the top 20 referrers and plain coverage notes.

### Who it is for

- **SEO and digital PR teams:** compare referring-domain profiles against competitors, and find sites that link to them but not to you.
- **Agencies:** do cheap first-pass backlink discovery for prospects and audits without a subscription.
- **Researchers:** run reproducible link-graph lookups on open data.

### How it works (and why it is honest)

1. Common Crawl publishes a domain-level web graph for each release. It is a compressed WebGraph (BVGraph) file of about 133 million domains and 2 billion links, and it includes the **transposed** graph, which holds the links pointing *to* each domain.
2. The Actor finds your domain's node id, then streams the transposed graph from the start and decodes it until it reaches your domain. Common Crawl does not publish an offsets file, so the scan always starts at the beginning. It resolves the referring ids to names and, if enabled, adds authority ranks.
3. For the top N referrers it looks up their homepage captures in the Common Crawl URL index and reads only those WARC records by byte range, to show real anchors.

All of this uses Common Crawl's public files. Nothing is crawled live.

#### Coverage: please read

- Common Crawl **samples** the web; it does not crawl every page. Treat counts as a **lower bound** of real referring domains. They are typically far below what commercial crawlers (Ahrefs, Majestic, Semrush) report, but they come from open data and can be reproduced.
- Links are counted at **domain level**: a domain counts if any of its crawled pages links to any host of your domain. Page-level URLs, anchors for every link, and nofollow status are **not** in the graph. Anchors are verified for **homepage links only**.
- The data is as fresh as the release (each release covers about three months of crawls).

### Speed and cost

Domains sort by reversed name, so the TLD decides how far into the graph file the Actor must read: `au.*` sits about 2% in, `com.*` roughly 10–50% (for example bharatfare.com 18%, siteverdict.com 49%) and `org.*` about 80%. Reading further means more GB and more minutes.

- **Batch your domains.** Up to 20 targets share one pass, at no extra download.
- **Run time depends on where your domain sorts.** Measured 15 Oct 2026 on the Jul–Sep 2026 release (4 GB memory):
  - **Pre-filled example / daily test:** `aph.gov.au` (Parliament of Australia, 2% into the graph), 100 rows and 2 homepage checks: about 1–2 minutes, with 11,000+ referring domains found.
  - `abc.net.au` with 200 rows and 3 homepage checks: about 6.5 minutes in the cloud before the 15 Oct speed fixes.
  - **Extended performance test (not part of the daily test):** `python.org` (84% into the graph) needs roughly 15–20 minutes; deep `.com` targets can take 10+ minutes. Set the run timeout to 30 minutes or more for these.
  - Homepage link checks (`anchorChecks`) each take several seconds because they query Common Crawl's archive politely; keep the number low for fast runs.
- **Run time limit:** the Actor watches its own run timeout. If the graph scan cannot reach a domain in time, it stops cleanly, saves results for any domain it did reach, lists the others with a "Skipped (run time limit)" note and an estimate of the timeout needed, and does not charge for them (the graph-scan event is charged only when at least one domain is reached). The status message then starts with "Partial (run time limit)". Keep the run timeout at 30 minutes or more for `.com`/`.org` targets.
- **Stalled downloads:** if Common Crawl sends no data for 60 seconds, the download is cut and resumed from the last byte received (up to 5 times). Other requests are limited to 60 seconds each, so a slow server cannot hang the run.
- **Homepage checks when Common Crawl's archive index is down:** each archive request is limited to 15 seconds, all homepage checks together to 2 minutes (and never past your run timeout), and after 3 index failures in a row the remaining checks are skipped. Rows whose check could not be completed show `homepageLink.status` = `INDEX_UNAVAILABLE` (with `found: null`, which does **not** mean "no link"), are not charged for the check, and the target summary says how many. The referring-domain results are unaffected.

| Event | Price |
|---|---|
| `graph-scan`: one shared pass per run | **$0.75** |
| `target-analysed`: per domain | $0.05 |
| `referring-domain`: per row output | $0.0005 |
| `anchor-check`: per referrer homepage verified | $0.005 |

Example: 5 competitors, 1,000 referring domains each, 10 anchor checks each, costs $0.75 + $0.25 + $2.50 + $0.25 = **$3.75** in one run.

Sample mode is free. Domains that are not in the graph are free, and so are targets skipped by the size cap. The $0.75 graph scan is only charged if your max total charge can also pay for at least one analysed domain; any domain the budget cannot cover is listed as "Skipped (maximum total charge reached)" and is not charged. `graph-scan` and `target-analysed` are charged only after the domain's first referring-domain row has been saved, so a run that stops or times out before writing a row costs nothing, and a domain with no referring domains is free. Unresolved `#id` rows and homepage checks that could not be completed are free.

### Output

- **Referring domains** view: one row per referrer, with authority rank, hosts, suffix, and homepage-link verification (anchors, nofollow, first-seen crawl).
- **Target summaries** view: totals, suffix mix, top referrers and coverage notes.
- **`SUMMARY`** in the key-value store: release, GB downloaded, duration and events charged.

Run with **Mode = Sample** to see the full pipeline on a tiny synthetic graph in Common Crawl's exact formats.

### FAQ

**Why not just call an API?** Common Crawl has no "backlinks" API. The graph files are the official way to use the link data, and this Actor does the heavy decoding for you.

**Is it accurate?** The decoder implements the WebGraph BVGraph format: gamma, zeta and unary codes, reference copy blocks, intervals and residuals. It is checked against Common Crawl's real files. Decoding the first 200,000 nodes of both the forward and the transposed graph must give exactly mirror-image link sets. Use `scripts/verify-graph.ts` to rerun that check.

**Can I get page-level backlinks?** No. That would mean processing the WAT files (hundreds of TB). This Actor is deliberately domain-level.

### Limits

- 20 domains per run. Default cap of 6 GB graph download (configurable up to 50 GB).
- The domain level is the registered domain, as defined by Common Crawl's public-suffix handling.
- The ranks lookup reads the top 2 million ranks by default, so referrers ranked lower show no rank.
- The Common Crawl index server is rate-limited. We pace requests at about 1 per 1.5 s and back off on 503 responses.

### Sources & compliance

- **Data:** Common Crawl web graphs (https://commoncrawl.org/web-graphs; release index https://index.commoncrawl.org/graphinfo.json), the CC URL index (https://index.commoncrawl.org/) and WARC files on https://data.commoncrawl.org/, all checked 2026-10-11.
- **Terms:** Common Crawl Terms of Use (https://commoncrawl.org/terms-of-use, last updated 2024-03-07, read 2026-10-11). The terms grant a limited licence to use the service, prohibit disrupting it and require you to respect third parties' rights in crawled content. The terms neither explicitly permit nor forbid commercial use; Common Crawl recommends legal advice for commercial use. This Actor outputs only link facts: domain names, ranks and short anchor texts. It does not output page content.
- **Polite access:** honest User-Agent `SiteVerdict/backlink-explorer-commoncrawl (+aman.thakran@bharatfare.com)`. Requests are sequential streams, interrupted downloads resume with HTTP Range so nothing is fetched twice, and index queries are paced. Nothing is crawled on third-party sites.
- **UK GDPR:** domain-level link data, with no personal data collected.

# Changelog

This Actor's version history is a separate document: https://apify.com/siteverdict/siteverdict-backlink-explorer-commoncrawl/changelog.md

# Actor input Schema

## `executionMode` (type: `string`):

<b>Live</b> scans Common Crawl's real web graph. <b>Sample</b> runs the full pipeline on a tiny synthetic graph in Common Crawl's exact file formats (free, no network).

## `domains` (type: `array`):

Registered domains to find links to (subdomains and URLs are reduced to the registered domain, e.g. blog.example.co.uk → example.co.uk). Up to 20 per run; they share one pass over the graph, so batching is much cheaper.

## `release` (type: `string`):

'latest' or a release id such as cc-main-2026-jul-aug-sep (see https://index.commoncrawl.org/graphinfo.json). Each release covers three monthly crawls.

## `maxReferringDomains` (type: `integer`):

Rows output per target (the summary always has the full count). Billed per row.

## `sortBy` (type: `string`):

Authority (harmonic-centrality rank from Common Crawl), size (hosts crawled) or name.

## `rankReferrers` (type: `boolean`):

Look up Common Crawl's harmonic-centrality rank for referring domains (reads the top of the ranks file only).

## `rankMaxLines` (type: `integer`):

How many top-ranked domains to read from the ranks file; referring domains below this rank show no rank.

## `anchorChecks` (type: `integer`):

For the top N referring domains, fetch their homepage captures from Common Crawl's archive (CDX index + WARC record, no live crawling) to show anchor text, nofollow and the first crawl in this release that contains the link. 0 = off. Billed per domain checked.

## `maxGraphGB` (type: `integer`):

Safety cap. The graph is read from the start up to the target's position (domains sort by reversed name, e.g. com.\* sits around 60-80% of the file); a target needing more than this is skipped (not charged).

## `maxNameLookupMB` (type: `integer`):

Cap on the vertex-list download used to turn referring ids into names; ids beyond it are shown as #id.

## Actor input object example

```json
{
  "executionMode": "live",
  "domains": [
    "aph.gov.au"
  ],
  "release": "latest",
  "maxReferringDomains": 100,
  "sortBy": "rank",
  "rankReferrers": true,
  "rankMaxLines": 2000000,
  "anchorChecks": 2,
  "maxGraphGB": 6,
  "maxNameLookupMB": 0
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `files` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "aph.gov.au"
    ],
    "maxReferringDomains": 100,
    "anchorChecks": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("siteverdict/siteverdict-backlink-explorer-commoncrawl").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": ["aph.gov.au"],
    "maxReferringDomains": 100,
    "anchorChecks": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("siteverdict/siteverdict-backlink-explorer-commoncrawl").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "aph.gov.au"
  ],
  "maxReferringDomains": 100,
  "anchorChecks": 2
}' |
apify call siteverdict/siteverdict-backlink-explorer-commoncrawl --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,siteverdict/siteverdict-backlink-explorer-commoncrawl"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ZC66yB3i0v2YMHLqK/builds/nQHMZK4IHqP8Sg290/openapi.json