# Semrush SEO Metrics Scraper (`axiomworks/semrush-domain-overview-scraper`) Actor

Scrape Semrush's public domain overview pages and return structured SEO metrics for each domain. Get authority score, backlinks, referring domains, estimated search traffic, top keywords, competitors, traffic by country and device, and six-month trends.

- **URL**: https://apify.com/axiomworks/semrush-domain-overview-scraper.md
- **Developed by:** [Axiom Works](https://apify.com/axiomworks) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.95 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Semrush Domain Overview Scraper

**Returns:** one row per domain with authorityScore, backlinks, referringDomains, organicTraffic, monthlyVisits, globalRank, trafficSources and aiTraffic.
**Input:** `{"domains": ["github.com", "bbc.com"], "maxItems": 10}`
**Best for:** competitor benchmarking, prospect list enrichment, monthly SEO and traffic monitoring.

```json
{
    "id": "bbc.com",
    "domain": "bbc.com",
    "authorityScore": 100,
    "referringDomains": 859231,
    "monthlyVisits": 354237748,
    "categoryRank": 9,
    "googlePenaltyRisk": "Low",
    "monthlyVisitsChangePercent": -0.1236
}
```

### What does Semrush Domain Overview Scraper do?

Semrush Domain Overview Scraper reads the public Semrush website overview page for each domain you supply and returns the numbers shown there as clean, structured records. You give it a list of domains or URLs, and for every domain it returns the authority score, backlinks, referring domains, estimated organic and paid search traffic, top organic keywords, organic competitors, traffic by country, the desktop and mobile split, and the six-month history of backlinks and traffic.

It is built for SEO teams, agencies, analysts and sales researchers who want a quick benchmark of many websites without opening each page by hand. Typical uses include comparing a client against competitors, enriching a list of prospect domains with authority and traffic figures, tracking how well-known sites change month to month, and feeding domain metrics into spreadsheets or dashboards.

The Actor only uses pages that anyone can open without signing in. It does not use the paid Semrush API and it does not log in to anything. Because of that, the data is the limited public view: a handful of top keywords, a few competitors and six months of history rather than the full lists you see in a paid account. Only well-known domains have a public overview page. For smaller sites Semrush returns no page, and the Actor skips them with a warning instead of producing an empty record.

Each record carries the exact page it came from in the `sourceUrl` field and a stable `id`, which is the normalized domain. Duplicated domains in your input are collapsed, so you never pay twice for the same one.

### What data can you get?

Every record contains the fields below. Fields the page does not show for a particular domain are returned as `null`.

| Field | Type | Description |
|---|---|---|
| `id` | string | Stable record id, the normalized domain |
| `domain` | string | Domain the overview describes |
| `sourceUrl` | string | Semrush overview page the data came from |
| `brandName` | string | Brand name shown by Semrush |
| `categories` | array | Category names Semrush assigns to the site |
| `globalRank` | integer | Global traffic rank |
| `countryRank` | integer | Traffic rank in the site's top country |
| `authorityScore` | integer | Authority Score from 0 to 100 |
| `backlinks` | integer | Total backlinks |
| `referringDomains` | integer | Number of referring domains |
| `organicTraffic` | integer | Estimated monthly organic search traffic |
| `paidTraffic` | integer | Estimated monthly paid search traffic |
| `monthlyVisits` | integer | Estimated monthly visits |
| `pagesPerVisit` | number | Average pages per visit |
| `timeOnSiteSeconds` | number | Average time on site in seconds |
| `bounceRate` | number | Bounce rate as a fraction between 0 and 1 |
| `backlinksHistory` | array | Monthly backlinks, newest first |
| `trafficHistory` | array | Monthly organic, paid and total traffic |
| `topKeywords` | array | Top organic keywords with position, search volume, CPC and traffic share |
| `competitors` | array | Organic competitors with name, relevance and visits |
| `trafficByCountry` | array | Top countries with traffic, share, desktop share and mobile share |
| `trafficByDevice` | object | Latest desktop and mobile visits and shares, plus monthly history |
| `dataDate` | string | Date of the Semrush data snapshot |
| `countryCode` | string | Country the Semrush data refers to |
| `categoryRank` | integer | Rank within the main category |
| `categoryRankName` | string | Category the category rank refers to |
| `googlePenaltyRisk` | string | Backlink Google penalty risk level shown by Semrush |
| `backlinksChangePercent` | number | Month-over-month backlinks change as a fraction |
| `referringDomainsChangePercent` | number | Month-over-month referring domains change as a fraction |
| `organicTrafficChangePercent` | number | Month-over-month organic traffic change as a fraction |
| `paidTrafficChangePercent` | number | Month-over-month paid traffic change as a fraction |
| `monthlyVisitsChangePercent` | number | Month-over-month visits change as a fraction |
| `referringDomainsHistory` | array | Monthly referring domain counts |
| `visitsHistory` | array | Monthly total visits |
| `trafficSources` | array | Where visitors come from, with host, source, channel, share, visits and previous visits |
| `outgoingTraffic` | array | Sites visitors go to next, with host, share and visits |
| `aiTraffic` | array | Visits from AI assistants such as ChatGPT and Claude, with share and history |
| `favicon` | string | Favicon URL listed by Semrush |

### How to use Semrush Domain Overview Scraper

1. Open the Actor in Apify Console and go to the Input tab.
2. Paste the domains you want into the Domains list. You can use bare domains such as `amazon.com` or full URLs such as `https://www.github.com/`. The `www.` prefix and any path are removed for you.
3. Set Max items to the number of records you want. The Actor stops as soon as that many records have been returned.
4. Leave the proxy setting on its default unless you have a reason to change it. Requests go out directly, and if the site answers with a block or rate limit, the Actor retries with backoff on a fresh Apify Proxy session. Challenge detection checks the page title and short visible text, ignoring scripts and styles. Retries for each target are bounded, and no backoff is added after the final attempt.
5. Click Start. When the run finishes, open the Output tab to browse the table, or download the data as JSON, CSV or Excel.

The Actor makes about one request per second, with two requests in flight at most, so a list of 50 domains takes roughly a minute. That pace is deliberate: it keeps the load on the target site small and makes blocks unlikely.

If a domain has no public overview page, the log shows a warning with the reason and the run carries on with the next domain. If none of your domains has data, or every supplied target is skipped as an unrecognized domain, the run finishes successfully with zero items and a status message suggesting well-known domains. Unexpected errors are caught separately for each domain, so other targets can still return data. If the input is not valid at all, for example an empty domain list, the run fails right away with a clear message.

### Input

| Field | Type | Required | Description |
|---|---|---|---|
| `domains` | array of strings | Yes | Domains or URLs to look up, for example `["amazon.com", "github.com"]` |
| `maxItems` | integer | No | Maximum number of records to return, from 1 to 100000. Default 50 |
| `proxyConfiguration` | object | No | Proxy settings. Direct by default, with an automatic Apify Proxy fallback when blocked |

Example input:

```json
{
    "domains": ["amazon.com", "github.com", "bbc.com"],
    "maxItems": 3
}
```

### Output

Each domain becomes one item in the dataset. A shortened example record:

```json
{
    "id": "github.com",
    "domain": "github.com",
    "sourceUrl": "https://www.semrush.com/website/github.com/overview/",
    "brandName": "GitHub",
    "globalRank": 28,
    "authorityScore": 100,
    "backlinks": 5335220083,
    "referringDomains": 4115198,
    "organicTraffic": 70689927,
    "paidTraffic": 24372,
    "monthlyVisits": 1212551945,
    "backlinksHistory": [
        { "displayDate": "2026-08-01", "backlinks": 5335220083 },
        { "displayDate": "2026-07-01", "backlinks": 7724668284 }
    ],
    "topKeywords": [
        { "keyword": "github", "intents": "2", "position": 1, "searchVolume": 1220000, "cpc": 3.29, "traffic": 0.0547 }
    ],
    "competitors": [
        { "name": "stackoverflow.com", "slug": "stackoverflow.com", "relevance": 0.23, "visits": 25915762 }
    ],
    "dataDate": "2026-08-15",
    "categoryRank": 1,
    "googlePenaltyRisk": "Low",
    "monthlyVisitsChangePercent": 0.0648,
    "trafficSources": [
        { "host": "Direct", "share": 0.619, "visits": 574779767, "previousVisits": 538970535 }
    ],
    "aiTraffic": [
        { "id": "chatgpt.com", "name": "ChatGPT", "visits": 7665006, "share": 0.00678 }
    ]
}
```

Fields added in the latest version (category rank, penalty risk, month-over-month changes, traffic sources, outgoing traffic and AI traffic) are `null` when the page does not show them for a domain.

The numbers are the estimates Semrush shows publicly. They are not exact counts, and they change every month.

### How much does it cost?

The Actor is charged per result: you pay for each domain record that is returned, and nothing for domains that have no data or for runs that fail. See the Pricing tab of the Actor page for the current rate. Apify's free plan includes monthly platform credit, which is enough to try the Actor on a few dozen domains.

Because charging is per result, Max items is also your spending cap. Set it to the number of domains you actually need.

### Use with AI agents (MCP)

You can call this Actor from any MCP-compatible assistant through the Apify MCP server. Add the following to your client configuration:

```json
{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com?tools=axiomworks/semrush-domain-overview-scraper"
        }
    }
}
```

Example prompts you can give your assistant:

- "Get the Semrush authority score and referring domains for stackoverflow.com and shopify.com."
- "Compare organic traffic for walmart.com and target.com and list their top keywords."
- "For these ten domains, return the competitors Semrush lists and the monthly backlinks history."

### Integrations and API

You can run the Actor from code with the Apify API clients.

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("axiomworks/semrush-domain-overview-scraper").call(
    run_input={"domains": ["amazon.com", "github.com"], "maxItems": 2}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["domain"], item["authorityScore"], item["backlinks"])
```

**JavaScript**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('axiomworks/semrush-domain-overview-scraper').call({
    domains: ['amazon.com', 'github.com'],
    maxItems: 2,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => process.stdout.write(`${item.domain} ${item.authorityScore}\n`));
```

**cURL**

```bash
curl -X POST "https://api.apify.com/v2/acts/axiomworks~semrush-domain-overview-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["amazon.com", "github.com"], "maxItems": 2}'
```

Apify also lets you schedule runs, send results to Google Sheets, Slack or a webhook, and connect the dataset to tools such as Zapier or Make. This is useful for a monthly refresh of a fixed list of domains.

### FAQ

**Why did a domain return no record?**
Semrush publishes overview pages only for sites with enough data. Smaller or very new sites return a not-found page. The Actor logs a warning and moves on, and you are not charged for it.

**Why are there only a few keywords and competitors?**
The public page shows a short preview: the top keywords, three competitors and six months of history. The full lists need a paid Semrush account, which this Actor does not use.

**How fresh is the data?**
Semrush updates the public pages about once a month. The `dataDate` field tells you the date of the snapshot a record was built from.

**Does it need a proxy?**
Usually not. Requests go out directly. If the site answers with a 403 or 429, the Actor retries with backoff and falls back to Apify Proxy on a fresh session.

**Can I pass URLs instead of domains?**
Yes. The Actor keeps only the host name, drops `www.` and removes duplicates.

**Does the Actor return the full backlink list?**
No. It returns the summary figures shown on the public overview page, such as total backlinks, referring domains and authority score.

**How many domains can I run at once?**
Up to the Max items limit. The Actor processes about one domain per second, so large lists simply take longer.

### Is it legal to scrape Semrush?

The Actor reads only pages that are open to everyone, with no login and no access to paid or private data. It makes a small number of polite requests and does not collect personal data. Whether scraping is appropriate in your case still depends on your jurisdiction and your use of the data. Website terms of service may restrict automated access, so review them and take legal advice where you need it. This Actor is an independent tool and is not affiliated with or endorsed by Semrush.

### Feedback

If a field is missing, a domain you expected is skipped or the page layout changes and the data stops parsing, open an issue from the Actor page's Issues tab with the domain and the date of the run. Bug reports with a concrete example are fixed fastest.

# Actor input Schema

## `domains` (type: `array`):

Required. List of website domains or full URLs, e.g. \['amazon.com', 'github.com', 'https://www.bbc.com/news']. Protocol, www and paths are stripped. Only well-known sites have a public Semrush overview; others return no record and the run status says which were skipped. Duplicates are collapsed.

## `maxItems` (type: `integer`):

Maximum number of domain records to return, one per domain: integer 1-100000, default 50, e.g. 3. The Actor stops after this many records.

## `proxyConfiguration` (type: `object`):

Optional. Proxy settings object, e.g. {"useApifyProxy": true}. Default is direct requests; on a 403/429 block the Actor automatically retries through Apify Proxy, so leave unchanged unless runs are blocked.

## Actor input object example

```json
{
  "domains": [
    "amazon.com",
    "github.com",
    "bbc.com"
  ],
  "maxItems": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "amazon.com",
        "github.com",
        "bbc.com"
    ],
    "maxItems": 3,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("axiomworks/semrush-domain-overview-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "amazon.com",
        "github.com",
        "bbc.com",
    ],
    "maxItems": 3,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("axiomworks/semrush-domain-overview-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "amazon.com",
    "github.com",
    "bbc.com"
  ],
  "maxItems": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call axiomworks/semrush-domain-overview-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,axiomworks/semrush-domain-overview-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WDJOBWypXCY5mxRg7/builds/MFzaPd8RGdZeVxdQA/openapi.json