# Semrush Scraper - Traffic, Authority, Backlinks & Keywords (`scrapesage/semrush-scraper`) Actor

Scrape Semrush public website overviews: monthly visits, Authority Score, global/country/category rank, organic vs paid traffic, backlinks, referring domains, top keywords with CPC, competitors, traffic by country, device split and AI-assistant referrals.

- **URL**: https://apify.com/scrapesage/semrush-scraper.md
- **Developed by:** [Scrape Sage](https://apify.com/scrapesage) (community)
- **Categories:** SEO tools, Agents, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$6.00 / 1,000 website overview scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Semrush Scraper - Traffic, Authority Score, Backlinks & Keywords

Turn any list of domains into a clean competitive-intelligence table. This actor reads Semrush's **public website-overview pages** and returns the whole page as structured data - monthly visits, Authority Score, global/country/category rank, organic vs paid search traffic, backlink profile, top organic keywords with CPC, competitors, per-country traffic with device splits, and **AI-assistant referral traffic**.

**No Semrush login. No API key. No browser.** Keyless HTTP against pages Semrush publishes for search engines.

### Why this Semrush scraper?

| Data | Typical Semrush scrapers | This actor |
|---|---|---|
| Monthly visits + change % | ✅ | ✅ |
| Authority Score | ✅ | ✅ |
| Global / country rank | partial | ✅ + trend (up/down/flat) |
| Category rank + category name | ❌ | ✅ |
| Pages per visit, time on site, bounce rate | partial | ✅ |
| **Organic vs paid** search traffic, each with change % | ❌ | ✅ |
| **Backlinks + referring domains**, each with change % | ❌ | ✅ |
| **6-month history** — visits, organic/paid, backlinks, referring domains | ❌ | ✅ |
| **Top organic keywords** with position, volume, **CPC**, traffic share | ❌ | ✅ |
| Organic competitors with relevance + their visits | partial | ✅ |
| Traffic by country with **desktop/mobile share per country** | ❌ | ✅ |
| Desktop vs mobile visits + 6-month device history | ❌ | ✅ |
| **AI-assistant referral traffic** (ChatGPT, Perplexity…) | ❌ | ✅ |
| Referrals in / out (who sends traffic here, where it goes) | ❌ | ✅ |
| Google penalty risk | ❌ | ✅ |

**53 fields per domain.** Uncovered domains are reported honestly and **never charged**.

### Use cases

- **Competitor benchmarking** - drop in 50 competitor domains, get one comparable table.
- **Lead qualification** - score prospects by real traffic, Authority Score and backlink profile before you pitch.
- **SEO due diligence** - check a site's organic trend and backlink growth before an acquisition.
- **Agency reporting** - refresh a client + competitor set on a schedule and push to Sheets.
- **AI-visibility tracking** - see how much traffic ChatGPT and friends actually send a domain.

### How to use

1. Add your domains under **Domains** (one per line) - or paste/link a whole list under **Import domains from a file**.
2. Leave everything else at its default and hit **Start**.
3. Export as JSON, CSV or Excel, or pull it from the API.

### Input

```json
{
    "domains": ["booking.com", "stripe.com"],
    "includeKeywords": true,
    "includeCompetitors": true,
    "includeAiTraffic": true,
    "outputFields": [],
    "maxItems": 100
}
```

- **domains** - domains to analyse. Full URLs and Semrush overview links are accepted and reduced to the registrable domain.
- **startUrls** - paste Semrush overview URLs directly. Also accepts the Console's **Link remote text file** option.
- **urlsFromFile** - bulk import: paste the whole list, or give **one link** to a `.txt`/`.csv`, Google Sheet, Drive share or key-value-store record. Blank lines, `#` comments, CRLF, a UTF-8 BOM, quotes and extra CSV columns all parse; duplicates are analysed once.
- **includeHistory / includeKeywords / includeCompetitors / includeCountries / includeAiTraffic / includeDevices / includeReferrals** - toggle each section. All come from the same page load, so turning one off makes the record leaner, not cheaper.
- **outputFields** *(pick-list)* - tick the fields you want and every record is trimmed to exactly those. Nothing to memorise - the dropdown lists all 53 with plain-English labels and their exact key.
- **maxItems** - cap per run.
- **proxyConfiguration** - Apify datacenter by default; Semrush serves these pages cleanly to it.

### Output

One record per domain (`type: "website"`):

```json
{
    "type": "website",
    "domain": "stripe.com",
    "url": "https://www.semrush.com/website/stripe.com/overview/",
    "country": "US",
    "categories": ["Finance"],
    "authorityScore": 79,
    "googlePenaltyRisk": "Low",
    "globalRank": 163,
    "globalRankTrend": "up",
    "countryRank": 164,
    "countryRankName": "United States",
    "categoryRank": 6,
    "categoryRankName": "Finance",
    "visits": 179937086,
    "visitsChangePercent": -0.0757,
    "pagesPerVisit": 2.7837,
    "timeOnSiteSec": 484,
    "bounceRate": 0.6082,
    "organicTraffic": 96442310,
    "organicChangePercent": 0.0312,
    "paidTraffic": 1204882,
    "backlinks": 224518903,
    "backlinksChangePercent": 0.0141,
    "referringDomains": 633826,
    "desktopVisits": 70856331,
    "mobileVisits": 109080755,
    "desktopShare": 0.3938,
    "mobileShare": 0.6062,
    "topKeywords": [
        { "keyword": "stripe", "position": 1, "searchVolume": 1830000, "cpc": 3.11, "trafficShare": 0.0421 }
    ],
    "competitors": [{ "domain": "gocardless.com", "relevance": 0.58, "visits": 3204118 }],
    "trafficByCountry": [
        { "country": "US", "countryName": "United States", "traffic": 52207086, "trafficShare": 0.2901, "desktopShare": 0.3938, "mobileShare": 0.6062 }
    ],
    "aiTraffic": [{ "source": "ChatGPT", "traffic": 227075, "share": 0.0005 }],
    "referralsIn": [{ "host": "github.com", "share": 0.11, "visits": 402118 }],
    "visitsHistory": [{ "date": "2026-06-01", "visits": 179937086 }],
    "dataDate": "2026-06-15",
    "scrapedAt": "2026-08-09T19:44:00.000Z"
}
```

#### What to expect (field coverage)

Measured over a deliberately varied sample (mega-sites, SaaS, retail, news, a small niche tool):

| Field | Coverage |
|---|---|
| `domain`, `visits`, `authorityScore`, `globalRank`, `countryRank`, `bounceRate`, `pagesPerVisit`, `timeOnSiteSec` | 100% |
| `organicTraffic`, `paidTraffic`, `backlinks`, `referringDomains` + their change % | 100% |
| `topKeywords`, `trafficByCountry`, `aiTraffic`, `referralsIn`/`referralsOut`, all history series | 100% |
| `competitors` | ~85% - Semrush omits the set when too few organic keywords overlap |
| `paidChangePercent` | ~85% - no delta exists when the domain ran no paid search last month |
| `categoryRank`, `categoryRankName` | ~57% - only for domains Semrush has classified |
| `brandName`, `favicon` | ~57% - only where Semrush holds a brand entity |

**Not available, and we say so rather than shipping an empty column:** Site Audit metrics (site health, crawled pages, thematic reports) require a logged-in Semrush project - the public page ships them as zeros, so they are deliberately excluded from the record.

**Domains Semrush does not cover** (below its traffic threshold) end the run with a clear message naming them, and are **never charged**.

### Automate & schedule

Pair with Apify **Schedules** to refresh a competitor set weekly and push straight into Google Sheets, Slack or your warehouse via Apify integrations.

### Use with AI assistants (MCP)

Expose this actor as a tool to Claude, ChatGPT or any MCP client via the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp) and ask for competitive data in plain language.

### Agent-ready: autonomous payments (x402 & Skyfire)

This actor is **agent-ready** — AI agents can discover it, run it, and **pay for it autonomously**, with no Apify account and no human in the loop. It uses [pay-per-event](https://docs.apify.com/platform/actors/publishing/monetize/pay-per-event) pricing and [limited permissions](https://docs.apify.com/platform/actors/development/permissions), so it qualifies for Apify's agentic-payment standards:

- **[x402](https://docs.apify.com/platform/integrations/x402)** — an open, HTTP-native payment protocol. Agents pay per run in USDC on the Base network directly through the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp) — no account, no API key.
- **[Skyfire](https://docs.apify.com/platform/integrations/skyfire)** — agent-to-service payments for fully autonomous AI-agent workflows.

Building an AI agent, MCP tool, or autonomous data pipeline? This scraper is ready to plug in and pay as it goes.

### Pricing

Pay per event: **$0.006 per website overview**. Domains Semrush does not cover cost nothing.

### FAQ

**Does this need a Semrush subscription?** No. It reads only the public overview pages Semrush publishes for search engines.

**Why is a domain "not covered"?** Semrush publishes an overview only above a traffic threshold. Use the registrable domain (`example.com`, not a sub-path), and expect very small or brand-new sites to be absent.

**Are the numbers Semrush's own?** Yes - they are Semrush's published estimates, read from the page as-is. No modelling or re-estimation.

**Can I import a big list?** Yes - paste it, link it, or use the Start-URLs *Link remote text file* option. A file that cannot be read says so and charges nothing.

### Need help?

Open an issue on the actor's **Issues** tab - feature requests genuinely do get built.

***

*Not affiliated with, endorsed by, or sponsored by Semrush. "Semrush" is a trademark of its respective owner and is used here only to describe what this tool reads. Scrapes public pages only.*

# Actor input Schema

## `domains` (type: `array`):

Domains to analyse, one per line - e.g. <code>booking.com</code>, <code>stripe.com</code>, <code>bbc.co.uk</code>. Full URLs and Semrush overview links are accepted too and reduced to the registrable domain. Semrush only publishes an overview for domains above a traffic threshold; anything below it is reported as <i>not covered</i> and is never charged.

## `startUrls` (type: `array`):

Paste Semrush overview URLs directly, e.g. <code>https://www.semrush.com/website/booking.com/overview/</code>. <b>Bulk import:</b> use this field's <i>Link remote text file</i> option to load a hosted list (one entry per line) - or paste the list into <b>Import domains from a file</b> below.

## `urlsFromFile` (type: `string`):

Bulk-load domains without adding them one at a time. Two ways to use it: <b>(1)</b> paste your whole list straight in, one domain per line - open your .txt/.csv, select all, paste; or <b>(2)</b> give a single link to the file and it is downloaded for you - a public <code>.txt</code>/<code>.csv</code>, a Google Sheet or Drive share link (auto-converted to CSV export), or an Apify key-value-store record. Blank lines, <code>#</code> comments and extra CSV columns are ignored, and duplicates are only analysed once. If the file cannot be read the run says so and charges nothing.

## `includeHistory` (type: `boolean`):

Add the month-by-month series: visits, organic vs paid search traffic, backlinks and referring domains (up to 6 months each).

## `includeKeywords` (type: `boolean`):

Add the domain's top organic keywords with position, monthly search volume, CPC and share of traffic.

## `includeCompetitors` (type: `boolean`):

Add the competing domains Semrush ranks against this one, with a relevance score and their monthly visits.

## `includeCountries` (type: `boolean`):

Add the top countries sending traffic, each with its traffic share and desktop/mobile split.

## `includeAiTraffic` (type: `boolean`):

Add referral traffic coming from AI assistants (ChatGPT, Perplexity and friends) with volume and share - the newest signal on the page and the one most competitors do not export.

## `includeDevices` (type: `boolean`):

Add the desktop vs mobile visit counts and their month-by-month history.

## `includeReferrals` (type: `boolean`):

Add the sites sending traffic TO this domain and the sites it sends traffic to, with share and volume.

## `outputFields` (type: `array`):

Pick the fields you want and every record is trimmed to exactly those - handy for lean CSV/Sheets exports. <b>Nothing to memorise:</b> open the dropdown and tick, or start typing (<code>traffic</code>, <code>rank</code>, <code>backlink</code>) to filter. Leave empty to get every field.

## `maxItems` (type: `integer`):

Maximum number of domains to analyse in one run. Pagination stops automatically when this is reached.

## `proxyConfiguration` (type: `object`):

Proxy settings. Semrush serves these public pages cleanly to Apify datacenter proxies, which are used by default. If you ever see blocks, switch to Residential.

## Actor input object example

```json
{
  "domains": [
    "booking.com",
    "stripe.com"
  ],
  "includeHistory": true,
  "includeKeywords": true,
  "includeCompetitors": true,
  "includeCountries": true,
  "includeAiTraffic": true,
  "includeDevices": true,
  "includeReferrals": true,
  "maxItems": 100,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All analysed domains as JSON items in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "booking.com",
        "stripe.com"
    ],
    "urlsFromFile": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapesage/semrush-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "booking.com",
        "stripe.com",
    ],
    "urlsFromFile": "",
}

# Run the Actor and wait for it to finish
run = client.actor("scrapesage/semrush-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "booking.com",
    "stripe.com"
  ],
  "urlsFromFile": ""
}' |
apify call scrapesage/semrush-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapesage/semrush-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aCSugNNd63ANKUgTq/builds/UNjaOFf7GhaRvY4EZ/openapi.json