# Semrush Scraper (`mlg14/semrush-scraper`) Actor

Extract public domain overview metrics and top website rankings, including visits, search traffic, backlinks, competitors, and traffic trends.

- **URL**: https://apify.com/mlg14/semrush-scraper.md
- **Developed by:** [MLG Data](https://apify.com/mlg14) (community)
- **Categories:** SEO tools, Automation
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Semrush Scraper

Scrape Semrush public domain overviews and top website rankings into a structured dataset. Export Semrush data to CSV, JSON, or Excel for traffic benchmarking, backlink checks, competitor research, and ranking reports. This Actor offers a practical Semrush API alternative for the metrics visible on public website pages; it does not require a site account.

A domain overview returns one record per domain. A top websites run returns one record per ranked website in a country and industry. Choose `both` to collect overview records first and then add the selected ranking table to the same dataset. Every record includes `recordType`, `domain`, `sourceUrl`, and `dataDate`, so mixed exports remain easy to filter.

### What data can you extract from Semrush?

The domain overview contains public traffic, search, backlink, engagement, and competitor samples. Ranking records contain visits, device share, growth, and the leading traffic source. The site publishes estimates; treat the numbers as directional measures rather than first-party analytics.

| Field | Description | Example |
| --- | --- | --- |
| `recordType` | Identifies a `domain` or `ranking` row. | `domain` |
| `domain` | Normalized website domain. | `archive.org` |
| `sourceUrl` | Public source page. | `https://www.semrush.com/website/archive.org/overview/` |
| `dataDate` | Date attached to the source metrics. | `2026-08-15` |
| `updatedAt` | Date the overview was updated. | `2026-09-18` |
| `brandName` | Public website label. | `Internet Archive` |
| `country` | Country attached to the overview summary. | `US` |
| `categories` | Public category labels. | `Online Services` |
| `globalRank` | Worldwide rank for a domain. | `147` |
| `countryRank` | Rank within the overview country. | `136` |
| `categoryRank` | Rank within a category, when shown. | `6` |
| `authorityScore` | Public 0–100 authority score. | `91` |
| `backlinks` | Estimated number of backlinks. | `1887667482` |
| `referringDomains` | Estimated distinct referring domains. | `1269407` |
| `googlePenaltyRisk` | Public risk label. | `Low` |
| `organicTraffic` | Estimated organic search traffic. | `22553842` |
| `paidTraffic` | Estimated paid search traffic. | `0` |
| `totalSearchTraffic` | Organic plus paid search traffic. | `22553842` |
| `monthlyVisits` | Estimated visits for the latest available month. | `219290927` |
| `monthlyVisitsChangePercent` | Change in overview visits, in percent. | `-1.72` |
| `desktopVisits` | Estimated desktop visits. | `102636143` |
| `mobileVisits` | Estimated mobile visits. | `116654784` |
| `pagesPerVisit` | Mean pages viewed per visit. | `7.4943` |
| `visitDurationSeconds` | Mean visit duration in seconds. | `622` |
| `bounceRatePercent` | Share of visits that bounce, in percent. | `47.7` |
| `topCountry` | First country in the public traffic table. | `US` |
| `topCountryTraffic` | Visits estimated for that country. | `61576421` |
| `topOrganicKeywords` | Public sample of organic keywords, positions, search volume, CPC, and traffic share. | Array of keyword objects |
| `organicCompetitors` | Public sample of competing domains, relevance, and visits. | Array of domain objects |
| `trafficByCountry` | Country-level visit and device mix estimates. | Array of country objects |
| `searchTrafficHistory` | Historical organic and paid search traffic. | Array of dated points |
| `visitHistory` | Historical monthly visits. | Array of dated points |
| `backlinksHistory` | Historical backlink totals. | Array of dated points |
| `referringDomainsHistory` | Historical referring domain totals. | Array of dated points |
| `assistantReferrals` | Public referral source estimates and histories. | Array of source objects |
| `rank` | Position in the chosen top websites table. | `1` |
| `rankingCountry` | Country slug used for the ranking. | `global` |
| `rankingIndustry` | Industry slug used for the ranking. | `all` |
| `desktopSharePercent` | Percentage of ranking visits from desktop. | `31.22` |
| `mobileSharePercent` | Percentage of ranking visits from mobile. | `68.78` |
| `momChangePercent` | Month-over-month visit change in percent. | `-0.12` |
| `yoyChangePercent` | Year-over-year visit change in percent. | `0.71` |
| `topTrafficSource` | Largest traffic source category. | `direct` |

Fields belong to their respective record types. A domain row has overview fields and public sample arrays; a ranking row has `rank`, ranking filters, device shares, growth measures, and traffic source. Common fields make it possible to combine both in one export without confusing the two measurements. A blank value means the public page omitted that measure. The Actor never fills missing fields with guessed values.

### How to scrape Semrush

1. Set `mode` to `domain`, `topWebsites`, or `both`.
2. For domain overviews, paste one or more website domains or full URLs into `domains`. The Actor uses each URL's host and removes a leading `www.`.
3. For top websites, choose `rankingCountry` and `rankingIndustry`. Use the slugs in public ranking page URLs, such as `global` and `all`.
4. Set `maxItems` if you need a strict output cap, then start the Actor. The dataset is available in JSON, CSV, and spreadsheet formats when the run finishes.

The `both` mode processes the requested domains before the ranking table. Set `maxItems` above the number of requested domains if you want ranking rows as well. If your cap is reached during the domain phase, the ranking page is not requested.

### Input

| Name | Type | Default | Description |
| --- | --- | --- | --- |
| `mode` | String | `domain` | Select domain overviews, top websites, or both. |
| `domains` | String array | Example prefill | Domains or URLs for `domain` and `both`. Maximum 100 entries. Duplicate normalized domains are collected once. |
| `rankingCountry` | String | `global` | Country slug used in the public top websites URL. |
| `rankingIndustry` | String | `all` | Industry slug used in the public top websites URL. |
| `maxItems` | Integer | `200` | Maximum total records. Set `0` for no Actor cap. |
| `proxyConfiguration` | Object | Default proxy | Optional proxy configuration for public page requests. |

A realistic input for a combined report is:

```json
{
  "mode": "both",
  "domains": ["wikipedia.org", "archive.org"],
  "rankingCountry": "global",
  "rankingIndustry": "all",
  "maxItems": 200
}
```

For a ranking-only run, set `mode` to `topWebsites` and leave `domains` empty. For a domain-only run, the ranking filters have no effect. Enter an industry or country slug exactly as it appears in a public ranking URL; the Actor checks that the page returned the requested filters and stops if it did not.

### Output example

These selected fields come from the `archive.org` domain record in a successful 94-record golden run on September 27, 2026. The full item also includes keyword, competitor, country, history, and referral arrays.

```json
{
  "recordType": "domain",
  "domain": "archive.org",
  "sourceUrl": "https://www.semrush.com/website/archive.org/overview/",
  "dataDate": "2026-08-15",
  "updatedAt": "2026-09-18",
  "brandName": "Internet Archive",
  "country": "US",
  "categories": ["Online Services"],
  "globalRank": 147,
  "countryRank": 136,
  "authorityScore": 91,
  "backlinks": 1887667482,
  "referringDomains": 1269407,
  "organicTraffic": 22553842,
  "paidTraffic": 0,
  "totalSearchTraffic": 22553842,
  "monthlyVisits": 219290927,
  "desktopVisits": 102636143,
  "mobileVisits": 116654784,
  "pagesPerVisit": 7.4943,
  "visitDurationSeconds": 622,
  "topCountry": "US",
  "topCountryTraffic": 61576421
}
```

The same run returned two domain records and 92 ranking records for `global/all`. The count is the number of entries exposed on that public page at run time; it is not a fixed promise of 100 records. Ranking pages can expose a different number of rows for another country or industry.

### Use cases

- **Competitive benchmarking:** Compare monthly visits, organic search traffic, paid search traffic, authority, and referring domains for a short list of sites. Keep source dates beside each metric so reports do not mix periods silently.
- **Prospect review:** Screen domains for traffic scale, backlink profile, and engagement before a manual review. The public overview offers a quick first pass when a full private research account is unnecessary.
- **Market landscape reports:** Export the top websites for a country and industry, then sort by month-over-month or year-over-year visit change to find growing sites.
- **Channel planning:** Compare the leading traffic source in rankings with the search traffic estimates on a domain overview. The measures come from different public sections, so retain `recordType` and `dataDate` when joining them.
- **Periodic snapshots:** Schedule repeated runs to capture changes in public authority, visits, traffic mix, and rankings. Store each run's date because the site's underlying month may update on a different schedule from your run.
- **Content research:** Inspect the small public keyword and competitor samples attached to domain overviews. These samples identify topics worth checking manually; they are not complete keyword exports.

### How much does it cost to scrape Semrush?

The current result event price is **$0.001 per record, or $1 per 1,000 records**. The charge follows delivered dataset items, and the Actor's platform compute is included in that event price for the user. A combined run that returns two domain rows and 92 ranking rows delivers 94 records, so its result charge is **$0.094**. A batch of 50 successful domain overviews would cost **$0.05** in result events. Ten monthly ranking snapshots with 90 rows each would deliver 900 records and cost **$0.90** in result events.

The number of public ranking entries varies. A domain that has no public overview does not produce a result event. `maxItems` prevents additional result events after the chosen cap. Account-level billing and the current event price should be checked on the Actor's pricing panel before a larger run. The examples above describe result events and do not promise a particular site response or a fixed ranking size.

### Tips for best results

Use a plain domain for overviews, such as `wikipedia.org`. Full URLs also work; the path and query are removed because the overview describes the domain. Keep domain batches focused if you need to investigate a failed site individually. Invalid domains are rejected before requests begin, and repeated domains in the same input are deduplicated.

For rankings, copy the country and industry slugs from a public top websites page. `global/all` is a broad starting point. An industry table in a specific country can be more useful when the goal is to compare direct peers. The Actor reads the table's embedded state from one page; there is no page-by-page pagination of deeper rankings. Use separate runs for separate country and industry combinations.

Use `both` when you want a compact benchmark: collect several domain overviews, then add one ranking table. Filter the exported dataset by `recordType` before comparing columns. `monthlyVisits` appears in both row types, but domain and ranking pages may have different dates or estimation methods. Preserve `dataDate` and `sourceUrl` in downstream reports.

### Limits

The Actor uses publicly available website pages. A public domain overview shows a limited keyword sample and a limited competitor sample; it does not expose the full private research dataset. The tested overview showed five organic keywords and three competitors for each sampled domain. Public backlinks are aggregate counts and short histories, not a list of every linking page. The public page also does not provide the leader's paid keyword list, ad copy, full keyword difficulty research, or complete country-specific domain dashboard.

Anonymous calls to the keyword research endpoint returned an operation-forbidden response during development. Therefore this Actor has no keyword research mode and does not turn an input keyword into volume, difficulty, related questions, or a search results page. The public overview's keyword sample is tied to a domain. An on-page audit of arbitrary third-party URLs is also outside this Actor's scope.

The public domain overview's country value is chosen by the source. Adding a country query to the tested overview URL did not change its country. `rankingCountry` filters only top websites. Some sites have no public overview and return a missing page. Other pages can temporarily fail or change their embedded data format. The Actor logs a domain failure and continues with the next domain. A run with no records fails rather than returning an apparently successful empty export.

Rankings are limited to the entries embedded in the selected public page. In the tested worldwide table there were 92 entries; a tested country-and-industry table had 99. Ranking positions are assigned in the displayed order. Estimates and historical series can change as the source refreshes data. Proxy retries handle access interruptions where possible, but they cannot provide data that the public page does not expose.

### Use with AI agents (MCP)

The input schema names each parameter and explains which modes use it. An agent can create a run, wait for completion, and read the default dataset. Two example prompts:

> Collect public domain overview metrics for wikipedia.org and archive.org. Return monthly visits, organic traffic, authority score, referring domains, and the data date.

> Get the global top websites across all industries. Show the ten rows with the largest positive month-over-month visit change, including rank, visits, and source date.

Ask for `recordType` and `dataDate` in mixed-mode summaries. That keeps domain overviews separate from ranking rows and makes estimates traceable to the public source month.

### FAQ

**Is the data public?** Yes. The Actor reads pages accessible without a site login. Follow the site's terms and applicable law, and use exported data responsibly.

**Do I need a site subscription or key?** No subscription or key is required for the public modes documented here. Private keyword research and full dashboard data are not included.

**Do I need to configure proxies?** The default configuration uses a proxy automatically. Override it only when your setup requires a different routing choice. Access can vary by region and over time.

**How fast is a run?** A ranking table takes one public page request. Domain mode makes one overview request per unique domain, with retries on access problems. The observed golden run completed with 94 records, but request time and site behavior vary.

**Can I schedule and monitor runs?** Yes. Schedule the Actor and inspect each run's status and dataset. Keep `dataDate` because several runs may refer to the same source month.

**Can I export to a spreadsheet?** Yes. Export the dataset as CSV or a spreadsheet file, or read the JSON records through the dataset API. Nested sample arrays remain easiest to inspect in JSON.

**Why is a field empty?** The source may not publish that measure for the domain or ranking. For example, a category rank may be absent. The Actor leaves missing values empty and does not estimate them.

### Integrations

Read the dataset through the platform API, attach a webhook to a finished run, or feed exported rows to a spreadsheet, reporting workflow, or automation service. Filter by `recordType` before loading mixed-mode rows into a fixed-column report. Scheduling supports recurring snapshots; keep the run date and `dataDate` together to distinguish collection time from the source period.

### Support

Open an issue on the Issues tab; we reply within 24h and add fields on request.

# Actor input Schema

## `mode` (type: `string`):

Choose domain overviews, top website rankings, or both in one dataset.

## `domains` (type: `array`):

Public website domains or URLs to analyze. Used in domain and both modes; one overview record per available domain.

## `rankingCountry` (type: `string`):

Country slug for top websites, such as global, us, or de. Used in topWebsites and both modes.

## `rankingIndustry` (type: `string`):

Industry slug for top websites, such as all, finance, or education. Used in topWebsites and both modes.

## `maxItems` (type: `integer`):

Stop after this many output records across all selected modes. Set 0 for no cap.

## `proxyConfiguration` (type: `object`):

Proxy configuration for page requests. The default uses Apify Proxy.

## Actor input object example

```json
{
  "mode": "domain",
  "domains": [
    "wikipedia.org"
  ],
  "rankingCountry": "global",
  "rankingIndustry": "all",
  "maxItems": 200,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Domain overviews and top website rankings in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "wikipedia.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("mlg14/semrush-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["wikipedia.org"] }

# Run the Actor and wait for it to finish
run = client.actor("mlg14/semrush-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "wikipedia.org"
  ]
}' |
apify call mlg14/semrush-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mlg14/semrush-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/j2OUA8LOI0Nbo38zp/builds/OXjpiVHepDETTNluq/openapi.json