# 🧪 Semrush Website Scraper — Traffic, Ranks & AI Traffic (`thenetaji/semrush-website-scraper`) Actor

Read any website's whole free Semrush profile as one row: global, country and industry rank, authority score, monthly visits, bounce rate, time on site, top countries, organic and paid search, top keywords with volume and CPC, backlinks, referrers, and referral traffic from each AI assistant.

- **URL**: https://apify.com/thenetaji/semrush-website-scraper.md
- **Developed by:** [The Netaji](https://apify.com/thenetaji) (community)
- **Categories:** SEO tools, Marketing, Business
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $6.80 / 1,000 website profiles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Semrush Website Scraper

A website's whole free Semrush profile as one spreadsheet row. Each row carries the global,
country and industry rank with the movement on each, the authority score, monthly visits,
pages per visit, time on site and bounce rate, the desktop-versus-mobile split with six
months of history, the top five countries by traffic, organic and paid search traffic with
six months of history, the top five organic keywords with their position, search volume and
cost per click, backlinks and referring domains with their histories, the rivals Semrush
names, the top five sources visitors arrive from and destinations they leave for, and
referral traffic from each individual AI assistant.

That last one is the panel worth knowing about. ChatGPT, Claude, Gemini, Perplexity, Grok
and coding agents each get their own traffic figure, their own share and their own six
months of history, and it is on the free page.

### Accepted input

`domains` is required and takes one website per line, either as a domain such as
`stripe.com` or as any link to the site. A link is reduced to its host.

Semrush keys these pages on the **registrable domain**. `www.` is stripped because
`www.stripe.com` is a 404 on Semrush while `stripe.com` is a 200, so removing it turns a
broken request into a working one. Any other subdomain — `blog.stripe.com`,
`support.stripe.com` — genuinely has no page of its own and is skipped with a line in the
run log rather than being quietly rewritten to its parent, which would answer a question you
did not ask with numbers for a different site.

`maxItems` defaults to `100` and caps how many websites the run reads. One website is one
request and one row here, so it is a cap on the list rather than on a page of results. `0`
means no limit.

There is **no country input**, and its absence is deliberate. The same page fetched from US,
GB, DE and IN exits came back identical within two bytes, with the same database named in
all four, and Semrush's own `db` and `database` parameters were ignored. A parameter that
appears to select a market and does not is worse than a missing one.

```json
{
  "domains": ["stripe.com", "gocardless.com", "https://adyen.com/pricing"],
  "maxItems": 100
}
```

### Response fields

```json
{
  "requested_domain": "stripe.com",
  "domain": "stripe.com",
  "url": "https://www.semrush.com/website/stripe.com/overview/",
  "as_of": "2026-07-15",
  "updated_at": "2026-08-12",
  "database": "US",
  "global_rank": 322,
  "global_rank_change_places": -1,
  "country_rank": 149,
  "country_code": "US",
  "country_name": "United States",
  "category_rank": 4,
  "category_name": "Finance",
  "category_slug": "finance",
  "authority_score": 92,
  "visits": 203227314,
  "visits_change_percent": 0.1294,
  "pages_per_visit": 4.6,
  "time_on_site_seconds": 498,
  "bounce_rate": 0.6072,
  "organic_search_traffic": 41208312,
  "paid_search_traffic": 182004,
  "backlinks": 151822916,
  "referring_domains": 289110,
  "google_penalty_risk": "Low",
  "ai_other_traffic": 149279270,
  "ai_other_share": 0.93069,
  "top_keywords": [
    { "keyword": "stripe", "position": 1, "search_volume": 673000, "cpc": 2.99 }
  ],
  "ai_traffic": [
    { "source": "chatgpt.com", "name": "ChatGPT", "traffic": 798467, "share": 0.00498 }
  ],
  "unavailable_panels": ["siteAudit"]
}
```

The example is trimmed; the full row also carries `categories`, `visits_history`,
`device_history`, `top_countries`, `search_traffic_history`, `backlinks_history`,
`referring_domains_history`, `competitors`, `traffic_incoming`, `traffic_outgoing` and
`related_domains`.

Fields Semrush states no figure for are returned as null rather than omitted, so every row
has the same shape.

**null is never rendered as zero, and that distinction is the point.** Semrush genuinely has
no current traffic estimate for some websites — openai.com and doordash.com among them — and
a zero in that place would be a traffic figure nobody measured. `unavailable_panels` names
the sections Semrush itself flagged for that domain, under its own names, and the run log
says so in words when the headline `visits` is missing.

`bounce_rate` is a **fraction**: `0.6072` is 60.72%. So is `visits_change_percent`:
`0.1294` is +12.94%. The rank movements are the exception — they are stated in **positions**
and named `_change_places` for it, and a negative number there is an improvement because the
rank number fell.

`top_keywords` and the other subtrees are republished in Semrush's own shape. There is no
measured field list to rename them into, and renaming fields nobody has seen populated
publishes a permanently-null column and calls it a schema. The keyword `intents` code inside
is republished undecoded for the same reason.

`traffic_incoming` and `traffic_outgoing` each carry a `domain` field alongside the source
label, and it is non-null exactly when that source has a Semrush page of its own. So a
traffic-journey row feeds straight back into another run of this Actor without parsing a
label.

### Behaviour on partial results

A website Semrush has no page for is skipped with a line in the run log and the rest of the
list still runs — losing nineteen good rows to one domain that has no page is the wrong
trade. An entry that is not a domain at all is skipped before any request is made, because
noticing costs nothing and a request that was never going to resolve is worse than a message
naming the shape.

A website with a page but no current traffic estimate is a **successful** row. The ranks, the
authority score, the backlinks and often six months of device history are all still there;
`visits` is null and `unavailable_panels` says which sections Semrush marked empty. That is
stable across requests — openai.com returned the identical set as request 1 and again as
request 24 of one walk — so it is a gap in Semrush's index rather than something a retry
fixes.

### Frequently asked questions

**Are these live numbers?**
No, and nothing here pretends otherwise. They are monthly estimates. Every row carries the
month it describes in `as_of`, and two runs a week apart return the same figures. Running
this daily buys nothing; running it monthly is the shape of the data.

**Is there a view limit, and does it start returning zeroes?**
No. That failure is real on some traffic-analytics sites — they meter anonymous profile views
and then serve the same page with every metric zeroed — so it was tested for directly here.
Sixty consecutive fetches on one identity with no delay returned sixty of sixty, and every
repeated domain matched its first reading exactly: stripe.com at 203,227,314 visits on
request 1 and again on request 40. Zero anomalies in sixty.

**Why can I not choose a country database?**
Because there is nothing for the choice to select. The same page fetched from US, GB, DE and
IN residential exits came back byte-identical within two bytes, with `database` reading `US`
in all four, and both of Semrush's own database parameters were ignored. The `database` field
on every row states what the figures are actually measured in.

**Why did `blog.example.com` come back with nothing?**
Because Semrush publishes a page per registrable domain and that subdomain has none. It is
not rewritten to `example.com`, because that would return numbers for the whole site under
the name of the blog. Pass the registrable domain if that is what you meant.

**Why are there only five countries and five keywords?**
Because five is what the free page carries. There is no parameter that returns more, and one
that appeared to would be a lie in the input form.

**Is a Semrush account or API key required?**
No. No account, session cookie or key of any kind is supplied to the Actor or needed by it.
Everything it returns is what the overview page shows an anonymous visitor — open the `url`
on any row and you see the same numbers.

### Related Actors

[Semrush Competitors Scraper](https://apify.com/thenetaji/semrush-competitors-scraper) takes
the same website list and returns each rival with the subject's own metric set, so the two
line up in one table. The `competitors` panel on a row here gives a name, a similarity score
and a visits figure that is frequently null; that Actor gives every rival visits, authority
score, bounce rate, pages per visit, time on site and all three ranks.

[Semrush Top Websites](https://apify.com/thenetaji/semrush-trending-scraper) goes the other
way: instead of looking up a website you already know, it returns the hundred most visited
websites for a country and an industry. Every `domain` it produces is a valid input here.

# Actor input Schema

## `domains` (type: `array`):

The websites to look up, one per line, as a domain such as stripe.com or as any link to the site. A full link is accepted and reduced to its host. A `www.` prefix is removed, because the site publishes a page for `stripe.com` and none for `www.stripe.com`; any other subdomain genuinely has no page of its own and comes back empty rather than being quietly rewritten to its parent. These are monthly estimates, not live counters — every row carries the month it describes in `as_of`, and two runs a week apart return the same figures.

## `maxItems` (type: `integer`):

Maximum number of websites to read. Set 0 for no limit. One website is one request and one row here, so this is a cap on the list rather than on a page of results.

## Actor input object example

```json
{
  "domains": [
    "stripe.com"
  ],
  "maxItems": 10
}
```

# Actor output Schema

## `dataset` (type: `string`):

All records scraped by this run

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "stripe.com",
        "gocardless.com"
    ],
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("thenetaji/semrush-website-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "stripe.com",
        "gocardless.com",
    ],
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("thenetaji/semrush-website-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "stripe.com",
    "gocardless.com"
  ],
  "maxItems": 10
}' |
apify call thenetaji/semrush-website-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thenetaji/semrush-website-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tRxudG1WhVcw24u6A/builds/VuVUxSXabYVKWHED2/openapi.json