# Semrush Bulk Website Stats Scraper (`khadinakbar/semrush-bulk-website-stats-scraper`) Actor

Collect source-dated public website stats in bulk: traffic, Authority Score, backlinks, rankings, engagement, and available keyword and country samples. Returns validated records and explicit no-data diagnostics.

- **URL**: https://apify.com/khadinakbar/semrush-bulk-website-stats-scraper.md
- **Developed by:** [Khadin Akbar](https://apify.com/khadinakbar) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 website stats records

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Samrush Bulk Website Stats Scraper

Collect public Semrush website overview statistics for a list of domains in one Apify run. The implementation accepts domain names or website URLs, checks each unique domain once, and returns dated metric records with source URLs. It uses the public website overview page without a Semrush account, API key or paid subscription.

**Development status:** this package is implemented and tested locally. The owner has confirmed permission for the source and intended use. The final cloud acceptance matrix and measured competitive pricing are pending. “Free” describes the Semrush source; the proposed Apify Actor pricing is Pay per event plus platform usage. No pricing has been activated.

### Best-fit workflow

Use this Actor when you need to compare a website shortlist using public monthly traffic estimates and authority metrics. Avoid it when the job needs logged-in reports or domains without public coverage. For example, an agency can combine 100 customer and competitor domains, remove duplicates, and export the successful records into a reporting spreadsheet. Keep the reporting month beside each metric: retrieval today does not make the underlying estimate today's traffic.

When you need a small Authority Score and backlink shortlist, use the existing [Semrush Authority Score Scraper](https://apify.com/khadinakbar/semrush-domain-authority-scraper). This bulk package adds engagement, rank, source-date and available public table fields. It is a public-overview collector, with coverage limited to domains Semrush exposes on those pages.

### Input and first run

Add domains to the Input tab, choose your domain limit, and start a run after deployment. Review `OUTPUT.outcome` and `RUN_SUMMARY` before consuming the Dataset.

```json
{
  "domains": ["google.com", "github.com", "wikipedia.org"],
  "maxDomains": 100,
  "includeDetails": true,
  "requestIntervalSeconds": 2,
  "requestTimeoutSeconds": 30,
  "maxRetries": 2
}
```

| Input | Behavior |
| --- | --- |
| `domains` | Domain strings or HTTP(S) website URLs. |
| `startUrls` | Website URL entries, combined with `domains`. |
| `maxDomains` | First 1–1,000 unique valid domains; default 100. |
| `includeDetails` | Include public table samples; default true. |
| `requestIntervalSeconds` | 1–30 seconds between domain checks; default 2. |
| `requestTimeoutSeconds` | 5–60 seconds per HTTP attempt; default 30. |
| `maxRetries` | 0–3 retries for transient failures; default 2. |

At least one valid domain is required. URLs become hostnames: paths and query parameters do not select page-level traffic. `www` is removed; other subdomains remain separate. Credentials, IP addresses, unsupported schemes and nonstandard ports are rejected. Invalid and duplicate entries are counted without storing the original URL or its query parameters.

### Website stats output

Each successful domain produces one validated Dataset record. The following is a shortened example from a public page retrieved on October 14, 2026; its metrics cover September 2026.

```json
{
  "domain": "google.com",
  "reportMonth": "2026-09",
  "sourceUpdatedAt": "2026-10-11",
  "visits": 108899227055,
  "authorityScore": 100,
  "pagesPerVisit": 6.6669,
  "averageVisitDurationSeconds": 801,
  "bounceRatePercent": 32.18,
  "backlinks": 47653661167,
  "referringDomains": 64203372,
  "globalRank": 1,
  "sourceUrl": "https://www.semrush.com/website/google.com/overview/"
}
```

| Field group | Included data |
| --- | --- |
| Traffic | Visits, monthly change, organic and paid search visit estimates. |
| Engagement | Pages per visit, average duration in seconds, bounce rate in percent. |
| Authority | Authority Score, backlinks, referring domains and public penalty-risk label. |
| Ranking | Global, country and category rank, with their country/category labels. |
| Detail samples | Available country shares, top organic keywords, competitor samples, categories, visits and device histories. |
| Provenance | Domain, source URL, retrieval time, reporting month and per-section source dates. |

Percentage fields use 0–100 units; monthly changes may be negative. Zero remains zero. Unavailable scalar fields are `null` and named in `missingMetrics`. Detail arrays are empty when unavailable and omitted when `includeDetails` is false. Keyword and country tables are limited public samples, with a maximum of 20 entries per array. Different sections may use different reporting dates; inspect `sourceDates` and `keywordDatabase` before combining them. Export the Dataset as JSON, CSV or Excel using Apify's export controls.

### Proposed Pay per event pricing

The private first-test price is $0.40 per 1,000 records on Free/Starter, $0.35 on Scale and $0.30 on Business, plus $0.00005 startup scaled by memory. Apify platform usage is billed separately. These event fees are below the verified narrow public-overview competitors, but total customer cost must include usage and can be higher for slow or low-yield jobs. The previous $5-per-1,000 proposal was rejected. Final launch pricing and worst-case costs remain under validation. Check the current [Pricing tab](https://apify.com/khadinakbar/semrush-bulk-website-stats-scraper/pricing) after private deployment. For example, 100 Starter records have $0.04 in result fees, plus startup and measured usage.

Set the run's `maxTotalChargeUsd` to bound event fees and use `maxDomains` to bound requests. Collection stops before more requests when the SDK reports no remaining result-event capacity. Every plan receives the same field set and collection behavior.

### API and agent workflow

After private deployment, use the Apify API with your token in the Authorization header:

```bash
curl --request POST \
  'https://api.apify.com/v2/acts/khadinakbar~semrush-bulk-website-stats-scraper/runs' \
  --header "Authorization: Bearer $APIFY_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{"domains":["google.com","github.com"],"maxDomains":2}'
```

Wait for the run to finish, read `OUTPUT`, then fetch items from its `defaultDatasetId`. The output schema links to the Dataset and both terminal records. Hosted MCP discovery and execution remain untested until deployment.

**AI agent prompt:**

> Check the public Semrush website stats for these domains, return the report month and source URL with each result, and list unavailable domains separately from upstream failures. Limit the run to 100 unique domains and report any partial outcome.

### Limits, retries and recovery

The Actor uses one worker with a delay between requests. It honors short `Retry-After` waits, applies bounded transient retries, and stops the batch on access denial or exhausted rate limits. Some domains have no public overview. Locked traffic-cost, advertising-spend and premium reports are excluded. These are Semrush estimates, with the source's coverage and update cycle.

`COMPLETE` means the requested bounded work finished; `PARTIAL` preserves useful rows alongside gaps, caps or interruption. `VALID_EMPTY` means no public data was available. `INVALID_INPUT` describes a fixable request or insufficient event budget. `UPSTREAM_FAILED` and `CONFIG_ERROR` fail the run when no useful data can be confirmed.

Progress is checkpointed per domain. A resumed run with matching input skips completed domains. A pending dataset write is treated as uncertain and stops further billing until reconciled; the implementation does not promise exactly-once delivery across interrupted writes. Start a fresh run for a fresh snapshot. Automatic restart recovery, cloud billing and hosted MCP still require live acceptance tests.

### Builder's note and access requirements

The public overview embeds structured Astro data. Parsing that payload preserves exact numbers and distinguishes a real zero from a missing value. Sections marked unavailable are excluded, and schema validation occurs before any billable write. This avoids substituting rounded display labels or creating empty billable rows.

Semrush's Terms of Service currently prohibit scraping its content. The owner has confirmed permission covering automated collection and the intended data use. Callers are responsible for their own authorized use. This package does not grant those rights. Semrush is a trademark of its owner. This independent Actor is not affiliated with, associated with, or endorsed by Semrush.

### Verification and freshness

The parser was checked against a public overview retrieved on 2026-10-14 and a sanitized metric fixture. Local tests also cover synthetic zero/no-data, locked sections, mismatched domains, schema drift, request limits, retries, deduplication and ambiguous writes. These local checks establish implementation behavior; final-build cloud run, charge, restart and MCP evidence remains pending.

For parser changes or missing coverage, use the Actor's Issues tab after deployment and include the domain, source month and run ID.

# Actor input Schema

## `domains` (type: `array`):

Enter public website domains or HTTP(S) website URLs to collect their Semrush public overview stats. Duplicate domains are checked once. At least one domain or start URL is needed. Only domains with public Semrush coverage can return records.

## `startUrls` (type: `array`):

Enter the websites you want to look up. URLs are converted to hostnames and combined with domains. Paths and queries do not select page-level traffic. Do not supply credentials in URLs.

## `maxDomains` (type: `integer`):

Limit unique domains checked per run. The first domains in your list are used. A smaller limit bounds request time and result fees. Domains excluded by the cap are reported in the summary.

## `includeDetails` (type: `boolean`):

Include available country, keyword, competitor, category, visits-history and device-history tables. These are limited public samples. Missing or locked sections stay empty. Disable for a compact metric-only record.

## `requestIntervalSeconds` (type: `integer`):

Wait this many seconds between domain requests. Collection uses one worker. Increase this value for larger batches. Rate-limit or access-denied responses stop the batch after bounded retries.

## `requestTimeoutSeconds` (type: `integer`):

Bound each HTTP attempt including reading the response body. Network failures may be retried. Unavailable metrics are described in RUN_SUMMARY. Useful records remain available after partial runs.

## `maxRetries` (type: `integer`):

Retry timeouts, HTTP 429 and server errors up to this count. Retry-After waits over 60 seconds stop the attempt. HTTP 401 and 403 are not retried. Retries do not add result-event charges.

## Actor input object example

```json
{
  "domains": [
    "google.com",
    "github.com"
  ],
  "maxDomains": 100,
  "includeDetails": true,
  "requestIntervalSeconds": 2,
  "requestTimeoutSeconds": 30,
  "maxRetries": 2
}
```

# Actor output Schema

## `dataset` (type: `string`):

Validated records for domains with usable public metrics.

## `output` (type: `string`):

Compact result counts, outcome and charge-cap status.

## `runSummary` (type: `string`):

No-data domains, upstream failures, progress, duplicate counts and event counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "google.com",
        "github.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("khadinakbar/semrush-bulk-website-stats-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "google.com",
        "github.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("khadinakbar/semrush-bulk-website-stats-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "google.com",
    "github.com"
  ]
}' |
apify call khadinakbar/semrush-bulk-website-stats-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,khadinakbar/semrush-bulk-website-stats-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VeJ8WEQuZGfjFxV1H/builds/cHjdojxmAaSo14OTd/openapi.json