# SimilarWeb Website Traffic & Competitor Scraper (`midnight_static/similarweb-website-traffic-scraper`) Actor

Bulk website traffic estimates and competitor analysis. 56 fields per domain: monthly visits, bounce rate, pages per visit, traffic sources, top countries, top keywords with CPC, rankings with history and similar sites. Each record names its data month and flags verified analytics vs estimate.

- **URL**: https://apify.com/midnight\_static/similarweb-website-traffic-scraper.md
- **Developed by:** [Midnight Static](https://apify.com/midnight_static) (community)
- **Categories:** SEO tools, Marketing, Business
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.20 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SimilarWeb Website Traffic Scraper

Hand it a list of domains. Get back a full competitive profile for each one — traffic, three rankings with history, channel mix, geography, keywords, audience, tech and competitors. **56 fields per domain**, as JSON, CSV or Excel.

### Three things that are different here

**1. You pay only for records that contain data.**
A domain that comes back blocked, failed, uncovered or empty is counted in the run report and then dropped. It is never written to your dataset and never billed. Check the other traffic scrapers on this Store: you will not find this promise on any of them, and you will find their users saying so in the issues.

**2. Every record tells you how good it is.**
`snapshotDate` gives the month the data covers. `isDataFromGoogleAnalytics` tells you whether the numbers are the site's own verified analytics or an estimate. `lockedSections` names exactly which sections SimilarWeb is hiding behind its own paywall for that domain. `filledFieldCount`, `httpStatus`, `fetchedAt` and `sourceUrl` let you trace and rank any row. You never have to wonder whether a blank means zero, missing, or paywalled.

**3. It reads the page everyone can see.**
Most scrapers in this category call SimilarWeb's internal data endpoint. That endpoint fails often, and a failed call still costs you a result when the Actor writes an empty row. This one parses the public profile page instead. Ten out of ten domains returned a complete record in testing, with no retries.

### What you get

**Traffic and engagement**
Monthly visits, month-over-month change, bounce rate, pages per visit, average visit duration.

**Three rankings, with history**
Global, country and category rank — each with the previous month's value, the change, and a three-month history series. Most traffic scrapers give you today's number and nothing to compare it against.

**Where the traffic comes from**
Ten traffic-source channels with their share (direct, organic, referral, social, paid, mail, affiliate and more), top countries with share and share change, top referring sites and top outgoing sites.

**Search**
Organic vs paid split, total keyword count, and the top keywords with their cost per click.

**Audience and competition**
Age distribution, male/female split, similar sites with an affinity score, and the sites ranked immediately around this domain.

**Technology and advertising**
Count of detected technologies and technology categories, advertising networks and advertising sites.

### Typical uses

Competitive research before entering a market. Qualifying a partner or an acquisition target. Tracking whether a competitor's traffic is rising or falling month over month. Enriching a list of domains with traffic size. Checking which channels a site actually depends on before copying its strategy.

### Fields worth knowing about

**`snapshotDate`** tells you which month the data covers. Traffic estimates are monthly, and a number without its month is not comparable to anything. Not every scraper reports this.

**`isDataFromGoogleAnalytics`** tells you whether the numbers come from the site's own verified analytics or from estimation. When a site owner connects their analytics, the figures stop being an estimate. If you are making a decision on this data, you want to know which kind you are looking at.

**`lockedSections`** names the sections SimilarWeb keeps behind its own paywall for that specific domain — typically referrals, outgoing traffic or advertising. When a section is locked, the public page simply shows less, and a scraper that does not tell you this leaves you guessing whether the data is missing or genuinely zero. `isPartial` is the one-line version of the same answer.

**`sourceUrl`, `fetchedAt`, `httpStatus` and `filledFieldCount`** travel with every record, so a row can be traced back to the exact page and moment it came from, and you can sort a batch by how complete each record is.

### Example record (trimmed)

```json
{
  "domain": "github.com",
  "snapshotDate": "2026-08-01T00:00:00+00:00",
  "globalRank": 50,
  "globalRankPrevious": 49,
  "globalRankChange": -1,
  "countryCode": "US",
  "countryRank": 81,
  "categoryRank": 4,
  "categoryId": "computers_electronics_and_technology/programming_and_developer_software",
  "visitsTotal": 649321442,
  "visitsChange": 0.0179,
  "bounceRate": 0.3666,
  "pagesPerVisit": 5.77,
  "avgVisitDuration": "00:06:24",
  "isDataFromGoogleAnalytics": false,
  "trafficSources": [{ "source": "direct", "rank": 1, "share": 0.526 }],
  "topCountries": [{ "countryCode": "US", "share": 0.186, "shareChange": -0.026 }],
  "topKeywords": [{ "keyword": "github", "cpc": 1.63 }],
  "similarCompetitors": [{ "domain": "stackoverflow.com", "affinity": 1, "categoryRank": 81 }],
  "globalRankHistory": [{ "date": "2026-06-01T00:00:00+00:00", "rank": 47 }],
  "lockedSections": ["incomingReferrals", "outgoingReferrals", "ads"],
  "isPartial": true,
  "filledFieldCount": 54,
  "httpStatus": 200,
  "fetchedAt": "2026-09-16T17:20:32.878Z",
  "sourceUrl": "https://www.similarweb.com/website/github.com/"
}
```

### Input

Paste domains one per line. Bare domains and full URLs both work — `github.com`, `www.github.com`, `https://github.com/features` all resolve to the same record. Duplicates are removed.

`maxItems` caps how many domains are processed so a long list cannot produce an unexpected charge. `concurrency` controls how many run in parallel.

### Failures are loud

The run fails with an explanation rather than finishing quietly with less data than you expected. If the page structure changes, you get a parse error naming the affected domains, not an empty dataset. Counters are reconciled at the end of every run: requested, written, robots-blocked, HTTP-failed, parse-failed. The full breakdown is written to the `RUN_STATS` key.

Transient gateway errors are retried up to three times with a fresh proxy session before a domain is given up on.

### Limits and compliance

`robots.txt` is checked before every request and a disallowed path is never fetched — the run reports it and skips, it does not force its way through.

No personal data is collected. The output is built from an explicit list of known fields, so nothing outside that list can leak into your dataset.

Data is a monthly estimate from a public page. Very small sites often have no traffic data at all; those domains are counted and reported rather than written as empty records.

### Pricing

**$6.00 per 1,000 records, everything included.** No start fee, no minimum charge per run, and no separate platform usage line on your bill. The number on this page is the number you pay.

Paid Apify plans pay less, automatically — there is nothing to apply for: Bronze $5.40, Silver $4.80, Gold and above $4.20 per 1,000 records. The Free plan price is unchanged at $6.00.

You are charged per record written, and a record is only written when it contains data. Domains that are skipped by robots.txt, fail after three retries, or have no traffic data at all are reported in the run statistics and cost you nothing.

### Measured performance

Run on 16 September 2026 against a deliberately hostile list of 150 domains: 30 large sites, 40 mid-size SaaS products, 30 small tools, 20 very low traffic sites, and 30 domains that do not exist at all.

| | |
|---|---|
| Records written and charged | 115 |
| Domains reported and not charged | 35 |
| Non-existent domains that produced a charge | 0 |
| Parse failures | 0 |
| Paths blocked by robots.txt and skipped | 0 |
| Median fields filled per record | 54 of 56 |
| Records carrying traffic figures | 115 of 115 |
| Wall clock | 15 minutes at concurrency 8 |

Every one of the 30 non-existent domains was counted in the run report and dropped. None of them reached your dataset and none of them reached your bill. Four real domains returned gateway errors after three retries each and were also reported rather than written.

# Actor input Schema

## `domains` (type: `array`):

Domains to analyze, one per line. Accepts bare domains or full URLs (github.com, https://www.stripe.com/pricing). Subdomains and paths are stripped; duplicates are removed.

## `maxItems` (type: `integer`):

Hard cap on how many domains are processed, so a long list cannot run up an unexpected charge. Extra domains are ignored.

## `concurrency` (type: `integer`):

How many domains are fetched in parallel. Each page takes roughly 15-25 seconds, so higher values finish sooner. Lower it if you see timeouts.

## Actor input object example

```json
{
  "domains": [
    "github.com",
    "stripe.com",
    "notion.so"
  ],
  "maxItems": 100,
  "concurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One record per domain: traffic, rankings, sources, geography, keywords, tech and competitors.

## `runStats` (type: `string`):

Counters for the run: written records, robots-blocked, HTTP failures, parse failures.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "github.com",
        "stripe.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("midnight_static/similarweb-website-traffic-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "github.com",
        "stripe.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("midnight_static/similarweb-website-traffic-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "github.com",
    "stripe.com"
  ]
}' |
apify call midnight_static/similarweb-website-traffic-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,midnight_static/similarweb-website-traffic-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kHlBmuX9wuqSD7wC0/builds/4ygxHyFhk3CI2Im6T/openapi.json