# Similarweb Traffic Crawler (`yeekal/similarweb-traffic-crawler`) Actor

Collect public Similarweb traffic estimates in bulk with fast HTTP requests, including rankings, engagement, traffic sources, countries, keywords, competitors, and AI referrals.

- **URL**: https://apify.com/yeekal/similarweb-traffic-crawler.md
- **Developed by:** [yeekal](https://apify.com/yeekal) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Similarweb Traffic Scraper: Website Traffic, Rankings & Competitors

Collect public Similarweb traffic estimates for one website or up to 200 domains per run. This Apify Actor is optimized for parallel collection, fast turnaround, and predictable resource usage.

Enter domains or URLs and receive one normalized Dataset item per successful website. Results are ready for export to JSON, CSV, Excel, or an automated Apify workflow.

### Data you can collect

Available fields depend on what Similarweb publishes for each domain.

| Data | Output field | Common use |
| --- | --- | --- |
| Estimated monthly visits | `visits`, `monthlyVisits` | Compare reach and traffic trends |
| Global, country, and category rank | `globalRank`, `overview` | Qualify and benchmark websites |
| Bounce rate, pages per visit, visit duration | `engagement` | Evaluate audience quality |
| Direct, search, social, referral, ads, and AI traffic | `trafficSources` | Analyze acquisition channels |
| Leading countries | `topCountries` | Identify important markets |
| Top search keywords | `topKeywords` | Support SEO and content research |
| Similar websites | `competitors` | Discover alternatives and market peers |
| AI referral data | `aiTraffic` | Track traffic from AI assistants |

### Why use this Actor?

- Process up to 200 domains in one run.
- Process domain jobs concurrently for faster batch completion.
- Normalize full URLs, international domains, casing, and duplicates.
- Retry temporary failures automatically.
- Skip failed domains in the Dataset and log a concise error code for monitoring.
- Return compact normalized records without duplicating the upstream payload.

### Input example

```json
{
    "domains": [
        "google.com", "facebook.com", "youtube.com", "github.com", "microsoft.com",
        "amazon.com", "wikipedia.org", "baidu.com", "apify.com", "bing.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"],
        "apifyProxyCountry": "US"
    },
    "maxConcurrency": 10,
    "maxRetries": 2,
    "requestTimeoutSecs": 30,
    "includeRaw": false
}
```

### Input parameters

| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `domains` | string array | Required | Domains or URLs to analyze. Maximum 200 values per run. |
| `proxyConfiguration` | object | US residential proxy | Standard Apify proxy configuration. Residential proxies are recommended for reliability. |
| `maxConcurrency` | integer | `10` | Simultaneous domain requests. Range: 1-50. |
| `maxRetries` | integer | `2` | Retries after temporary failures. Range: 0-5. |
| `requestTimeoutSecs` | number | `30` | Timeout for each request. Range: 5-120 seconds. This is not the total Actor run timeout. |
| `includeRaw` | boolean | `false` | Include the complete source response for troubleshooting. Increases Dataset size and storage usage. |

Each domain is processed independently. Concurrency controls how many domain jobs can run at the same time.

### Output example

```json
{
    "domain": "example.com",
    "status": "ok",
    "fetchedAt": "2026-08-06T08:00:00.000Z",
    "snapshotDate": "2026-06-01",
    "visits": 123456,
    "globalRank": 1234,
    "overview": {
        "siteName": "example.com",
        "title": "Example",
        "category": "business",
        "countryRank": { "rank": 500, "countryCode": "US" },
        "categoryRank": { "rank": 20, "category": "Business" }
    },
    "engagement": {
        "bounceRate": 0.42,
        "pagesPerVisit": 3.7,
        "timeOnSiteSeconds": 185.4
    },
    "monthlyVisits": {
        "2026-04-01": 118000,
        "2026-05-01": 121000,
        "2026-06-01": 123456
    },
    "trafficSources": {
        "direct": 0.41,
        "searchOrganic": 0.35,
        "referrals": 0.12
    },
    "topCountries": [
        { "countryCode": "US", "country": "United States", "share": 0.48 }
    ],
    "topKeywords": [],
    "competitors": [],
    "aiTraffic": null,
    "request": {
        "attempts": 1,
        "elapsedMs": 620,
        "httpStatus": 200,
        "proxyUsed": true,
        "proxyCountry": "US"
    }
}
```

If every attempt fails, the Actor logs the domain, a concise error code, and retry count, then omits that domain from the Dataset. Failed domains do not create Dataset items; when the Actor is priced per result event, they do not trigger a result event.

In the Apify console, `Traffic results` is a table view that selects summary fields such as visits and rank. `All fields` shows the complete JSON item. Both views use the same Dataset; API calls and Dataset exports return the complete item, not only the `Traffic results` columns.

### Performance and proxy settings

The default concurrency is ten and the allowed range is 1-50. Use 10 for small batches and 50 for 100-200-domain batches when the account can sustain the parallel workload. More concurrency reduces elapsed time but does not reduce the number of domains submitted.

US Apify Residential Proxy is recommended for reliability. You can provide another standard Apify proxy configuration when your workflow requires a different region or pool.

Retries are handled per domain with backoff, while other domains continue processing. A failed domain does not block the rest of the batch.

Results are normalized into compact records so downstream exports and workflows do not need to handle a duplicated source payload.

### How it works

1. Normalize and deduplicate the submitted hostnames.
2. Schedule domain jobs concurrently.
3. Collect the available traffic, ranking, engagement, and referral metrics.
4. Retry temporary failures with backoff.
5. Normalize traffic, rank, geography, keyword, competitor, and AI referral fields.
6. Write one success record per successful domain to the default Dataset.

Recommended configurations:

- Small batch: 10 domains, `maxConcurrency: 10`, `maxRetries: 2`, `requestTimeoutSecs: 30`.
- Large batch: 100-200 domains, `maxConcurrency: 50`, `maxRetries: 1-2`, `requestTimeoutSecs: 30`.

Submitting multiple domains in one run is generally faster and more economical than starting one run per domain because Actor startup and proxy setup are shared. Keep batches at or below 200 domains so failures, retries, and run timeout remain easy to monitor.

# Actor input Schema

## `domains` (type: `array`):

Domain names or website URLs. Hostnames are normalized, lowercased, and deduplicated. Maximum 200 domains per run.

## `proxyConfiguration` (type: `object`):

US residential proxies are recommended for reliable batch collection.

## `maxConcurrency` (type: `integer`):

Maximum number of domains requested at the same time. Use 10 for small batches and up to 50 for 100-200-domain batches when your proxy plan can sustain the load.

## `maxRetries` (type: `integer`):

Retries after a temporary network error, block, rate limit, or invalid response.

## `requestTimeoutSecs` (type: `number`):

Maximum time allowed for each Similarweb request.

## `includeRaw` (type: `boolean`):

Include the complete source response under raw for troubleshooting. This can substantially increase Dataset size and storage usage.

## Actor input object example

```json
{
  "domains": [
    "google.com",
    "facebook.com",
    "youtube.com",
    "github.com",
    "microsoft.com",
    "amazon.com",
    "wikipedia.org",
    "baidu.com",
    "apify.com",
    "bing.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ],
    "apifyProxyCountry": "US"
  },
  "maxConcurrency": 10,
  "maxRetries": 2,
  "requestTimeoutSecs": 30,
  "includeRaw": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "google.com",
        "facebook.com",
        "youtube.com",
        "github.com",
        "microsoft.com",
        "amazon.com",
        "wikipedia.org",
        "baidu.com",
        "apify.com",
        "bing.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("yeekal/similarweb-traffic-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "google.com",
        "facebook.com",
        "youtube.com",
        "github.com",
        "microsoft.com",
        "amazon.com",
        "wikipedia.org",
        "baidu.com",
        "apify.com",
        "bing.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("yeekal/similarweb-traffic-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "google.com",
    "facebook.com",
    "youtube.com",
    "github.com",
    "microsoft.com",
    "amazon.com",
    "wikipedia.org",
    "baidu.com",
    "apify.com",
    "bing.com"
  ]
}' |
apify call yeekal/similarweb-traffic-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=yeekal/similarweb-traffic-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/28iY88Nua3NHeE7Ca/builds/BVWe6xAamVvpg8uq1/openapi.json