# SimilarWeb Scraper - Website Traffic & Analytics (`fascinating_lentil/similarweb-traffic-scraper`) Actor Scrapes SimilarWeb for public website traffic data: global rank, monthly visits, engagement, traffic sources, top countries, and more. - **URL**: https://apify.com/fascinating\_lentil/similarweb-traffic-scraper.md - **Developed by:** [Md Jakaria Mirza](https://apify.com/fascinating_lentil) (community) - **Categories:** Lead generation, SEO tools, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 website scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## SimilarWeb Website Traffic Scraper - Rank, Visits & Traffic Sources Scrape public SimilarWeb website traffic data — global rank, monthly visits, engagement metrics, traffic sources, and top countries — for any list of domains. Get competitor traffic insights without an expensive SimilarWeb subscription, and export to JSON, CSV, Excel, or HTML, or pull via the Apify API — no SimilarWeb login and no API key required. This SimilarWeb scraper reads each domain's public traffic data, validates and deduplicates your domain list automatically, and runs through residential proxies with anti-bot retries so results stay reliable. Every clean record is saved to the Apify Dataset, ready to export or feed into your own analysis. ### What It Extracts For each domain (one record in the dataset): - `domain` - `globalRank`, `countryRank`, `categoryRank` - `categoryName` - `monthlyVisits` (estimated visits) - `visitDuration` (average, seconds), `pagesPerVisit`, `bounceRate` - `topCountries` — top source countries with traffic percentage - `trafficSources` — breakdown of `direct`, `search`, `social`, `referral`, `email`, `displayAds` - `similarWebUrl` — link to the SimilarWeb profile page - `scrapedAt` timestamp The output record also includes `trafficChangeMoM`, `topReferringDomains`, `topSearchKeywords`, `topSocialNetworks`, and `technologies` fields. These are part of the schema but are not provided by the public data endpoint, so they are currently returned as `null` or empty arrays (see Known Limits). ### Use Cases 1. **Competitor research**: Benchmark competitor traffic volume and channel mix against your own performance. 2. **Digital marketing analysis**: See which channels (direct, search, social, referral, email, display) drive traffic to target websites. 3. **SEO and market strategy**: Compare global, country, and category rank across a set of domains to prioritize where to compete. 4. **Investor due diligence**: Validate website traffic claims with independent third-party estimates before investing. 5. **Market sizing**: Estimate market opportunity by scraping traffic across an entire list of industry players in one run. ### Pricing This Actor uses Apify Pay Per Event pricing. You are charged once per website that is successfully scraped and saved with data — blocked or not-found domains are not billed. Apify platform compute and proxy usage are billed separately by Apify. | Event name | Price per event | 1,000 websites | 10,000 websites | | --- | ---: | ---: | ---: | | `website-scraped` | $0.005 | $5.00 | $50.00 | ### Input | Field | Type | Required | Default | Description | | --- | --- | --- | --- | --- | | `domains` | string[] | Yes | `["amazon.com"]` | Website domains to scrape (e.g. `amazon.com`). Enter without `https://` or `www.`. | | `maxResults` | integer | No | `10` | Maximum number of domains to scrape from the list (1–1000). | | `proxyConfiguration` | object | No | `{ useApifyProxy: true, apifyProxyGroups: ["RESIDENTIAL"] }` | Proxy settings. Residential proxies are strongly recommended for SimilarWeb. | ### How to Scrape SimilarWeb (Step by Step) 1. Click **Try for free** / **Run**. 2. Add the domains you want to analyze to `domains` (e.g. `amazon.com`, `flipkart.com`) — no `https://` or `www.` needed. 3. Set `maxResults` to cap how many domains run (start small to test). 4. Keep residential proxies enabled and run the Actor. 5. Export results from the Apify Dataset as JSON, CSV, Excel, or HTML, or pull them via the Apify API. ### Sample Output ```json { "domain": "amazon.com", "globalRank": 11, "countryRank": null, "categoryRank": 1, "categoryName": "E-commerce and Shopping", "monthlyVisits": "2700000000", "visitDuration": 312, "pagesPerVisit": 8.9, "bounceRate": 32.5, "trafficChangeMoM": null, "topCountries": [ { "country": "US", "trafficPercentage": 85.2 }, { "country": "DE", "trafficPercentage": 3.1 }, { "country": "GB", "trafficPercentage": 2.8 } ], "trafficSources": { "direct": 65.2, "search": 22.1, "social": 3.5, "referral": 5.8, "email": 2.1, "displayAds": 1.3 }, "topReferringDomains": [], "topSearchKeywords": [], "topSocialNetworks": [], "technologies": [], "similarWebUrl": "https://www.similarweb.com/website/amazon.com/", "scrapedAt": "2026-06-10T08:30:00.000Z" } ```` ### How It Works 1. Normalizes each domain (strips `https://`, `www.`, and paths), validates it, and removes duplicates. 2. Requests SimilarWeb's public JSON data endpoint (`data.similarweb.com/api/v1/data`) for each domain through residential proxies — more reliable than the Cloudflare-gated, login-walled website DOM. 3. Parses rank, engagement, traffic sources, and top countries from the response. 4. Saves the clean `WebsiteRecord` to the Apify Dataset. 5. Charges `website-scraped` once the record is saved. Blocked or empty responses are skipped without charging. ### Known Limits - Only **public** traffic estimates are returned. Detailed breakdowns behind SimilarWeb's login wall are not available. - `topReferringDomains`, `topSearchKeywords`, `topSocialNetworks`, and `technologies` are included in the output schema but are **not supplied by the public data endpoint**, so they are currently returned as empty arrays. `trafficChangeMoM` is returned as `null` for the same reason. - `topCountries` reports ISO country codes (e.g. `US`, `DE`, `GB`) rather than full country names. - `monthlyVisits` is an estimated value returned as a numeric string. Percentage fields are rounded to one decimal place. - SimilarWeb may block automated traffic; residential proxies are strongly recommended and domains that return no data are skipped (and not billed). ### License Apache-2.0. See `LICENSE`. # Actor input Schema ## `domains` (type: `array`): List of website domains to scrape (e.g. amazon.com). Enter domains without https:// or www. ## `maxResults` (type: `integer`): Maximum number of domains to scrape from the list. ## `proxyConfiguration` (type: `object`): Proxy settings. Residential proxies strongly recommended for SimilarWeb. ## Actor input object example ```json { "domains": [ "amazon.com" ], "maxResults": 10, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "amazon.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("fascinating_lentil/similarweb-traffic-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["amazon.com"] } # Run the Actor and wait for it to finish run = client.actor("fascinating_lentil/similarweb-traffic-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "amazon.com" ] }' | apify call fascinating_lentil/similarweb-traffic-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=fascinating_lentil/similarweb-traffic-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "SimilarWeb Scraper - Website Traffic & Analytics", "description": "Scrapes SimilarWeb for public website traffic data: global rank, monthly visits, engagement, traffic sources, top countries, and more.", "version": "1.0", "x-build-id": "csH4joZrIHUmagq3b" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/fascinating_lentil~similarweb-traffic-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-fascinating_lentil-similarweb-traffic-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/fascinating_lentil~similarweb-traffic-scraper/runs": { "post": { "operationId": "runs-sync-fascinating_lentil-similarweb-traffic-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/fascinating_lentil~similarweb-traffic-scraper/run-sync": { "post": { "operationId": "run-sync-fascinating_lentil-similarweb-traffic-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "domains" ], "properties": { "domains": { "title": "Domains", "type": "array", "description": "List of website domains to scrape (e.g. amazon.com). Enter domains without https:// or www.", "default": [ "amazon.com" ], "items": { "type": "string" } }, "maxResults": { "title": "Max Results", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of domains to scrape from the list.", "default": 10 }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Proxy settings. Residential proxies strongly recommended for SimilarWeb.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```