# Similarweb Scraper (`morph_coder/similarweb-scraper`) Actor Extract website traffic data, traffic source breakdowns, monthly visits, similar sites, WHOIS info, and on-page keyword density for any domain. - **URL**: https://apify.com/morph\_coder/similarweb-scraper.md - **Developed by:** [Morph Coder](https://apify.com/morph_coder) (community) - **Categories:** SEO tools, Lead generation, E-commerce - **Stats:** 4 total users, 3 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $1.00 / 1,000 result items This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## SimilarWeb Traffic Scraper **Actor ID:** `morph_coder/similarweb-scraper` SimilarWeb Traffic Scraper is a powerful, production-ready Apify Actor designed to extract website traffic intelligence, competitor data, domain WHOIS records, and on-page SEO keyword density in bulk. This Actor is highly optimized to avoid detection using rotated residential proxies and custom TLS/HTTP2 fingerprints. > This Actor is **not affiliated** with SimilarWeb, SimilarSites, or AITDK. Users must comply with applicable terms of service and local laws. --- ### ❶ Key Features - 🌍 **Comprehensive Website Analytics (SimilarWeb Mode):** Extract global rank, country rank, category rank, total monthly visits, 3-month historical visits, and traffic source breakdowns (direct, search, referrals, social, mail, display ads). - 🖼️ **Asset Reconstruction:** Reconstruct website icons, desktop previews, and mobile previews directly from metadata, eliminating the need for heavy browser instances. - 🚦 **Geographic Breakdown:** Get the top 5 traffic-driving countries with percentage shares resolved to full country names. - 🤝 **Similar Sites Discovery (Similar Sites Mode):** Retrieve up to 20 similar competitor websites from Similarsites.com, including site descriptions and total visits. - 🔍 **SEO & Domain Intelligence (AITDK Mode):** Perform structured domain WHOIS lookups (via RDAP) and homepage n-gram keyword density analysis (1-to-5 word phrases) with multi-language stop-word filtering. - ⚡ **Concurrency & Rotated Proxies:** Supports bulk lists processed in parallel. Utilizes unique proxy sessions per request to ensure maximum bypass success rates. --- ### ❷ Scraper Modes The Actor operates in three distinct search modes selected via the `searchType` input: 1. **`similarweb` (Traffic Data - Default):** Retrieves core website metrics, rank history, monthly visits, and traffic sources. 2. **`similar_sites` (Competitors):** Retrieves a list of related/alternative websites with similarity scores. 3. **`aitdk` (SEO & WHOIS):** Returns WHOIS registry details (creation date, expiration date, registrar, nameservers) and visible page keyword density. --- ### ❸ Input Parameters The Actor accepts the following JSON properties: | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `urls` | `array` | No | `["google.com"]` | List of webpage URLs, domains, or websites to analyze. | | `domains` | `array` | No | — | Alternative input list of domains (merged with `urls`). | | `websites` | `array` | No | — | Alternative input list of website domains (merged with `urls`). | | `searchType` | `string` | No | `"similarweb"` | Mode of scraping: `"similarweb"`, `"similar_sites"`, or `"aitdk"`. | | `include_base_data` | `boolean` | No | `true` | When enabled, returns base SimilarWeb traffic and ranking data. | | `include_similar_sites` | `boolean` | No | `false` | When enabled, includes SimilarSites competitor lists. | | `include_indepth_data` | `boolean` | No | `false` | Includes additional historical trends and company details if available. | | `output_mode` | `string` | No | `"aggregated"` | Format for output items: `"aggregated"` or `"individual"`. | | `proxyConfiguration` | `object` | No | — | Proxy configuration. Residential proxy rotation is highly recommended. | #### Sample Input JSON ```json { "urls": ["sportsdigest.in"], "searchType": "similarweb", "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ```` *** ### ❹ Output Examples #### SimilarWeb Mode (Traffic & Ranks) ```json { "url": "https://similarweb.com/website/sportsdigest.in", "name": "sportsdigest.in", "title": "Football News: Live Scores, Results & Match Updates", "description": "Football news, live scores, match results...", "category": "sports/sports", "icon": "https://site-images.similarcdn.com/image?url=sportsdigest.in&t=2&s=1&h=47c9aefee98a6b...", "previewDesktop": "https://site-images.similarcdn.com/image?url=sportsdigest.in&t=1&s=1&h=47c9aefee98a6b...", "previewMobile": "https://site-images.similarcdn.com/image?url=sportsdigest.in&t=4&s=1&h=47c9aefee98a6b...", "globalRank": { "rank": 441621 }, "countryRank": { "countryCode": "IN", "rank": 36173 }, "categoryRank": { "category": "Sports/Sports", "rank": 135 }, "engagements": { "visits": 49385, "timeOnSite": 371.7, "pagePerVisit": 9.62, "bounceRate": 0.33 }, "trafficSources": { "direct": 0.58, "referrals": 0.08, "mail": 0.01, "social": 0.03, "search": 0.29, "paidReferrals": 0 }, "topKeywords": [ { "name": "hardik pandya trade to csk", "estimatedValue": 970, "volume": 980 } ], "topCountries": [ { "countryCode": "IN", "countryName": "India", "countryUrlCode": "india", "visitsShare": 0.85 } ], "estimatedMonthlyVisits": { "2026-03-01": 61051, "2026-04-01": 54721, "2026-05-01": 49385 }, "scrapedAt": "2026-06-21T13:38:31.361Z", "snapshotDate": "2026-05-01T00:00:00+00:00" } ``` #### Similar Sites Mode (Competitors) ```json { "SiteName": "sportsdigest.in", "SimilarSites": [ { "Site": "iplt20.com", "TotalVisits": 1353414.01, "CategoryRank": null, "TopCountryRank": 5418, "Description": "The official IPL website with live scores, match updates..." } ] } ``` #### AITDK Mode (WHOIS + Keyword Density) ```json { "SiteName": "sportsdigest.in", "Whois": { "registrationDate": "2021-03-07T03:46:41.533Z", "expirationDate": "2028-03-07T03:46:41.533Z", "registrar": "GoDaddy", "registrantCountry": "IN", "nameservers": ["samara.ns.cloudflare.com", "porter.ns.cloudflare.com"], "status": ["client update prohibited", "client delete prohibited"] }, "KeywordDensity": { "1word": [ { "keyword": "sports", "count": 17, "total": 803, "density": 2.12 } ], "2words": [ { "keyword": "world cup", "count": 10, "total": 1013, "density": 0.99 } ] } } ``` *** ### ❺ Pricing & Monetization This Actor is configured with custom pay-per-event pricing on the Apify Store: - **Custom Event (`processed-item`):** Charged once per successfully scraped domain. - **Compute & Data Transfer:** Billed in standard platform usage rates. - **Apify Proxy:** Residential proxy data transmission usage is billed separately per gigabyte. *** # Actor input Schema ## `urls` (type: `array`): List of domains, websites, or webpage URLs to analyze (e.g. google.com, https://facebook.com). ## `domains` (type: `array`): Alternative array for domain list. Will be merged with URLs. ## `websites` (type: `array`): Alternative array for website list. Will be merged with URLs. ## `searchType` (type: `string`): Choose between SimilarWeb Traffic, Similar Sites (competitors), or AITDK (WHOIS + Keyword Density). ## `include_base_data` (type: `boolean`): Include standard traffic statistics, global and category ranks. ## `include_similar_sites` (type: `boolean`): Include competitor sites from SimilarSites.com. ## `include_indepth_data` (type: `boolean`): Include additional in-depth details such as company information and historical traffic trends. ## `output_mode` (type: `string`): Aggregated format (combine data in single item) or Individual (one item per record). ## `proxyConfiguration` (type: `object`): Proxy settings. Residential proxies are highly recommended to prevent blocks. ## Actor input object example ```json { "urls": [ "google.com" ], "searchType": "similarweb", "include_base_data": true, "include_similar_sites": false, "include_indepth_data": false, "output_mode": "aggregated" } ``` # Actor output Schema ## `results` (type: `string`): Dataset items from this run. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "google.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("morph_coder/similarweb-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["google.com"] } # Run the Actor and wait for it to finish run = client.actor("morph_coder/similarweb-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "google.com" ] }' | apify call morph_coder/similarweb-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=morph_coder/similarweb-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Similarweb Scraper", "description": "Extract website traffic data, traffic source breakdowns, monthly visits, similar sites, WHOIS info, and on-page keyword density for any domain.", "version": "0.0", "x-build-id": "Y5lpmdEHJ0bGW9aik" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/morph_coder~similarweb-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-morph_coder-similarweb-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/morph_coder~similarweb-scraper/runs": { "post": { "operationId": "runs-sync-morph_coder-similarweb-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/morph_coder~similarweb-scraper/run-sync": { "post": { "operationId": "run-sync-morph_coder-similarweb-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "urls": { "title": "URLs / Domains", "type": "array", "description": "List of domains, websites, or webpage URLs to analyze (e.g. google.com, https://facebook.com).", "items": { "type": "string" } }, "domains": { "title": "Domains (Alternative)", "type": "array", "description": "Alternative array for domain list. Will be merged with URLs.", "items": { "type": "string" } }, "websites": { "title": "Websites (Alternative)", "type": "array", "description": "Alternative array for website list. Will be merged with URLs.", "items": { "type": "string" } }, "searchType": { "title": "Search Mode", "enum": [ "similarweb", "similar_sites", "aitdk" ], "type": "string", "description": "Choose between SimilarWeb Traffic, Similar Sites (competitors), or AITDK (WHOIS + Keyword Density).", "default": "similarweb" }, "include_base_data": { "title": "Include Base Data", "type": "boolean", "description": "Include standard traffic statistics, global and category ranks.", "default": true }, "include_similar_sites": { "title": "Include Similar Sites", "type": "boolean", "description": "Include competitor sites from SimilarSites.com.", "default": false }, "include_indepth_data": { "title": "Include In-Depth Data", "type": "boolean", "description": "Include additional in-depth details such as company information and historical traffic trends.", "default": false }, "output_mode": { "title": "Output Mode", "enum": [ "aggregated", "individual" ], "type": "string", "description": "Aggregated format (combine data in single item) or Individual (one item per record).", "default": "aggregated" }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy settings. Residential proxies are highly recommended to prevent blocks." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```