# Similarweb Scraper (`vortex_data/similarweb-scraper`) Actor โšก Get Similarweb data for any domain in seconds โ€” traffic, ranks, top keywords, AI traffic share (ChatGPT/Claude/Gemini), similar sites, company info & WHOIS. No login or API key. Bulk parallel scrape, captcha-resilient. Export to JSON/CSV/Excel. Perfect for SEO, lead gen & competitor analysis. - **URL**: https://apify.com/vortex\_data/similarweb-scraper.md - **Developed by:** [VortexData](https://apify.com/vortex_data) (community) - **Categories:** Lead generation, SEO tools, Automation - **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 1 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 domain analyseds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## ๐Ÿ” Similarweb Scraper > ๐Ÿ“Š **Website intelligence for any domain in seconds.** Global and > country rankings, engagement metrics, traffic sources, top keywords, > AI traffic share, similar sites and WHOIS โ€” all in one parallel run, > exported as JSON, > CSV, Excel or any other format Apify supports. ### ๐Ÿ’Ž What is Similarweb Scraper? Similarweb Scraper is a fast, captcha-resilient web scraper that pulls the same data the Similarweb web app shows you โ€” without requiring a Similarweb account, login, or API key. Behind the scenes it talks to Similarweb's own SPA data endpoint using a real Chrome TLS / JA3 fingerprint via [`curl_cffi`](https://github.com/lexiforest/curl_cffi) and routes every request through a fresh Apify Residential proxy session, so you get reliable, production-grade data for any domain. You give it a list of domains and pick which datasets you want. The Actor returns one merged record per domain (or one record per dataset per domain, your choice) ready to drop into a spreadsheet, BI tool, warehouse, or AI agent. ### ๐Ÿš€ What can Similarweb Scraper do? - ๐Ÿ—‚๏ธ **Scrape three independent datasets** in a single run: - ๐Ÿ“Š **Base data** โ€” global / country / category ranks, monthly visits, bounce rate, pages per visit, time on site, traffic-source split (direct, search, referral, social, paid, mail), top organic keywords with volume and CPC, AI traffic share per LLM (ChatGPT / Claude / Gemini / Perplexity / Copilot). - ๐Ÿชž **Similar sites** โ€” competitors and alternatives with their traffic, category and ranking. - ๐Ÿ†” **AITDK** โ€” WHOIS via RDAP (registrar, registration / expiration dates, name servers, EPP status, DNSSEC) plus on-page keyword density analysis of the domain's homepage. - โšก **Process domains in parallel** โ€” up to 10 concurrent fetches by default, each with isolated retry budgets so a flaky stream cannot drain attempts of another. - ๐Ÿ›ก๏ธ **Captcha-resilient** โ€” uses Similarweb's open SPA endpoint that serves `200 OK` to Chrome TLS fingerprints, no captcha solving required for base data and similar sites. - ๐Ÿ”„ **Per-request IP rotation** โ€” every HTTP call gets a fresh Apify Residential proxy session, so a blocked address costs at most one attempt. - ๐ŸŒ **Three input formats** โ€” accepts `example.com`, `www.example.com`, or `https://example.com`. The domain is extracted automatically. - ๐Ÿ“ค **Two output shapes** โ€” *aggregated* (one merged record per domain) or *individual* (one record per dataset). ### โ˜๏ธ Remember the Apify platform Running this Actor on Apify gives you everything that comes with the platform out of the box: managed **Residential proxies** with global exit IPs, **scheduling** (run hourly / daily / weekly), free storage in **Apify Datasets** with export to JSON / CSV / Excel / JSONL / XML / RSS, **webhooks and integrations** (Make, Zapier, n8n, Google Sheets, Slack, Airtable, Pipedream), and a **REST API + Python / JavaScript SDKs** to plug results into your own pipelines. ### ๐Ÿ—๏ธ What data can this Actor extract? | Field group | Examples | | ------------------- | ------------------------------------------------------------------------------------------------- | | Rankings | Global rank ยท country rank ยท category rank | | Engagement | Total visits ยท monthly visits (3 months) ยท bounce rate ยท pages / visit ยท time on site | | Traffic sources | Direct ยท search ยท referral ยท social ยท paid ยท mail (as shares) | | AI traffic share | ChatGPT ยท Claude ยท Gemini ยท Perplexity ยท Copilot โ€” current + 3-month history | | Top keywords | Keyword ยท estimated value ยท search volume ยท CPC | | Country breakdown | Top countries with share + monthly visit estimates per country | | Similar sites | Up to 20 related sites with traffic, ranks, category and thumbnails | | WHOIS (via RDAP) | Registrar ยท IANA ID ยท registration / expiration / last-changed dates ยท name servers ยท EPP status | | Keyword density | Top-20 non-stopword tokens from the homepage with count and density | | Assets | Desktop / mobile screenshots ยท favicon | ### ๐ŸŽฏ How to use Similarweb Scraper 1. Click **Try for free** on the Actor's Apify Store page. 2. In the **Domains** field, paste the list of websites you want to analyse โ€” one per line, any format (`example.com`, `www.example.com`, or `https://example.com`). 3. Pick exactly one **Dataset to fetch**. Start with `base_data` if you just want the standard Similarweb dashboard data. 4. Choose an **Output shape**: *Aggregated* if you'll open the results in a spreadsheet, *Individual* if you'll join the streams downstream. 5. Click **Start**. Each domain costs about half a second of compute on the base endpoint; AITDK takes longer because it scrapes each homepage. 6. When the run finishes, open **Storage โ†’ Dataset** and export to JSON, CSV, Excel, JSONL, XML or RSS. Or pull the results through the API: `https://api.apify.com/v2/datasets/{dataset_id}/items`. ### ๐Ÿ’ฐ Pricing This Actor is billed by Apify **compute units (CU)** consumed and **Apify Proxy traffic** used โ€” see Apify's [platform pricing](https://apify.com/pricing). Typical CU consumption per run: | Datasets selected | Approx. CU per 100 domains | | ----------------------------- | -------------------------- | | `base_data` only | ~0.01 CU | | `base_data` + `similar_sites` | ~0.02 CU | | `base_data` + `aitdk` | ~0.05 CU (homepage fetch) | | All three datasets | ~0.06 CU | Costs scale roughly linearly with the number of domains. The Apify free tier (5 USD / month) covers thousands of base-data lookups. ### ๐Ÿ“ฅ Input The form has three fields only โ€” everything else has sensible defaults: | Field | Type | Default | | -------------- | -------------- | ------------------------------------------------------------------------ | | `domains` | array | **required** | | `datasets` | enum | `base_data` โ€” one of `base_data`, `similar_sites`, `aitdk` | | `output_mode` | enum | `aggregated` (or `individual`) | #### Example input ```json { "domains": ["openai.com", "apple.com", "reddit.com"], "datasets": "base_data", "output_mode": "aggregated" } ```` ### ๐Ÿ“ค Output Each item conforms to the dataset schema and is rendered in the Apify Console as five tables: **๐Ÿ“Š Overview ยท ๐Ÿšฆ Traffic sources ยท ๐Ÿ’ซ Engagement ยท ๐Ÿค– AI traffic share ยท ๐Ÿ†” AITDK (WHOIS + keywords)**. #### Example item (aggregated, abridged) ```json { "domain": "openai.com", "types_included": ["base_data", "aitdk"], "rankGlobal": 207, "country": "US", "countryRank": 306, "category": "ai_chatbots_and_tools", "categoryRank": 6, "title": "OpenAI", "totalVisits": 195737812, "bounceRate": 0.5937, "pagesPerVisit": 2.59, "timeOnSite": 138.72, "socialTraffic": 0.0287, "searchTraffic": 0.2154, "directTraffic": 0.3840, "referralTraffic": 0.1038, "aiTrafficShareChatgpt": 0.8825, "aiTrafficShareClaude": 0.0029, "aiTrafficShareGemini": 0.0106, "topKeywords": [ {"keyword": "chatgpt", "estimatedValue": 20907500.0, "searchVolume": 173339160.0, "cpc": 0.14}, {"keyword": "chat gpt", "estimatedValue": 5688810.0, "searchVolume": 95011780.0, "cpc": 0.14} ], "aitdk_data": { "whois_registrar": "MarkMonitor Inc.", "whois_registration_date": "2007-01-19T19:28:24Z", "whois_expiration_date": "2029-01-19T19:28:24Z", "whois_name_servers": ["ns1-02.azure-dns.com", "ns2-02.azure-dns.net"], "keyword_density_total_words": 283, "keyword_density": [ {"keyword": "chatgpt", "count": 14, "density": 0.0495}, {"keyword": "research", "count": 13, "density": 0.0459} ] } } ``` ### ๐Ÿ”— Integrate Similarweb Scraper anywhere Apify Actors run on a REST API โ€” every run, dataset and webhook is addressable from your code: ```bash ## Trigger a run from anywhere curl -X POST "https://api.apify.com/v2/acts/~similarweb-scraper/runs?token=" \ -H "Content-Type: application/json" \ -d '{"domains": ["openai.com"], "datasets": ["base_data"]}' ## Read results from the run's default dataset curl "https://api.apify.com/v2/datasets//items?format=json" ``` Or use the [official Python](https://docs.apify.com/api/client/python/) and [JavaScript](https://docs.apify.com/api/client/js/) clients. ### โ“ FAQ **๐Ÿ”‘ Do I need a Similarweb account or API key?** No. This Actor talks to Similarweb's public SPA endpoint directly. No login, no API key, and no scraping the captcha-gated in-depth pages. **๐Ÿ†• Is the data fresh?** Yes โ€” it's the same JSON Similarweb's UI loads. The `snapshotDate` field on every record tells you exactly which month it represents. Similarweb refreshes its traffic data monthly. **โฐ Can I run this on a schedule?** Yes โ€” open the Actor in Apify Console, go to **Schedules** and pick hourly, daily, weekly or a custom cron. Combine with [webhooks](https://docs.apify.com/platform/integrations/webhooks) to push fresh data into Google Sheets, Slack, Make, Zapier or your own backend automatically. **โš–๏ธ Is web scraping legal?** Public web pages are generally legal to scrape, but you must respect copyright, terms of service, and personal-data protection laws (GDPR in the EU and similar regulations elsewhere). This Actor only extracts publicly visible data โ€” no personal data is collected. See Apify's [legal blog](https://blog.apify.com/is-web-scraping-legal/) for details. ### ๐Ÿ’ฌ Support - Found a bug or have a feature request? Open an issue in the Actor's Issues tab on Apify Console. - Questions about Apify itself? Visit [docs.apify.com](https://docs.apify.com) or the [Apify Discord community](https://discord.com/invite/jyEM2PRvMU). ### ๐Ÿ“ Changelog See [`CHANGELOG.md`](CHANGELOG.md) for the full release history. The Actor follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html). # Actor input Schema ## `domains` (type: `array`): Websites to analyse. Accepts any of these formats: `example.com`, `www.example.com`, `https://example.com`. The domain is extracted automatically. ## `datasets` (type: `string`): Pick exactly one dataset to fetch. ## `output_mode` (type: `string`): How the streams are written to the dataset. ## Actor input object example ```json { "domains": [ "openai.com", "apple.com", "reddit.com", "github.com", "stackoverflow.com", "google.com", "youtube.com", "twitter.com", "linkedin.com", "amazon.com" ], "datasets": "base_data", "output_mode": "aggregated" } ``` # Actor output Schema ## `dataset` (type: `string`): JSON / CSV / XLSX / JSONL / XML / RSS export of every record pushed during the run. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "openai.com", "apple.com", "reddit.com", "github.com", "stackoverflow.com", "google.com", "youtube.com", "twitter.com", "linkedin.com", "amazon.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("vortex_data/similarweb-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": [ "openai.com", "apple.com", "reddit.com", "github.com", "stackoverflow.com", "google.com", "youtube.com", "twitter.com", "linkedin.com", "amazon.com", ] } # Run the Actor and wait for it to finish run = client.actor("vortex_data/similarweb-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "openai.com", "apple.com", "reddit.com", "github.com", "stackoverflow.com", "google.com", "youtube.com", "twitter.com", "linkedin.com", "amazon.com" ] }' | apify call vortex_data/similarweb-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=vortex_data/similarweb-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Similarweb Scraper", "description": "โšก Get Similarweb data for any domain in seconds โ€” traffic, ranks, top keywords, AI traffic share (ChatGPT/Claude/Gemini), similar sites, company info & WHOIS. No login or API key. Bulk parallel scrape, captcha-resilient. Export to JSON/CSV/Excel. Perfect for SEO, lead gen & competitor analysis.", "version": "0.3", "x-build-id": "N1u8iJl8tpBfBpbxq" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/vortex_data~similarweb-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-vortex_data-similarweb-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/vortex_data~similarweb-scraper/runs": { "post": { "operationId": "runs-sync-vortex_data-similarweb-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/vortex_data~similarweb-scraper/run-sync": { "post": { "operationId": "run-sync-vortex_data-similarweb-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "domains" ], "properties": { "domains": { "title": "๐ŸŒ Domains", "minItems": 1, "type": "array", "description": "Websites to analyse. Accepts any of these formats: `example.com`, `www.example.com`, `https://example.com`. The domain is extracted automatically.", "items": { "type": "string" } }, "datasets": { "title": "๐Ÿ“ฆ Datasets to fetch", "enum": [ "base_data", "similar_sites", "aitdk" ], "type": "string", "description": "Pick exactly one dataset to fetch.", "default": "base_data" }, "output_mode": { "title": "๐Ÿ“ค Output shape", "enum": [ "aggregated", "individual" ], "type": "string", "description": "How the streams are written to the dataset.", "default": "aggregated" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```