SimilarWeb Website Scraper - Traffic, WHOIS & Ranking

SimilarWeb traffic, rankings, WHOIS and keyword density for any domain

Pull SimilarWeb traffic metrics, global and country rankings, AI chatbot referrals, RDAP WHOIS, and 1-to-5-word keyword density for any domain. No login, no contract, no Partner-API tier.

$3 per 1,000 results on paid Apify plans, $5 per 1,000 on the free plan. Plus a small Actor start fee ($0.005 per GB of memory). Failed or empty lookups are free.

Two modes in one actor:

• traffic - SimilarWeb metrics: global, country and category rank, 3 months of visits, engagement, the full traffic-source split, top countries, keywords with search volume and CPC, and AI chatbot referrals.
• domainAnalysis - RDAP WHOIS + keyword density from the homepage HTML.

Why this scraper

• 💰 $3 per 1,000 results. Flat per-row pricing, both modes bill the same, plus a small Actor start fee per run.
• 🤖 AI chatbot traffic, broken down. See how many visits a site gets from ChatGPT, Perplexity, Gemini and the rest, ranked by share, with 3 months of history per chatbot.
• 🔑 Keyword volume and CPC. Top keywords ship with monthly search volume and cost per click, not just the keyword string.
• 📊 The whole traffic split, not just the biggest slice. Direct, organic search, paid search, organic social, paid social, referrals, mail, display ads, affiliate and AI, each with its real percentage.
• ✅ GA-verified flag. dataSource: "ga-verified" when SimilarWeb's data is backed by a Google Analytics integration, "estimated" otherwise. Nobody else exposes this.
• 📝 Bulk WHOIS without an API key. RDAP auto-routes by TLD: .com, .io, .net, country TLDs, and most others.
• 🔍 1-to-5-word keyword density. N-gram phrase frequency on any homepage with English stopword filtering.
• 🚫 Typos are free. A domain SimilarWeb has no data for is never billed, even though it returns a full-looking response.
• 🔄 Migration-safe. Long runs survive server migrations; already-processed domains are never fetched or billed twice.

What you get

📊 Global / country / category rank	📈 3‑month visit history	🎯 Bounce rate, pages/visit, duration	🔗 Traffic source split
🌍 Top countries with share	🔑 Top keywords + volume + CPC	🤖 AI chatbot referrals + prompts	🖼️ Site screenshot
✅ GA‑verified flag	📝 WHOIS via RDAP	🧮 1‑to‑5‑word keyword density	🤖 AI referral share + trend

Traffic mode fields

Field	Type	What it is
domain, siteName, title, description	string	Identity
globalRank, countryRank, categoryRank	int / object	Ranking signals
category	string	SimilarWeb category
totalVisits	number	Latest month's visit volume
estimatedMonthlyVisits	object	Visits per month for the last 3 months
bounceRate, pagesPerVisit, avgVisitDuration, engagementMonth	number / string	Engagement metrics
trafficSources	object	Direct, search, social, referrals, paid, mail (fractions ≈ 1.0)
trafficSourcesRanked	array	The full ten-way split, ranked, each with its own percentage
topCountries	array	Country code, numeric country id, share
topKeywords	array (max 5)	Keyword, monthly search volume, CPC, estimated value
aiTrafficDetails	object	Total AI visits, AI share of referrals, ranked chatbots, 3-month history per chatbot, and top prompts
aiChatbotsRanked	array	Chatbots ranked by referral volume
dataSource	enum	"ga-verified" or "estimated"
snapshotDate, isSmall	misc	Side data
largeScreenshot	string	Screenshot URL for the domain
globalCategoryRank, serverNotice	null	Kept for backward compatibility; usually null
_meta, _error	object / string	Forward compatibility + failure reason

Domain analysis mode fields

Field	Type	What it is
domain	string	Normalized input
whois	object	registrar, createdDate, updatedDate, expiresDate, registrantOrg, registrantCountry, nameServers
whoisError	string	"rdap_not_found", "rdap_rate_limited", "rdap_unreachable"
keywordDensity	object	{ "1": [...], "2": [...], ..., "5": [...] }. Each entry has ngram, count, frequency
keywordDensityError	string	"html_fetch_failed", "empty_body", "cloudflare_blocked", etc.
htmlFetchedBytes	int	Bytes pulled (capped at 1 MB)
htmlFetchProxyTier	string	Which tier landed the body ("user", "direct", "datacenter", "residential")
_error	string	Set when every configured subtask failed

How to scrape SimilarWeb

1. Create a free Apify account. 30 seconds, no card.
2. Open the SimilarWeb Scraper in the Apify Console.
3. Paste your domains. https://, www., and trailing slashes are stripped for you.
4. Click Start. Both modes are fast: a batch of 8 domains finishes in about 10 seconds.
5. Export the dataset as JSON, CSV, or Excel, or fetch via API.

Smart monthly cache

SimilarWeb publishes its public data once a month, so fetching the same domain twice in a month returns identical numbers. The scraper uses a shared monthly cache: if a domain was already fetched this month, you get the same row back in about a second instead of a full page load. Rows served this way carry _servedFromCache: true, and a built-in freshness guard re-fetches live the moment SimilarWeb publishes a new monthly snapshot. Set forceFresh: true to always bypass the cache.

Proxy options: Apify Proxy or bring your own

By default the scraper uses Apify Proxy and you pay nothing extra for it. Two things make it work on tough domains with no setup from you.

• Residential routing. Both modes fetch through residential proxies, the only way SimilarWeb serves clean data. If one pool cannot get through, paid Apify plans automatically fall back to a second pool.
• Block detection. Anti-bot responses are recognised and the domain retries on a fresh IP. Persistent blocks return a row with an _error field instead of crashing the run.

Bring your own proxies if you already have a residential or ISP plan:

"proxyConfiguration": {
  "useApifyProxy": false,
  "proxyUrls": [
    "http://user:pass@proxy-a.example.com:8080",
    "http://user:pass@proxy-b.example.com:8080"
  ]
}

When you supply proxy URLs, the run uses only those addresses. We never quietly switch to a pool of ours, so a run stays on the proxy you chose and you are never billed for our bandwidth instead of yours. Multiple URLs are rotated per session. If your pool cannot load a page, that domain is reported as unavailable rather than retried elsewhere.

How much does it cost

Pay-per-result. $3 per 1,000 results on paid Apify plans, $5 per 1,000 on the free plan. Both modes bill the same rate, plus a small Actor start fee ($0.005 per GB of memory, about $0.01 per run at default settings). No charge for failed or empty results.

• Apify Free plan ($5/month credit): about 5,000 results/month.
• Apify Starter plan ($29/month): about 29,000 results/month.

Platform compute and proxy are on us, not your bill.

Input

Both modes share domains + maxItems + maxConcurrency + proxyConfiguration. Mode-specific flags toggle the rest.

{
  "mode": "traffic",
  "domains": ["google.com", "amazon.com", "github.com"],
  "maxItems": 100,
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": true }
}

For domainAnalysis:

{
  "mode": "domainAnalysis",
  "domains": ["github.com", "stripe.com"],
  "includeWhois": true,
  "includeKeywordDensity": true,
  "keywordDensityNGrams": [1, 2, 3, 4, 5],
  "keywordDensityTopN": 50
}

Field	Type	Default	Note
mode	enum	"traffic"	"traffic" or "domainAnalysis". Strict: typos fail at the gateway.
domains	string[]	prefilled sample	Each item ≤253 chars. Schemes and www. stripped automatically.
maxItems	int	none	Caps the number of rows processed.
maxConcurrency	int	2	Domains processed in parallel. Traffic mode caps at 4; domainAnalysis can go up to 15.
includeWhois	bool	true	DomainAnalysis mode. RDAP lookup via rdap.org.
includeKeywordDensity	bool	true	DomainAnalysis mode. Fetches the homepage (≤1MB) and tokenizes.
keywordDensityNGrams	int[]	[1,2,3,4,5]	Sizes to compute, each between 1 and 8.
keywordDensityTopN	int	50	Top N n‑grams returned per size.
proxyConfiguration	object	Apify Proxy	Traffic mode uses residential; domainAnalysis falls back from datacenter to residential. Set useApifyProxy: false and pass proxyUrls to bring your own.

Output

You can download the dataset in JSON, HTML, CSV, or Excel, or stream it through the Apify API.

Traffic mode sample row (stackoverflow.com)

{
  "domain": "stackoverflow.com",
  "siteName": "stackoverflow.com",
  "description": "Stack Overflow | The World's Largest Online Community for Developers",
  "globalRank": 3640,
  "countryRank": { "country": "US", "countryId": 840, "rank": 3195 },
  "categoryRank": { "rank": 82, "category": "Computers_Electronics_and_Technology/Programming_and_Developer_Software" },
  "category": "computers_electronics_and_technology/programming_and_developer_software",
  "totalVisits": 21821926,
  "estimatedMonthlyVisits": { "2026-05-01": 27252094, "2026-06-01": 23846315, "2026-07-01": 21821926 },
  "bounceRate": 0.6916238782361016,
  "pagesPerVisit": 1.75,
  "avgVisitDuration": 133.9,
  "engagementMonth": "2026-07",
  "trafficSources": {
    "direct": 0.0848, "search": 0.8442, "social": 0.0181,
    "referrals": 0.0316, "paidReferrals": 0.0079, "mail": 0.0025
  },
  "trafficSourcesRanked": [
    { "source": "SearchOrganic", "percentage": 0.8363798788872433, "rank": 1 },
    { "source": "Direct", "percentage": 0.08475899623874678, "rank": 2 }
  ],
  "topCountries": [
    { "countryCode": "US", "countryId": 840, "share": 0.22430745394081328 }
  ],
  "topKeywords": [
    { "keyword": "c", "volume": 4453010, "cpc": 0.78, "estimatedValue": 230580 }
  ],
  "aiTrafficDetails": {
    "totalAiVisits": 140441.171,
    "aiReferralShare": 0.006435788809932891,
    "aiTrafficTier": "<1M",
    "topChatbots": [ { "name": "chatgpt.com", "share": 55.23023936614536 } ],
    "chatbotTrends": [ { "name": "chatgpt.com", "history": [ { "date": "2026-05-01", "share": 52.1 } ] } ],
    "topPrompts": []
  },
  "aiChatbotsRanked": [
    { "name": "chatgpt.com", "rank": 1 },
    { "name": "perplexity.ai", "rank": 2 }
  ],
  "largeScreenshot": "https://site-images.similarcdn.com/image?url=stackoverflow.com&t=1&s=1",
  "dataSource": "estimated",
  "snapshotDate": "2026-07-01T00:00:00+00:00",
  "isSmall": false,
  "_meta": { "schemaVersion": 1, "policy": 0 },
  "_error": null
}

Domain analysis mode sample row (github.com)

{
  "domain": "github.com",
  "whois": {
    "registrar": "MarkMonitor Inc.",
    "createdDate": "2007-10-09T18:20:50Z",
    "updatedDate": "2024-09-07T09:16:32Z",
    "expiresDate": "2026-10-09T18:20:50Z",
    "registrantOrg": null,
    "registrantCountry": null,
    "nameServers": ["dns1.p08.nsone.net", "ns-421.awsdns-52.com"]
  },
  "whoisError": null,
  "keywordDensity": {
    "1": [
      { "ngram": "github", "count": 53, "frequency": 0.0525 },
      { "ngram": "code", "count": 25, "frequency": 0.0248 }
    ],
    "2": [
      { "ngram": "explore github", "count": 10, "frequency": 0.0099 },
      { "ngram": "github copilot", "count": 8, "frequency": 0.0079 }
    ],
    "3": [
      { "ngram": "github advanced security", "count": 3, "frequency": 0.003 }
    ]
  },
  "keywordDensityError": null,
  "htmlFetchedBytes": 566856,
  "htmlFetchProxyTier": "user",
  "_error": null
}

Use cases

• Competitive analysis and SEO audit: compare global rank, top keywords with search volume and CPC, and the full traffic-source split across rival domains.
• AI visibility tracking: measure how much traffic a site earns from ChatGPT, Perplexity and other assistants, and watch it move month to month.
• Lead generation and sales intelligence: enrich CRM records with traffic volume, top keywords, and WHOIS contact metadata.
• Domain investment research: pair WHOIS expiration dates with traffic trends to spot dropping or undervalued domains.
• Marketing budget allocation: break down traffic source share (direct, search, social, paid, referrals, mail) to decide where to spend.
• On-page content audit: use 1-to-5-word keyword density to check stuffing, content relevance, and phrase frequency on any homepage.
• Growth tracking: the 3-month visit trend and month-over-month rank changes show whether a domain is growing or shrinking.

Limitations

• AI referral data is currently unavailable. SimilarWeb removed AI chatbot referral data from its public pages, so aiTrafficDetails and aiChatbotsRanked return null. The fields stay in the output for backward compatibility and will be repopulated if the data comes back.
• No competitor list, company profile, or rank-change deltas. Traffic rows do not include competitor domains, SimilarWeb's company profile (employees, headquarters, revenue), month-over-month rank movement, or a relative monthly trend. You get three months of absolute visit counts instead, which most people were computing the trend from anyway.
• topPrompts is currently always empty. The field exists in aiTrafficDetails but SimilarWeb is not publishing prompt text on the public data it exposes. The chatbot names, shares and history are all populated.
• globalCategoryRank and serverNotice are usually null. They are kept in the output so existing integrations do not break on a missing key. categoryRank gives you the rank within the category.
• estimatedMonthlyVisits is reconstructed, not read directly. The public page publishes the latest month's total and each month's percentage change, but no per-month totals. Those two are combined to give absolute visits for the last 3 months. Expect small differences from figures shown inside a paid SimilarWeb account.
• Exact keyword volumes are not exposed on the public page; topKeywords returns keyword and CPC.
• Cloudflare-protected homepages block the HTML fetch. Keyword density returns keywordDensityError: "cloudflare_blocked" for sites like wsj.com; the WHOIS portion still works.
• topCountries and topKeywords are capped at 5 by SimilarWeb's public payload.
• Privacy-protected WHOIS records return null for registrantOrg and registrantCountry; that's the registrar redacting, not a scraper bug.
• Keyword density runs on the homepage only (max 1 MB of HTML). Sub-page audits aren't supported.
• A run can be blocked. SimilarWeb uses bot protection, and when it refuses a request the domain is retried on a fresh address. Domains that still could not be loaded are named in the run status rather than left for you to spot.
• You are not charged for rows we could not collect. Results are billed as they are delivered, so a domain that returned nothing never billed. A run that comes back short says so in its status message.
• Paid Apify plans get a second proxy pool. On a paid plan a blocked run automatically falls back to a second pool of addresses. On the free plan, where Apify covers the proxy cost, the run finishes with the results collected so far instead.

FAQ

How much does this SimilarWeb scraper cost?

Pay-per-result. You pay $3 per 1,000 results on paid plans ($5 per 1,000 on the free plan), plus a small Actor start fee (about $0.01 per run at default settings), and only when we actually return data. No charge for failed or empty lookups.

No subscription lock-in. Pause whenever.

Is it legal to scrape SimilarWeb?

Scraping publicly accessible pages is generally allowed in the US and most of the EU, as long as you don't collect personal data covered by GDPR or CCPA without a lawful basis. This actor only touches public endpoints, but how you use the output is on you.

Apify's full breakdown: Is web scraping legal?.

Can I integrate the SimilarWeb scraper with other tools?

Push results into Make, Zapier, Slack, Airbyte, GitHub, Google Sheets, Google Drive, and more. Apify treats every actor as a webhook source, so anything that consumes webhooks or pulls from an API works.

Full list: Apify integrations.

Can I run the SimilarWeb scraper through the Apify API?

Yes. Every run is available via the Apify REST API:

curl -X POST "https://api.apify.com/v2/acts/sourabhbgp~similarweb-scraper/runs?token=APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"traffic","domains":["google.com","amazon.com"]}'

Docs: Apify API reference.

Can I use this SimilarWeb scraper through an MCP Server?

Yes. Apify ships an MCP server that exposes every actor as a tool, so Claude Desktop, Cursor, and any other MCP-capable client can call this scraper. Setup: Apify MCP docs.

Your feedback

Bug, missing field, or odd behavior? Drop a note in the Issues tab. Reports go to a human and fixes usually ship the same week.