SimilarWeb Website Scraper - Traffic, WHOIS & Ranking

SimilarWeb traffic, rankings, WHOIS and keyword density for any domain

Pull SimilarWeb traffic metrics, global and country rankings, competitor lists, RDAP WHOIS, and 1-to-5-word keyword density for any domain. No login, no contract, no Partner-API tier.

$1 per 1,000 results until July 28, 2026, then $3 per 1,000 ($5 per 1,000 on the free plan). Plus a small Actor start fee ($0.005 per GB of memory). Failed or empty lookups are free.

Two modes in one actor:

• traffic - SimilarWeb metrics: ranks with month-over-month changes, visits, engagement, traffic sources, top countries, keywords, competitors, 3-month trend.
• domainAnalysis - RDAP WHOIS + keyword density from the homepage HTML.

Why this scraper

• 💰 $1 per 1,000 results. Flat per-row pricing, both modes bill the same, plus a small Actor start fee per run.
• 📈 Rank changes and trends. Global, country, and category rank each come with the month-over-month change, plus a 3-month visit trend.
• 🥊 Competitor list included. Each traffic row ships the domain's closest competitors as ranked by SimilarWeb.
• ✅ GA-verified flag. dataSource: "ga-verified" when SimilarWeb's data is backed by a Google Analytics integration, "estimated" otherwise. Nobody else exposes this.
• 📝 Bulk WHOIS without an API key. RDAP auto-routes by TLD: .com, .io, .net, country TLDs, and most others.
• 🔍 1-to-5-word keyword density. N-gram phrase frequency on any homepage with English stopword filtering.
• 🛡️ Block-resilient. Anti-bot blocks are detected and the request retries on a fresh IP. One bad domain never kills the batch.
• 🔄 Migration-safe. Long runs survive server migrations; already-processed domains are never fetched or billed twice.

What you get

📊 Global / country / category rank	📈 3‑month visit history	🎯 Bounce rate, pages/visit, duration	🔗 Traffic source split
🌍 Top countries with share change	🔑 Top 5 keywords + CPC	🥊 Competitor domains	📉 Month-over-month rank changes
✅ GA‑verified flag	📝 WHOIS via RDAP	🧮 1‑to‑5‑word keyword density	📸 Screenshot URL + isSmall

Traffic mode fields

Field	Type	What it is
domain, siteName, title, description	string	Identity
globalRank, countryRank, categoryRank, globalCategoryRank	int / object	Ranking signals
category	string	SimilarWeb category
totalVisits	number	Monthly visit volume
globalRankChange, countryRankChange, categoryRankChange	int	Month-over-month rank movement
bounceRate, pagesPerVisit, avgVisitDuration, engagementMonth	number / string	Engagement metrics
trafficSources	object	Direct, search, social, referrals, paid, mail (fractions ≈ 1.0)
trafficSourcesRanked	array	Same split, ranked by share
topCountries, topCountriesDetailed	array	Country code, share; detailed adds share change
topKeywords	array (max 5)	Keyword, CPC
competitors	array	Closest competitor domains with icons
monthlyTrend	array	3-month visit trend with percentage change
company	object	Company info when SimilarWeb exposes it
aiTrafficDetails, aiChatbotsRanked	null	Kept for backward compatibility; currently always null (see Limitations)
dataSource	enum	"ga-verified" or "estimated"
serverNotice, largeScreenshot, snapshotDate, isSmall	misc	Side data
_meta, _error	object / string	Forward compatibility + failure reason

Domain analysis mode fields

Field	Type	What it is
domain	string	Normalized input
whois	object	registrar, createdDate, updatedDate, expiresDate, registrantOrg, registrantCountry, nameServers
whoisError	string	"rdap_not_found", "rdap_rate_limited", "rdap_unreachable"
keywordDensity	object	{ "1": [...], "2": [...], ..., "5": [...] }. Each entry has ngram, count, frequency
keywordDensityError	string	"html_fetch_failed", "empty_body", "cloudflare_blocked", etc.
htmlFetchedBytes	int	Bytes pulled (capped at 1 MB)
htmlFetchProxyTier	string	Which tier landed the body ("user", "direct", "datacenter", "residential")
_error	string	Set when every configured subtask failed

How to scrape SimilarWeb

1. Create a free Apify account. 30 seconds, no card.
2. Open the SimilarWeb Scraper in the Apify Console.
3. Paste your domains. https://, www., and trailing slashes are stripped for you.
4. Click Start. Traffic mode loads each domain like a real visitor, roughly 1 to 2 domains per minute at default settings; domainAnalysis mode is much faster.
5. Export the dataset as JSON, CSV, or Excel, or fetch via API.

Smart monthly cache

SimilarWeb publishes its public data once a month, so fetching the same domain twice in a month returns identical numbers. The scraper uses a shared monthly cache: if a domain was already fetched this month, you get the same row back in about a second instead of a full page load. Rows served this way carry _servedFromCache: true, and a built-in freshness guard re-fetches live the moment SimilarWeb publishes a new monthly snapshot. Set forceFresh: true to always bypass the cache.

Proxy options: Apify Proxy or bring your own

By default the scraper uses Apify Proxy and you pay nothing extra for it. Two things make it work on tough domains with no setup from you.

• Residential routing. Traffic mode loads SimilarWeb through residential proxies, the only way SimilarWeb serves clean data. Domain-analysis HTML fetch falls back from datacenter to residential when a homepage bot-detects.
• Block detection. Anti-bot responses are recognised and the domain retries on a fresh IP. Persistent blocks return a row with an _error field instead of crashing the run.

Bring your own proxies if you already have a residential or ISP plan:

"proxyConfiguration": {
  "useApifyProxy": false,
  "proxyUrls": [
    "http://user:pass@proxy-a.example.com:8080",
    "http://user:pass@proxy-b.example.com:8080"
  ]
}

Your URLs are tried before Apify's tiers, so you only pay for Apify bandwidth when your pool gets blocked. Multiple URLs are rotated per session.

How much does it cost

Pay-per-result. $1 per 1,000 results until July 28, 2026. From that date: $3 per 1,000 on paid Apify plans, $5 per 1,000 on the free plan. Both modes bill the same rate, plus a small Actor start fee ($0.005 per GB of memory, about $0.01 per run at default settings). No charge for failed or empty results.

• Apify Free plan ($5/month credit): about 5,000 results/month.
• Apify Starter plan ($29/month): about 29,000 results/month.

Platform compute and proxy are on us, not your bill.

Input

Both modes share domains + maxItems + maxConcurrency + proxyConfiguration. Mode-specific flags toggle the rest.

{
  "mode": "traffic",
  "domains": ["google.com", "amazon.com", "github.com"],
  "maxItems": 100,
  "maxConcurrency": 2,
  "proxyConfiguration": { "useApifyProxy": true }
}

For domainAnalysis:

{
  "mode": "domainAnalysis",
  "domains": ["github.com", "stripe.com"],
  "includeWhois": true,
  "includeKeywordDensity": true,
  "keywordDensityNGrams": [1, 2, 3, 4, 5],
  "keywordDensityTopN": 50
}

Field	Type	Default	Note
mode	enum	"traffic"	"traffic" or "domainAnalysis". Strict: typos fail at the gateway.
domains	string[]	prefilled sample	Each item ≤253 chars. Schemes and www. stripped automatically.
maxItems	int	none	Caps the number of rows processed.
maxConcurrency	int	2	Domains processed in parallel. Traffic mode caps at 4 (real browser per domain); domainAnalysis can go up to 15.
includeWhois	bool	true	DomainAnalysis mode. RDAP lookup via rdap.org.
includeKeywordDensity	bool	true	DomainAnalysis mode. Fetches the homepage (≤1MB) and tokenizes.
keywordDensityNGrams	int[]	[1,2,3,4,5]	Sizes to compute, each between 1 and 8.
keywordDensityTopN	int	50	Top N n‑grams returned per size.
proxyConfiguration	object	Apify Proxy	Traffic mode uses residential; domainAnalysis falls back from datacenter to residential. Set useApifyProxy: false and pass proxyUrls to bring your own.

Output

You can download the dataset in JSON, HTML, CSV, or Excel, or stream it through the Apify API.

Traffic mode sample row (google.com)

{
  "domain": "google.com",
  "siteName": "google.com",
  "description": "Search the world's information, including webpages, images, videos and more.",
  "globalRank": 1,
  "globalRankChange": 0,
  "countryRank": { "country": "US", "rank": 1 },
  "countryRankChange": 0,
  "categoryRank": { "rank": 1, "category": "Search Engines" },
  "categoryRankChange": 0,
  "totalVisits": 86850607710,
  "bounceRate": 0.282,
  "pagesPerVisit": 8.71,
  "avgVisitDuration": 614,
  "engagementMonth": "2026-06",
  "trafficSources": {
    "direct": 0.925, "search": 0.008, "social": 0.029,
    "referrals": 0.017, "paidReferrals": 0.008, "mail": 0.008
  },
  "trafficSourcesRanked": [
    { "source": "direct", "rank": 1, "percentage": 0.925 }
  ],
  "topCountries": [
    { "countryCode": "US", "share": 0.244 },
    { "countryCode": "JP", "share": 0.056 }
  ],
  "topCountriesDetailed": [
    { "countryCode": "US", "countryUrlCode": "united-states", "share": 0.244, "shareChange": 0.01 }
  ],
  "topKeywords": [
    { "keyword": "gemini", "cpc": 0.24 }
  ],
  "competitors": [
    { "domain": "bing.com", "icon": "https://site-images.similarcdn.com/..." }
  ],
  "monthlyTrend": [
    { "date": "2026-04-01T00:00:00", "percentageValue": 22.77, "percentageChange": 1.2 }
  ],
  "company": { "name": null, "yearFounded": null },
  "aiTrafficDetails": null,
  "aiChatbotsRanked": null,
  "dataSource": "estimated",
  "snapshotDate": "2026-06-01T00:00:00+00:00",
  "isSmall": false,
  "_meta": { "schemaVersion": null, "policy": null },
  "_error": null
}

Domain analysis mode sample row (github.com)

{
  "domain": "github.com",
  "whois": {
    "registrar": "MarkMonitor Inc.",
    "createdDate": "2007-10-09T18:20:50Z",
    "updatedDate": "2024-09-07T09:16:32Z",
    "expiresDate": "2026-10-09T18:20:50Z",
    "registrantOrg": null,
    "registrantCountry": null,
    "nameServers": ["dns1.p08.nsone.net", "ns-421.awsdns-52.com"]
  },
  "whoisError": null,
  "keywordDensity": {
    "1": [
      { "ngram": "github", "count": 53, "frequency": 0.0525 },
      { "ngram": "code", "count": 25, "frequency": 0.0248 }
    ],
    "2": [
      { "ngram": "explore github", "count": 10, "frequency": 0.0099 },
      { "ngram": "github copilot", "count": 8, "frequency": 0.0079 }
    ],
    "3": [
      { "ngram": "github advanced security", "count": 3, "frequency": 0.003 }
    ]
  },
  "keywordDensityError": null,
  "htmlFetchedBytes": 566856,
  "htmlFetchProxyTier": "user",
  "_error": null
}

Use cases

• Competitive analysis and SEO audit: compare global rank, rank changes, top keywords, and traffic sources across competitor domains.
• Competitor discovery: every traffic row includes the domain's closest competitors as ranked by SimilarWeb.
• Lead generation and sales intelligence: enrich CRM records with traffic volume, top keywords, and WHOIS contact metadata.
• Domain investment research: pair WHOIS expiration dates with traffic trends to spot dropping or undervalued domains.
• Marketing budget allocation: break down traffic source share (direct, search, social, paid, referrals, mail) to decide where to spend.
• On-page content audit: use 1-to-5-word keyword density to check stuffing, content relevance, and phrase frequency on any homepage.
• Growth tracking: the 3-month visit trend and month-over-month rank changes show whether a domain is growing or shrinking.

Limitations

• AI referral data is currently unavailable. SimilarWeb removed AI chatbot referral data from its public pages, so aiTrafficDetails and aiChatbotsRanked return null. The fields stay in the output for backward compatibility and will be repopulated if the data comes back.
• Traffic mode is slower than before. SimilarWeb now requires loading the full page like a real visitor, roughly 1 to 2 domains per minute at default settings. Failed or empty lookups are still free.
• Exact keyword volumes are not exposed on the public page; topKeywords returns keyword and CPC.
• Cloudflare-protected homepages block the HTML fetch. Keyword density returns keywordDensityError: "cloudflare_blocked" for sites like wsj.com; the WHOIS portion still works.
• topCountries and topKeywords are capped at 5 by SimilarWeb's public payload.
• Privacy-protected WHOIS records return null for registrantOrg and registrantCountry; that's the registrar redacting, not a scraper bug.
• Keyword density runs on the homepage only (max 1 MB of HTML). Sub-page audits aren't supported.

FAQ

How much does this SimilarWeb scraper cost?

Pay-per-result. You pay $1 per 1,000 results until July 28, 2026, then $3 per 1,000 on paid plans ($5 per 1,000 on the free plan), plus a small Actor start fee (about $0.01 per run at default settings), and only when we actually return data. No charge for failed or empty lookups.

No subscription lock-in. Pause whenever.

Is it legal to scrape SimilarWeb?

Scraping publicly accessible pages is generally allowed in the US and most of the EU, as long as you don't collect personal data covered by GDPR or CCPA without a lawful basis. This actor only touches public endpoints, but how you use the output is on you.

Apify's full breakdown: Is web scraping legal?.

Can I integrate the SimilarWeb scraper with other tools?

Push results into Make, Zapier, Slack, Airbyte, GitHub, Google Sheets, Google Drive, and more. Apify treats every actor as a webhook source, so anything that consumes webhooks or pulls from an API works.

Full list: Apify integrations.

Can I run the SimilarWeb scraper through the Apify API?

Yes. Every run is available via the Apify REST API:

curl -X POST "https://api.apify.com/v2/acts/sourabhbgp~similarweb-scraper/runs?token=APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"traffic","domains":["google.com","amazon.com"]}'

Docs: Apify API reference.

Can I use this SimilarWeb scraper through an MCP Server?

Yes. Apify ships an MCP server that exposes every actor as a tool, so Claude Desktop, Cursor, and any other MCP-capable client can call this scraper. Setup: Apify MCP docs.

Your feedback

Bug, missing field, or odd behavior? Drop a note in the Issues tab. Reports go to a human and fixes usually ship the same week.