SimilarWeb Scraper

A high-performance Apify actor that extracts detailed SimilarWeb analytics, traffic data, and competitive intelligence for any list of domains. Get SimilarWeb rankings, engagement metrics, traffic sources, keyword data, AI traffic insights, and discover similar websites — all in one run.

SimilarWeb is one of the leading web analytics platforms, providing comprehensive data about website traffic, audience demographics, referral sources, and competitive positioning across industries.

Built for speed: all domains are processed in parallel, delivering SimilarWeb data in seconds rather than minutes.

Support: afrcanec@gmail.com

Pricing

This actor uses pay-per-use pricing (per 1,000 results). If you prefer a monthly subscription ($20/month for unlimited use), check out SimilarWeb Website Analytics instead.

Key Features

• Bulk Domain Analysis: Submit up to 50 domains at once — they are all processed simultaneously for maximum throughput.
• Rankings & Visibility: Global rank, country-specific rank, and category rank for every domain.
• Traffic & Engagement Metrics: Total visits, bounce rate, pages per visit, average session duration, and 3-month visit history.
• Traffic Source Breakdown: Percentage split across direct, search, social, referrals, email, and paid channels.
• Geographic Distribution: Top countries driving traffic to each website with share percentages.
• SEO & Keyword Intelligence: Top organic keywords with search volume, estimated value, and cost-per-click data.
• AI & Chatbot Traffic: See which AI platforms (ChatGPT, Perplexity, Claude, etc.) send traffic to a website and their distribution over time.
• Similar Sites Discovery: For every domain, get a list of related websites with their own traffic and ranking data.
• Flexible Proxy Support: Use Apify Proxy (RESIDENTIAL, DATACENTER) or bring your own HTTP proxies — automatically rotated across domains.
• Error Resilience: If one domain fails, the rest continue processing. Failed domains are logged with error details in the output.

Input

Field	Type	Required	Description
domains	string[]	Yes	List of domains to analyze, max 50 per run (e.g. google.com, facebook.com)
proxyConfiguration	object	No	Proxy settings — use Apify Proxy groups (RESIDENTIAL, DATACENTER) or your own custom proxies

Proxy Options

Configure proxies through the proxyConfiguration field in the Apify Console:

1. Apify Proxy — select a proxy group (RESIDENTIAL recommended for best results)
2. Custom proxies — add your own proxy URLs directly in the proxy configuration
3. No proxy — leave the field empty, requests go directly (works for small batches)

Handling Errors with Proxy Rotation

If you're seeing frequent failures (403, 429, or timeout errors), it's almost always a proxy issue. SimilarWeb actively blocks datacenter IPs and repeated requests from the same address.

What to do:

1. Use rotating residential proxies — this is the single most effective fix. Residential IPs rotate automatically on each request, making blocks far less likely.
2. Apify Proxy (RESIDENTIAL group) — if you don't want to source your own proxies, just select the RESIDENTIAL group in the proxy configuration. It works out of the box and handles rotation for you automatically.
3. Custom rotating proxies — if you bring your own, make sure they support automatic IP rotation (per-request or with a short TTL). Static or sticky sessions are not recommended for SimilarWeb scraping.

⚠️ If a domain fails, the actor does not stop — it logs the error and continues with the remaining domains. However, if you're seeing a high failure rate across many domains, switching to residential proxies will almost certainly fix it.

Proxy configuration examples

Apify Residential Proxy (recommended):

{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Custom rotating proxies:

{
  "proxyConfiguration": {
    "useApifyProxy": false,
    "proxyUrls": [
      "http://user:pass@proxy-provider.com:port",
      "http://user:pass@another-proxy.com:port"
    ]
  }
}

Why not DATACENTER?

Datacenter proxies are faster and cheaper, but SimilarWeb is known to aggressively block them. If you're on a budget, you can try DATACENTER first — but if you start seeing errors, switch to RESIDENTIAL.

Quick troubleshooting

Symptom	Likely cause	Fix
Most domains return 403/429	IP is blocked or rate-limited	Switch to RESIDENTIAL proxies
Occasional single-domain failures	Domain doesn't exist or has no SW data	Check the domain name; this is expected behavior
Timeouts on all domains	Proxy is down or unreachable	Verify proxy URLs or switch to Apify Proxy
Works for ~10 domains, then fails	Proxy IP is not rotating	Use per-request rotation, avoid sticky sessions

Output Fields

Each domain produces a JSON object containing:

Field	Description
SiteName	Domain name
Title	Website title
Description	Website meta description
Category	Industry category
GlobalRank	Worldwide traffic rank
CountryRank	Rank within the top traffic country
CategoryRank	Rank within industry category
Engagments	Visits, bounce rate, pages/visit, session duration
TrafficSources	% split by channel (direct, search, social, etc.)
TopCountryShares	Top 5 countries with traffic share %
EstimatedMonthlyVisits	Last 3 months of visit data
TopKeywords	Top organic keywords with volume and CPC
AiTrafficDetails	AI chatbot traffic sources and distribution
Competitors	Top similarity competitors
SimilarSites	Related websites with traffic and ranking data

Example Output

{
  "SiteName": "google.com",
  "Title": "Google",
  "Description": "Learn about the Certified Publisher Program...",
  "Category": "computers_electronics_and_technology/search_engines",
  "GlobalRank": {
    "Rank": 1
  },
  "CountryRank": {
    "Country": 840,
    "CountryCode": "US",
    "Rank": 1
  },
  "CategoryRank": {
    "Rank": "1",
    "Category": "Computers_Electronics_and_Technology/Search_Engines"
  },
  "Engagments": {
    "Visits": 85756574615,
    "VisitsFormatted": "85.76B",
    "BounceRate": 28.36,
    "PagePerVisit": 8.52,
    "TimeOnSite": 610,
    "Month": "1",
    "Year": "2026"
  },
  "EstimatedMonthlyVisits": {
    "2025-11-01": 82284033110,
    "2025-12-01": 84172772881,
    "2026-01-01": 85756574615
  },
  "EstimatedMonthlyVisitsFormatted": {
    "2025-11-01": "82.28B",
    "2025-12-01": "84.17B",
    "2026-01-01": "85.76B"
  },
  "TrafficSources": {
    "Social": 0.76,
    "Paid Referrals": 0.33,
    "Mail": 0.28,
    "Referrals": 4.46,
    "Search": 8.35,
    "Direct": 85.83
  },
  "TopCountryShares": [
    { "Country": 840, "CountryCode": "US", "Value": 24.59 },
    { "Country": 356, "CountryCode": "IN", "Value": 5.55 },
    { "Country": 392, "CountryCode": "JP", "Value": 5.47 },
    { "Country": 76, "CountryCode": "BR", "Value": 4.8 },
    { "Country": 826, "CountryCode": "GB", "Value": 3.66 }
  ],
  "TopKeywords": [
    { "Name": "gemini", "EstimatedValue": 151798290, "Volume": 100378910, "Cpc": 0.42 },
    { "Name": "gmail", "EstimatedValue": 119207790, "Volume": 120497090, "Cpc": 1.74 },
    { "Name": "google maps", "EstimatedValue": 87839390, "Volume": 63966790, "Cpc": 0.41 }
  ],
  "AiTrafficDetails": {
    "Traffic": {
      "Split": [
        { "Name": "chatgpt.com", "Rank": 1 },
        { "Name": "perplexity.ai", "Rank": 2 },
        { "Name": "claude.ai", "Rank": 3 }
      ]
    },
    "TotalVisits": 317693541.48
  },
  "SimilarSites": [
    { "Site": "bing.com", "TotalVisits": 1234567890, "CategoryRank": 2, "TopCountryRank": 3, "Description": "..." },
    { "Site": "yahoo.com", "TotalVisits": 987654321, "CategoryRank": 5, "TopCountryRank": 7, "Description": "..." }
  ],
  "SnapshotDate": "2026-01-01T00:00:00+00:00"
}

Use Cases

Competitive Analysis — Track competitors' traffic trends, top keywords, and traffic sources to identify gaps and opportunities in your market.

Marketing Strategy — Understand where your competitors get their traffic from (search, social, direct, paid) and allocate your marketing budget more effectively.

SEO Research — Discover which keywords drive the most traffic to any website, along with their search volume and CPC, to inform your content and SEO strategy.

Lead Generation & Sales Intelligence — Enrich your CRM with website traffic data to qualify leads, prioritize outreach, and understand prospect digital presence.

Investment & Due Diligence — Evaluate company digital performance using traffic trends, monthly visit history, and market positioning data for informed investment decisions.

Market Research — Analyze geographic traffic distribution, category rankings, and audience engagement across any industry or vertical.

Partnership Discovery — Find similar websites and potential partners in your niche using the built-in similar sites feature, complete with traffic and ranking data.

AI Traffic Monitoring — Track which AI platforms (ChatGPT, Perplexity, Claude, Gemini) are driving referral traffic to websites — a growing and valuable signal for digital strategy.

FAQ

How does it handle failed domains? Each domain is processed independently. If one fails (e.g. domain doesn't exist or data is unavailable), the error is logged and included in the output — the remaining domains continue processing without interruption.

How fast is it? All domains (up to 50 per run) are processed in parallel. Typical response time is 1-10 seconds per domain depending on proxy speed.

Does it work with subdomains? Yes, you can pass subdomains like translate.google.com or maps.google.com and get separate analytics for each.

Can I use my own proxies? Yes. Select "Custom proxies" in the proxy configuration field in Apify Console and add your proxy URLs there.

Which Apify Proxy group should I use? RESIDENTIAL is recommended — it provides the best success rate. DATACENTER is faster but may get blocked more often.

I'm getting lots of 403/429 errors — what should I do? Switch to RESIDENTIAL proxies with per-request rotation. See the Handling Errors with Proxy Rotation section above for detailed configuration examples and troubleshooting tips.