SimilarWeb Scraper — Website Traffic, Rankings, WHOIS, and Keyword Density

Extract comprehensive website analytics: traffic data, global and country rankings, engagement metrics, traffic sources, top keywords, AI referral traffic, similar sites, WHOIS domain registration info, and keyword density analysis. Analyze up to 50 domains in parallel — all data queried simultaneously for maximum speed.

Pricing

Plan	Price	Best For	Link
Pay per result	$2 / 1,000 results	One-time analysis, small batches	SimilarWeb Scraper
Monthly subscription	$20 / month	Regular monitoring, unlimited usage	SimilarWeb Website Analytics

Both plans have identical functionality — choose the one that fits your workflow.

Key Features

• Bulk domain analysis — submit up to 50 domains at once, all processed simultaneously
• Traffic estimates — monthly visits, bounce rate, pages per visit, average session duration, and 3-month visit history
• Global and country rankings — worldwide rank, country-specific rank, and industry category rank for every domain
• Traffic source breakdown — percentage split across direct, organic search, paid search, social, referral, email, and display ads
• Top organic keywords — keyword name, search volume, estimated value, and cost-per-click (CPC)
• Geographic distribution — top 5 countries driving traffic with share percentages
• AI traffic tracking — see which AI platforms (ChatGPT, Perplexity, Claude, Gemini) send referral traffic to each website
• Similar sites discovery — related websites with their own traffic data, category rank, and descriptions
• WHOIS data — domain registration date, expiration date, registrar, nameservers, registrant country, and domain status
• Keyword density analysis — 1-word through 5-word phrase frequency analysis with stop word filtering (50 languages)
• Flexible proxy support — use Apify Proxy (Residential or Datacenter) or bring your own HTTP proxies. All requests go through the configured proxy
• Error resilience — if one data type fails to load, the rest still return normally. Failed items are logged with error details

What Data You Get

SimilarWeb Traffic Data

Field	Description
SiteName	Domain name
Title	Website title
Description	Website meta description
Category	Industry category (e.g. computers_electronics_and_technology/search_engines)
GlobalRank	Worldwide traffic rank
CountryRank	Rank within the primary traffic country
CategoryRank	Rank within industry category
Engagments	Total visits, bounce rate, pages per visit, session duration
EstimatedMonthlyVisits	Last 3 months of visit data (raw numbers and formatted)
TrafficSources	Percentage split by channel: Direct, Search, Social, Referrals, Mail, Paid
TopCountryShares	Top 5 countries with traffic share percentages
TopKeywords	Top organic keywords with search volume, estimated value, and CPC
AiTrafficDetails	AI chatbot referral sources (ChatGPT, Perplexity, Claude) and total AI-driven visits
Competitors	Top similarity-based competitors
SnapshotDate	Data snapshot date

Similar Sites

Field	Description
SimilarSites	Array of related websites with Site, TotalVisits, CategoryRank, TopCountryRank, Description

WHOIS Domain Data

Field	Description
Whois.registrationDate	Domain registration date
Whois.expirationDate	Domain expiration date
Whois.lastUpdated	Last WHOIS record update
Whois.registrar	Domain registrar name
Whois.registrantCountry	Registrant country code
Whois.nameservers	List of DNS nameservers
Whois.status	Domain status flags (e.g. client transfer prohibited)

Keyword Density

Field	Description
KeywordDensity.1word	Single-word frequency (stop words filtered)
KeywordDensity.2words	2-word phrase frequency
KeywordDensity.3words	3-word phrase frequency
KeywordDensity.4words	4-word phrase frequency
KeywordDensity.5words	5-word phrase frequency

Each entry contains: keyword, count, total (total words on page), density (percentage). Sorted by density descending.

Use Cases

• Competitive analysis — track competitors' traffic trends, top keywords, and traffic sources to spot gaps and opportunities in your market
• SEO and keyword research — discover which keywords drive the most traffic to any website, along with search volume and CPC, to inform your content strategy
• On-page SEO audit — use keyword density analysis to check keyword stuffing, content relevance, and phrase optimization across 1-5 word n-grams
• Domain research — check domain age, expiration dates, registrar history, and DNS configuration via WHOIS data
• Marketing budget allocation — understand where competitors get traffic (search, social, direct, paid) and optimize your channel mix accordingly
• Lead generation and sales intelligence — enrich your CRM with website traffic data to qualify leads and prioritize outreach
• Investment due diligence — evaluate company digital performance using traffic trends, monthly visit history, domain age, and market positioning
• Partnership discovery — find similar websites and potential partners in your niche using built-in similar sites data
• AI traffic monitoring — track which AI platforms (ChatGPT, Perplexity, Claude, Gemini) drive referral traffic to websites

How It Works

1. Enter domains — provide up to 50 domains (e.g. google.com, shopify.com)
2. Configure proxy — select Residential proxy for best success rate, or use your own
3. Actor fetches data — all domains processed in parallel, all data types queried concurrently
4. Download results — structured JSON output in the Apify dataset, ready to export as JSON, CSV, or Excel

Input Parameters

Parameter	Type	Required	Description
domains	String[]	Yes	List of domains to analyze, max 50 per run (e.g. google.com, facebook.com)
proxyConfiguration	Object	No	Proxy settings — Apify Proxy groups (RESIDENTIAL, DATACENTER) or custom proxy URLs

Proxy configuration examples

Apify Residential Proxy (recommended):

{
    "domains": ["google.com", "facebook.com", "shopify.com"],
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": ["RESIDENTIAL"]
    }
}

Custom rotating proxies:

{
    "domains": ["google.com"],
    "proxyConfiguration": {
        "useApifyProxy": false,
        "proxyUrls": [
            "http://user:pass@proxy-provider.com:port"
        ]
    }
}

Output Example

{
    "SiteName": "google.com",
    "Title": "Google",
    "Description": "Search the world's information...",
    "Category": "computers_electronics_and_technology/search_engines",
    "GlobalRank": { "Rank": 1 },
    "CountryRank": { "Country": 840, "CountryCode": "US", "Rank": 1 },
    "CategoryRank": { "Rank": "1", "Category": "Computers_Electronics_and_Technology/Search_Engines" },
    "Engagments": {
        "Visits": 85756574615,
        "VisitsFormatted": "85.76B",
        "BounceRate": 28.36,
        "PagePerVisit": 8.52,
        "TimeOnSite": 610
    },
    "EstimatedMonthlyVisits": {
        "2025-11-01": 82284033110,
        "2025-12-01": 84172772881,
        "2026-01-01": 85756574615
    },
    "TrafficSources": {
        "Direct": 85.83,
        "Search": 8.35,
        "Referrals": 4.46,
        "Social": 0.76,
        "Paid Referrals": 0.33,
        "Mail": 0.28
    },
    "TopCountryShares": [
        { "CountryCode": "US", "Value": 24.59 },
        { "CountryCode": "IN", "Value": 5.55 },
        { "CountryCode": "JP", "Value": 5.47 }
    ],
    "TopKeywords": [
        { "Name": "gemini", "Volume": 100378910, "Cpc": 0.42 },
        { "Name": "gmail", "Volume": 120497090, "Cpc": 1.74 },
        { "Name": "google maps", "Volume": 63966790, "Cpc": 0.41 }
    ],
    "AiTrafficDetails": {
        "Traffic": {
            "Split": [
                { "Name": "chatgpt.com", "Rank": 1 },
                { "Name": "perplexity.ai", "Rank": 2 },
                { "Name": "claude.ai", "Rank": 3 }
            ]
        },
        "TotalVisits": 317693541
    },
    "SimilarSites": [
        { "Site": "bing.com", "TotalVisits": 1234567890, "CategoryRank": 2 },
        { "Site": "yahoo.com", "TotalVisits": 987654321, "CategoryRank": 5 }
    ],
    "Whois": {
        "registrationDate": "1997-09-15T04:00:00Z",
        "expirationDate": "2028-09-14T04:00:00Z",
        "lastUpdated": "2019-09-09T15:39:04Z",
        "registrar": "MarkMonitor Inc.",
        "registrantCountry": "US",
        "nameservers": ["NS1.GOOGLE.COM", "NS2.GOOGLE.COM", "NS3.GOOGLE.COM", "NS4.GOOGLE.COM"],
        "status": ["client delete prohibited", "client transfer prohibited", "client update prohibited"]
    },
    "KeywordDensity": {
        "1word": [
            { "keyword": "google", "count": 5, "total": 366, "density": 1.37 },
            { "keyword": "english", "count": 5, "total": 366, "density": 1.37 },
            { "keyword": "personalisierte", "count": 5, "total": 366, "density": 1.37 }
        ],
        "2words": [
            { "keyword": "cookies und", "count": 3, "total": 366, "density": 0.82 },
            { "keyword": "verwenden wir", "count": 3, "total": 366, "density": 0.82 }
        ],
        "3words": [
            { "keyword": "cookies und daten", "count": 3, "total": 366, "density": 0.82 }
        ],
        "4words": [
            { "keyword": "verwenden wir cookies und", "count": 3, "total": 366, "density": 0.82 }
        ],
        "5words": [
            { "keyword": "verwenden wir cookies und daten", "count": 2, "total": 366, "density": 0.55 }
        ]
    },
    "SnapshotDate": "2026-01-01T00:00:00+00:00"
}

Proxy and Troubleshooting

Residential proxies are recommended for the best success rate. SimilarWeb actively blocks datacenter IPs.

Symptom	Likely Cause	Fix
Most domains return 403/429	IP is blocked	Switch to RESIDENTIAL proxies
Single-domain failure	Domain has no SimilarWeb data	Check domain name; expected behavior
Timeouts on all domains	Proxy is down	Verify proxy URLs or use Apify Proxy
Works for ~10 domains, then fails	IP not rotating	Use per-request rotation, avoid sticky sessions
WHOIS data missing	Rate limit on WHOIS lookup	Normal for high-volume runs, other data still present

FAQ

How many domains can I analyze per run? Up to 50 domains in a single run, all processed in parallel. Typical response time is 1–10 seconds per domain.

Does the SimilarWeb scraper work with subdomains? Yes. Pass subdomains like translate.google.com or maps.google.com and get separate analytics for each.

What proxy should I use for SimilarWeb scraping? Residential proxies give the highest success rate. Datacenter proxies are faster and cheaper but get blocked more often. Start with RESIDENTIAL if you need reliability.

Can I use my own proxies instead of Apify Proxy? Yes. Select "Custom proxies" in the proxy configuration and add your proxy URLs. Make sure they support per-request rotation.

What is AI traffic data? SimilarWeb now tracks referral traffic from AI platforms like ChatGPT, Perplexity, Claude, and Gemini. This Actor extracts that data, showing which AI sources send traffic and their relative distribution.

What happens if a domain fails? The Actor continues processing all remaining domains. If some data types fail to load for a domain, the rest still return normally.

How does keyword density work? The Actor fetches the page HTML, strips non-visible elements (scripts, styles, iframes, etc.), tokenizes the visible text, and calculates n-gram frequencies for 1-5 word phrases. Stop words from 50 languages are filtered from single-word results.

What's the difference between the monthly and pay-per-use plans? Identical functionality. Monthly ($20/mo) is better for ongoing monitoring; pay-per-use ($2/1k results) is better for occasional analysis.

Having issues? Help me fix them faster

If you experience any problems, please share your run data with me so I can debug and improve the Actor:

1. Go to Apify Security Settings
2. Find "Share run data with developers"
3. In the "Manage list of Actors" section, check this Actor (or All Actors)
4. Save

This data is used only for debugging and helps me resolve issues much faster. Thank you!

Support

Questions, bug reports, or feature requests: afrcanec@gmail.com