📊 SimilarWeb Scraper

🎯 Get instant SimilarWeb traffic intelligence for any website — without paying for an enterprise account.

Pull rankings, monthly visits, traffic sources, top keywords, top countries, engagement stats and AI traffic referrals straight from SimilarWeb. Drop in a list of domains and get clean, structured JSON ready for your CRM, BI dashboard, or research stack.

---

🚀 What Does It Do?

This scraper automatically pulls SimilarWeb's domain analytics for any website you give it and returns structured, ready-to-use data. No manual browsing through dashboards — just paste your list of domains and hit Start.

You get global, country, and category rankings, three months of monthly visit history, the full traffic source breakdown (direct / search / social / referrals / mail / paid), top countries, top SEO keywords, competitors, and the new AI Traffic split (which chatbots are sending people to the site).

---

👥 Who Is This For?

🏢 Use Case	💬 How It Helps
📈 SEO Agencies	Audit thousands of client and competitor domains in minutes — rank, keywords, traffic sources
💼 Sales & Lead Gen Teams	Score and prioritize prospects by real traffic and rank instead of guessing
🔎 Market Researchers	Track market share across an industry with bulk traffic and engagement data
🛒 E-commerce Owners	See exactly how much traffic competitors get and where it comes from
🚀 Investors & VCs	Validate startup growth claims with independent traffic and engagement metrics
🤝 Partnership & BD Teams	Find high-traffic partners and qualify opportunities with hard numbers

---

✨ Features

• 🌐 Bulk Domain Lookups — Process hundreds or thousands of domains in a single run
• 📈 Traffic & Rankings — Global rank, country rank, category rank, and 3 months of monthly visits
• 🎯 Traffic Source Breakdown — Direct, search, social, mail, referrals, and paid referrals as percentages
• 🌍 Top Countries — Visit share by country with ISO codes for the top traffic origins
• 🔑 Top Keywords — Top organic keywords with estimated traffic value, search volume, and CPC
• 🤖 AI Traffic Split — See which AI chatbots (ChatGPT, Perplexity, Gemini, Copilot, Claude) are sending visitors
• 🏁 Competitor Discovery — Top similarity competitors per domain
• 📷 Site Screenshots — Direct CDN URL to a fresh screenshot of every site
• 🛡️ Smart Retries — Automatic proxy rotation when SimilarWeb rate-limits or stalls a request
• ⏱️ Configurable Timeout — Hung requests are aborted quickly so a single slow domain never blocks the run
• 🚀 Parallel Processing — Configurable concurrency processes multiple domains at the same time for fast bulk runs
• ⚡ Fast & Scalable — Lightweight HTTP fetches mean low compute cost even for thousands of domains
• 📤 Export Anywhere — Download results as JSON, CSV, Excel, or push to Google Sheets, Zapier, Make, or your CRM

---

🎛️ Filters & Options

Option	What It Does
🌐 Domains	Paste any list of domains (apify.com, github.com, …) — full URLs are accepted and normalized
🔗 Start URLs	Alternative input — paste full SimilarWeb or website URLs and the domain is extracted automatically
⚡ Max Concurrency	How many domains to fetch in parallel (default 2, up to 50). Higher is faster but may trigger more rate limits
🔁 Max Retries	How many times to retry per domain when SimilarWeb blocks or rate-limits a request
⏱️ Request Timeout	Seconds before a hung request is aborted and retried with a fresh proxy (default 5s)
🌐 Proxy Configuration	Apify Proxy settings — residential is required and used by default

---

📦 What You Get (Output Fields)

Every domain you scrape includes:

Site Info

Field	Example
domain	stackoverflow.com
siteName	stackoverflow.com
title	Newest 'postgresql' Questions
description	Stack Overflow | The World's Largest Online Community for Developers
category	computers_electronics_and_technology/programming_and_developer_software
largeScreenshot	https://site-images.similarcdn.com/image?url=stackoverflow.com&t=1&s=1&h=...
snapshotDate	2026-04-01T00:00:00+00:00
url	https://www.similarweb.com/website/stackoverflow.com/

Rankings

Field	Example
globalRank	2240
countryRank	2170
countryRankCountryCode	US
categoryRank	63
categoryRankCategory	Computers_Electronics_and_Technology/Programming_and_Developer_Software
globalCategoryRank	null

Engagement & Visits

Field	Example
monthlyVisits	31837008
engagementMonth	2026-04
bounceRate	70.26
pagesPerVisit	1.69
avgVisitDurationSeconds	141.72
monthlyVisitsHistory	[{"month":"2026-02","visits":42405407},{"month":"2026-03","visits":38044480},{"month":"2026-04","visits":31837008}]

Traffic Sources (% of total)

Field	Example
trafficSourceDirect	7.01
trafficSourceSearch	85.6
trafficSourceSocial	1.52
trafficSourceMail	0.24
trafficSourceReferrals	5.21
trafficSourcePaidReferrals	0.03

Audience & Keywords

Field	Example
topCountries	[{"countryCode":"US","countryNumeric":840,"share":22.21},{"countryCode":"IN","countryNumeric":356,"share":8.07}]
topKeywords	[{"keyword":"stack overflow","estimatedValue":149400,"volume":149150,"cpc":1.42}]
competitors	[{"domain":"github.com","affinity":0.87,"monthlyVisits":420000000}]

AI Traffic

Field	Example
aiTotalVisits	184250.5
aiReferralTraffic	0.0058
aiTrafficSplit	[{"chatbot":"chatgpt.com","rank":1},{"chatbot":"perplexity.ai","rank":2},{"chatbot":"gemini.google.com","rank":3}]

Meta

Field	Example
isSmall	false
isDataFromGa	false
scrapedAt	2026-05-06T20:35:00.000Z

---

📊 Example Output

{
    "domain": "stackoverflow.com",
    "siteName": "stackoverflow.com",
    "title": "Newest 'postgresql' Questions",
    "description": "Stack Overflow | The World's Largest Online Community for Developers",
    "category": "computers_electronics_and_technology/programming_and_developer_software",
    "largeScreenshot": "https://site-images.similarcdn.com/image?url=stackoverflow.com&t=1&s=1&h=04179ebf",
    "snapshotDate": "2026-04-01T00:00:00+00:00",
    "globalRank": 2240,
    "countryRank": 2170,
    "countryRankCountryCode": "US",
    "categoryRank": 63,
    "categoryRankCategory": "Computers_Electronics_and_Technology/Programming_and_Developer_Software",
    "globalCategoryRank": null,
    "monthlyVisits": 31837008,
    "engagementMonth": "2026-04",
    "bounceRate": 70.26,
    "pagesPerVisit": 1.69,
    "avgVisitDurationSeconds": 141.72,
    "monthlyVisitsHistory": [
        { "month": "2026-02", "visits": 42405407 },
        { "month": "2026-03", "visits": 38044480 },
        { "month": "2026-04", "visits": 31837008 }
    ],
    "trafficSourceDirect": 7.01,
    "trafficSourceSearch": 85.6,
    "trafficSourceSocial": 1.52,
    "trafficSourceMail": 0.24,
    "trafficSourceReferrals": 5.21,
    "trafficSourcePaidReferrals": 0.03,
    "topCountries": [
        { "countryCode": "US", "countryNumeric": 840, "share": 22.21 },
        { "countryCode": "IN", "countryNumeric": 356, "share": 8.07 },
        { "countryCode": "DE", "countryNumeric": 276, "share": 7.29 }
    ],
    "topKeywords": [
        { "keyword": "stack overflow", "estimatedValue": 149400, "volume": 149150, "cpc": 1.42 },
        { "keyword": "stackoverflow", "estimatedValue": 43680, "volume": 48040, "cpc": 1.9 }
    ],
    "competitors": [],
    "aiTotalVisits": 184250.5,
    "aiReferralTraffic": 0.0058,
    "aiTrafficSplit": [
        { "chatbot": "chatgpt.com", "rank": 1 },
        { "chatbot": "perplexity.ai", "rank": 2 },
        { "chatbot": "gemini.google.com", "rank": 3 }
    ],
    "isSmall": false,
    "isDataFromGa": false,
    "url": "https://www.similarweb.com/website/stackoverflow.com/",
    "scrapedAt": "2026-05-06T20:35:00.000Z"
}

---

📋 Dataset Views

The Apify Console gives you 4 ready-made table views to quickly browse your results:

View	What It Shows
📊 Overview	Domain, category, ranks, monthly visits and engagement stats at a glance
🚦 Traffic Sources	Side-by-side comparison of direct / search / social / mail / referrals / paid traffic
🏆 Rankings	Global, country, and category rankings — perfect for competitive benchmarking
📋 Full Details	Every single field — the complete dataset

---

❓ FAQ

🤔 Do I need a SimilarWeb account? No. You don't need any login, paid plan, or API key — just paste your list of domains and run.

🤔 What domains work? Anything SimilarWeb has data for. Popular sites return rich data; very small or new domains may have partial data or none at all.

🤔 How fresh is the data? SimilarWeb refreshes its dataset monthly — every record includes a snapshotDate and engagementMonth so you always know exactly which month the numbers represent.

🤔 What about subdomains and paths? SimilarWeb works at the registered domain level. Subdomains and paths in your input are normalized to the root domain automatically.

🤔 Can I scrape thousands of domains? Yes. The scraper processes domains in parallel (2 at a time by default, up to 50) with smart retries, so even very large lists complete in minutes — not hours.

🤔 Can I export the data? Yes — JSON, CSV, Excel, XML, HTML, RSS. You can also push data directly to Google Sheets, Zapier, Make, or any webhook/API endpoint.

🤔 How often should I run this? For fresh data, run monthly (right after SimilarWeb's data refresh). You can schedule automatic runs on Apify with just a few clicks.

🤔 Does it work with proxies? Yes — the scraper uses Apify residential proxies out of the box. SimilarWeb rate-limits datacenter IPs aggressively, so residential is required for reliable results.

---

🛠️ Need Custom Filters or Features?

I'm happy to customize this scraper for your specific needs! 🤝

Whether you need:

• 🎯 Additional filters (specific countries, categories, minimum traffic thresholds, growth filters)
• 📊 Extra data fields or custom output formats (e.g. growth-rate calculations, multi-month comparisons)
• 🔄 Integration with your CRM, Google Sheets, or database
• ⏰ Scheduled scraping with automatic deduplication and change tracking
• 🌐 Combined runs alongside other web-analytics platforms (Semrush, Ahrefs-style data) or competitor scrapers

👉 Don't hesitate to reach out via private message — I respond quickly and I'm always open to building exactly what you need. No request is too small or too specific!

---

⚖️ Legal & Ethical Use

This scraper collects only publicly available information from SimilarWeb. It does not access private dashboards, bypass authentication, or download paid datasets. Please use the data responsibly and in compliance with applicable laws and platform terms of service.