🔍 Similarweb Scraper — Website Traffic & Competitor Intelligence

Know how much traffic any website gets — and where it comes from. 📊 Paste a list of domains, press Start, and get clean, structured data: global rankings, monthly visits, traffic sources, top keywords, top countries, the AI chatbots sending visitors, similar/competitor sites, and domain WHOIS. No Similarweb account, no login, no API key needed. ✅

Feed it one domain to try it, or thousands to run a whole market analysis. One website in → one clean row out. Export to JSON, CSV, Excel, or Google Sheets. 📤

---

⚡ Quick start (30 seconds)

1. Paste your domains into Domains (e.g. github.com, nytimes.com).
2. Pick a Mode — start with All-in-one to get everything.
3. Press Start. 🚀

{
  "domains": ["github.com", "nytimes.com"],
  "mode": "all",
  "maxConcurrency": 10
}

That's it. Each domain becomes one row in the Output tab. Download it in any format you like.

---

📦 What you can collect — pick a Mode

Each run uses one mode. Here's what each returns:

Mode	🏷️ Value	What you get
⭐ All-in-one	Everything in one row	Base data + similar sites + WHOIS + keywords, merged per domain
📈 Base data	Traffic & rankings	Ranks, monthly visits, engagement, month-over-month growth, traffic sources, top keywords, top countries, AI traffic, screenshot
🔗 Similar sites	Competitors	Related/competitor sites with rank, category, visits & similarity grade, topic tags, related mobile apps
🔎 WHOIS + keywords	Domain & on-page SEO	Registrar, registration/expiry dates, nameservers, DNSSEC + on-page keyword density

💡 Not sure which to pick? Use All-in-one — it returns all of the above in a single row per domain.

---

📈 Mode 1 — Base data

The core traffic report. Example output for github.com:

{
  "domain": "github.com",
  "siteName": "github.com",
  "category": "computers_electronics_and_technology/programming_and_developer_software",
  "globalRank": 47,
  "countryRank": { "countryCode": "US", "rank": 76 },
  "engagement": {
    "visits": 615239605,
    "bounceRate": 0.363,
    "pagesPerVisit": 5.87,
    "timeOnSite": 383.3
  },
  "estimatedMonthlyVisits": {
    "2026-04-01": 631004092,
    "2026-05-01": 636075828,
    "2026-06-01": 615239605
  },
  "trafficChangeMoM": -0.0328,
  "screenshot": "https://site-images.similarcdn.com/image?url=github.com&t=1&h=...",
  "trafficSources": {
    "direct": 0.521, "searchOrganic": 0.253, "referrals": 0.099,
    "socialOrganic": 0.091, "genAi": 0.020
  },
  "topCountryShares": [
    { "countryCode": "US", "share": 0.193 },
    { "countryCode": "CN", "share": 0.113 }
  ],
  "topKeywords": [
    { "name": "github", "volume": 9009470, "cpc": 1.54 }
  ],
  "aiTraffic": [
    { "name": "chatgpt.com", "rank": 1 },
    { "name": "gemini.google.com", "rank": 2 },
    { "name": "claude.ai", "rank": 3 }
  ],
  "status": "ok"
}

📖 What each field means

Field	Meaning
globalRank	The site's rank worldwide (1 = most visited on the internet) 🌍
countryRank	Rank inside its top country
engagement.visits	Total visits in the latest month 👥
engagement.bounceRate	Share of visitors who leave after one page (0.36 = 36%)
engagement.pagesPerVisit	Average pages viewed per visit
engagement.timeOnSite	Average time on site, in seconds ⏱️
estimatedMonthlyVisits	Visit totals for the last 3 months (trend) 📅
trafficChangeMoM	Month-over-month change (−0.0328 = −3.3%) 📉📈
screenshot	🖼️ URL of a homepage screenshot (opens in browser; not a saved file)
trafficSources	Where visits come from — direct, search, social, referrals, paid, mail, and genAi (AI chatbots)
topCountryShares	Top countries by visit share 🗺️
topKeywords	Top search keywords with monthly volume and cpc (cost-per-click, $) 🔑
aiTraffic	🤖 Which AI platforms (ChatGPT, Gemini, Claude, Perplexity…) send visitors, ranked

---

🔗 Mode 2 — Similar sites

Find competitors and alternatives. Example for github.com:

{
  "domain": "github.com",
  "category": "Computers_Electronics_and_Technology/Programming_and_Developer_Software",
  "totalVisits": 513197610,
  "tags": [{ "Tag": "git" }, { "Tag": "open source" }, { "Tag": "programming" }],
  "similarSites": [
    {
      "site": "stackoverflow.com",
      "category": "Computers_Electronics_and_Technology/Programming_and_Developer_Software",
      "similarityRank": 1,
      "totalVisits": 69124392,
      "grade": 0.94
    },
    { "site": "sourceforge.net", "similarityRank": 2, "grade": 0.74 }
  ],
  "relatedApps": [
    { "Title": "GitHub", "Type": "ANDROID", "Ranking": 2,
      "Url": "https://play.google.com/store/apps/details?id=com.github.android" }
  ],
  "status": "ok"
}

📖 What each field means

Field	Meaning
similarSites[].site	A competitor/alternative domain
similarSites[].similarityRank	How similar it is (1 = most similar)
similarSites[].grade	Similarity score, 0–1 (higher = closer match) ⭐
similarSites[].totalVisits	That competitor's monthly visits
tags	Topics/interests associated with the site 🏷️
relatedApps	Related mobile apps (Android & iOS) with store links 📱

---

🔎 Mode 3 — WHOIS + keywords

Domain registration facts + what the homepage is "about". Example for github.com:

{
  "domain": "github.com",
  "whois": {
    "registrar": "MarkMonitor Inc.",
    "registrationDate": "2007-10-09T18:20:50Z",
    "expirationDate": "2026-10-09T18:20:50Z",
    "nameServers": ["DNS1.P08.NSONE.NET", "NS-421.AWSDNS-52.COM"],
    "dnssec": false
  },
  "keywordDensity": {
    "pageTitle": "GitHub · Build and ship software",
    "totalTokens": 855,
    "keywords": [
      { "token": "github", "count": 53, "frequency": 0.062 },
      { "token": "copilot", "count": 20, "frequency": 0.023 }
    ]
  },
  "status": "ok"
}

📖 What each field means

Field	Meaning
whois.registrar	Company the domain is registered with 🏢
whois.registrationDate / expirationDate	When the domain was registered / expires 📆
whois.nameServers	The domain's DNS servers
whois.dnssec	Whether DNSSEC security is enabled 🔒
keywordDensity.keywords	Most frequent words on the homepage, with count and frequency (share of all words) 🔑

---

⭐ Mode 4 — All-in-one

Everything above, merged into one row per domain — globalRank, engagement, trafficSources, topKeywords, aiTraffic, screenshot, similarSites, relatedApps, whois, keywordDensity. Run this when you want the full picture in a single export. 🧩

---

⌨️ Input options

Field	Description
Domains 🌐	Websites to analyse. Accepts example.com, www.example.com, or https://example.com/path — everything is normalised to a bare host automatically.
Dataset mode 🗂️	all, base_data, similar_sites, or aitdk. One mode per run.
Max concurrency ⚡	How many domains to process at the same time (higher = faster).

---

⚙️ Speed & memory per mode

Different modes do different amounts of work, so they have different memory recommendations. Pick the memory when you start a run (default 512 MB):

Mode	🧠 Recommended memory	⚡ Speed (50 domains)	Why
📈 Base data	256 MB	~28 sec	One fast source
🔗 Similar sites	256 MB	~32 sec	One fast source
🔎 WHOIS + keywords	512 MB	~60 sec	WHOIS/RDAP is slow + parses homepage HTML
⭐ All-in-one	512 MB	~40 sec (high concurrency)	Fetches all sources at once

💡 Faster runs: speed scales with Max concurrency — raising it processes more domains at the same time. For big jobs, bump concurrency up (to 50) and, for the heavier modes, keep memory at 512 MB.

💰 Cheaper runs: for Base data and Similar sites you can drop memory to 256 MB to roughly halve the compute cost with no downside.

---

🧪 Recipes — what people use it for

• 📊 Benchmark competitors → run Base data on your domain + rivals, compare globalRank, engagement, and trafficSources side by side.
• 🕵️ Discover new competitors → run Similar sites and sort similarSites by grade.
• 🤖 Track AI-driven traffic → the aiTraffic list shows which chatbots (ChatGPT / Gemini / Claude / Perplexity) send visitors — a fast-growing channel most tools ignore.
• 🔑 Keyword & SEO research → pull topKeywords (volume + CPC) from Base data, or homepage keywordDensity from WHOIS + keywords.
• 💼 Score sales leads → rank prospects by real traffic and rank instead of guessing.
• 📈 Validate growth claims → trafficChangeMoM and the 3-month estimatedMonthlyVisits trend show if a site is really growing.

---

📤 Output & export

Every domain becomes one row in the dataset. From the run's Output tab you can:

• 👀 Preview the data in a table
• 💾 Download as JSON, CSV, Excel, or HTML
• 🔗 Pull it via the Apify API, or push to Google Sheets / Zapier / Make and your CRM

Every row carries a status field so you always know what happened:

status	Meaning
✅ ok	Data collected successfully
⚠️ partial	Some of the data was available, some wasn't (WHOIS/all modes)
❌ failed	No data for this domain — check the error field (e.g. no_data, not_found)

🛡️ We never invent numbers. If a source has no data for a domain, the row is marked failed — you always get honest results.

---

💰 Pricing & FAQ

How much does it cost? The run cost is all-in — no hidden add-ons. Cost scales with the number of domains you analyse, so a quick 10-domain check is cheap and a 5,000-domain sweep is predictable.

Which sites work? Any public website. Very new or tiny sites may return failed / partial — Similarweb itself has limited data for low-traffic domains.

How fresh is the data? Traffic figures reflect Similarweb's latest monthly snapshot; each Base data row includes a snapshotDate.

Are competitors always included? The similarSites list (Similar sites / All modes) is your competitor list. The Base data competitors field is often empty — use Similar sites mode for reliable competitor discovery.

Can I get all four datasets at once? Yes — use All-in-one mode, or run the other modes separately and join on domain.

Are the screenshots downloaded? No — screenshot is a URL to a hosted image. Open it in a browser or download it yourself; nothing is stored on your side.