SimilarWeb Scraper

Traffic estimates for any domain: monthly visits, global, country and category rank, bounce rate, pages per visit, traffic sources, the keywords a site ranks for with CPC, its detected technology stack and its closest competitors.

One domain in, one row out. Pass a list and get a comparison table.

What a run returns

Real output, taken while writing this:

	github.com	ebay.com
Global rank	49	44
Monthly visits	637,885,711	669,411,467
Bounce rate	36.27%	38.55%
Pages per visit	5.87	6.75
Avg. visit	06:26	06:07
Organic / paid search	99.97% / 0.03%	83.27% / 16.73%
Keywords ranked for	3,756,914	5,847,215
Similar sites	stackoverflow, npmjs, pypi, gitlab	amazon, walmart, aliexpress, etsy

That organic-versus-paid split is the line worth staring at: eBay buys 16.73% of its search traffic, GitHub buys effectively none. Two sites of almost identical size with completely different economics.

Read traffic_bucket as a size, not a score

traffic_bucket is SimilarWeb's own magnitude band. It looks like a warning and is not: github.com is graded <1B on 638 million visits, and a small site is graded <20K. It tells you which bracket a site falls in, nothing about how much to trust the number.

Confidence follows the visit count instead. Below 50,000 monthly visits the estimate rests on very little panel data, and those rows carry low_confidence: true, with the affected domains listed in the run summary. A site with a few thousand visits is flagged; a site with hundreds of millions is not, whatever band it sits in.

Every figure here is an estimate, not measured analytics. They are most reliable on large sites and least on small ones, and the flag exists so that distinction survives into your spreadsheet.

Fields

• Ranking: global_rank, country_rank, category_rank, each with its change, plus rank_neighbours — the domains immediately above and below
• Traffic: monthly_visits, visits_change_percent, traffic_history, traffic_bucket, low_confidence
• Engagement: bounce_rate_percent, pages_per_visit, avg_visit_duration
• Channels: traffic_sources_percent across direct, organic, paid, referrals, social, mail and ads
• Search: organic_search_percent, paid_search_percent, ranking_keywords_count, top_keywords with volume and CPC where published
• Stack: technologies by category, e.g. analytics, advertising, ecommerce
• Competitive: similar_sites
• Company: company_name, company_founded, employee band, HQ country

top_keywords is worth calling out. It arrives with CPC attached, which makes it a keyword-research source as well as a traffic one, and it needs no key.

Reliability

The figures come from the same profile pages SimilarWeb publishes, read directly rather than through a third-party reseller, so there is no extra layer between you and the source.

Runs are consistent: repeated checks of the same domain returned the full payload every time, from more than one network path. That matters because traffic data tends to get pasted into a deck and quoted for a quarter, so a result that only works some of the time is worse than no result.

If a profile ever comes back without its data, the run says no_data and explains that the page shape may have changed, instead of handing you a row of zeroes. A run of zeroes that looks successful is the failure mode worth being loud about.

Input reference

Field	Type	Default
domains	list of domains or URLs	["github.com","ebay.com"]
limit	1-500	50
retries	1-8	4

Full URLs are reduced to the domain, so https://www.github.com/foo and github.com behave the same. A value that is not a domain is rejected with a message rather than fetched.

Typical uses

• Competitor benchmarking. Run your domain alongside three rivals and compare visits, engagement and the paid-versus-organic split in one table.
• Market sizing. similar_sites plus category_rank maps who else occupies a niche and how large it is.
• Lead qualification. Traffic, technology stack and company size in one row tells a sales team whether a prospect is worth the call.
• Keyword research without a key. top_keywords with CPC, per competitor.
• Technology prospecting. Filter on technologies to find every site in a list running a particular platform.
• Trend monitoring. Schedule it and join on domain; traffic_history and the rank-change fields show direction.

Notes on behaviour

A page that loads but carries no application state is reported as no_data with a note that the crawler-agent route may have closed, rather than returning a row of zeroes. A run of zeroes that looks successful is the failure worth being loud about, because traffic data is exactly the kind people paste into a deck without re-checking.

Domains are paced with a short randomised gap. Exits are pinned to one country.

Some fields are simply absent for smaller sites: similar_sites and the company block come back null rather than empty, because SimilarWeb publishes them only where it has the data.