Similarweb Scraper
No credit card required
Similarweb Scraper
No credit card required
A simple but powerful scraper for similarweb.com. Retrieve website popularity information and get it in a JSON/XML/CSV/Excel/HTML table format. Get data such as total visits, traffic sources, competitors, top countries, company info, etc..
Crawling failing
Hi there, seems the crawler is failing for this run
Hi there, I checked the run and it seems ok, because there are missing details about the company on the website too, if you check https://similarweb.com/website/www.usepalm.com Please let me know if you would encounter any other issues. Have a nice day!
gainful_governor-owner
It eventually got there but look how long it took? It failed many times before it succeeded
Yes, it can happen, but it isn't a bug. There are 10 retries set, it is needed on this website.
gainful_governor-owner
Okay understood, thank you
gainful_governor
Can you help me understand what is going on here?
2024-09-20T14:58:48.205Z DEBUG PlaywrightCrawler:SessionPool:Session: Could not set cookies. {"errorMessages":["Cookie not in this host's domain. Cookie:doubleclick.net Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:bat.bing.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:bing.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:www.linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:www.clarity.ms Request:www.similarweb.com"]}
Why does this happen sometimes?
gainful_governor
why do some runs take 40 seconds and others 6 minutes, why are the retries necessary?
gainful_governor
Tri⟁angle
