Similarweb Scraper
Pay $8.00 for 1,000 Results
Similarweb Scraper
Pay $8.00 for 1,000 Results
A simple but powerful scraper for similarweb.com. Retrieve website popularity information and get it in a JSON/XML/CSV/Excel/HTML table format. Get data such as total visits, traffic sources, competitors, top countries, company info, etc..
Tri⟁angleCrawling failing
Hi there, seems the crawler is failing for this run
Hi there, I checked the run and it seems ok, because there are missing details about the company on the website too, if you check https://similarweb.com/website/www.usepalm.com Please let me know if you would encounter any other issues. Have a nice day!
gainful_governor-owner
It eventually got there but look how long it took? It failed many times before it succeeded
Yes, it can happen, but it isn't a bug. There are 10 retries set, it is needed on this website.
gainful_governor-owner
Okay understood, thank you
gainful_governor
Can you help me understand what is going on here?
2024-09-20T14:58:48.205Z DEBUG PlaywrightCrawler:SessionPool:Session: Could not set cookies. {"errorMessages":["Cookie not in this host's domain. Cookie:doubleclick.net Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:bat.bing.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:bing.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:www.linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:www.clarity.ms Request:www.similarweb.com"]}
Why does this happen sometimes?
gainful_governor
why do some runs take 40 seconds and others 6 minutes, why are the retries necessary?
gainful_governor
