Similarweb Scraper avatar
Similarweb Scraper

Pricing

Pay per event

Go to Store
Similarweb Scraper

Similarweb Scraper

Developed by

Tri⟁angle

Tri⟁angle

Maintained by Apify

A simple but powerful scraper for similarweb.com. Retrieve website popularity information and get it in a JSON/XML/CSV/Excel/HTML table format. Get data such as total visits, traffic sources, competitors, top countries, company info, etc..

4.6 (7)

Pricing

Pay per event

79

Total users

3.1K

Monthly users

162

Runs succeeded

>99%

Issues response

1.3 days

Last modified

11 days ago

GG

Crawling failing

Closed

gainful_governor opened this issue
10 months ago

Hi there, seems the crawler is failing for this run

tri_angle avatar

Hi there, I checked the run and it seems ok, because there are missing details about the company on the website too, if you check https://similarweb.com/website/www.usepalm.com Please let me know if you would encounter any other issues. Have a nice day!

GO

gainful_governor-owner

10 months ago

It eventually got there but look how long it took? It failed many times before it succeeded

tri_angle avatar

Yes, it can happen, but it isn't a bug. There are 10 retries set, it is needed on this website.

GO

gainful_governor-owner

10 months ago

Okay understood, thank you

GG

gainful_governor

10 months ago

Can you help me understand what is going on here?

2024-09-20T14:58:48.205Z DEBUG PlaywrightCrawler:SessionPool:Session: Could not set cookies. {"errorMessages":["Cookie not in this host's domain. Cookie:doubleclick.net Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:bat.bing.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:bing.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:www.linkedin.com Request:www.similarweb.com","Cookie not in this host's domain. Cookie:www.clarity.ms Request:www.similarweb.com"]}

Why does this happen sometimes?

GG

gainful_governor

10 months ago

why do some runs take 40 seconds and others 6 minutes, why are the retries necessary?