Similarweb scraper avatar

Similarweb scraper

Try for free

3 days trial then $19.00/month - No credit card required now

Go to Store
Similarweb scraper

Similarweb scraper

curious_coder/similarweb-scraper
Try for free

3 days trial then $19.00/month - No credit card required now

Scrape similarweb to get insights for websites in bulk including website traffic, pages per visit, time spent on website, bounce rate, rank, etc

DY

Crawler keeps failing, even when re-running with different proxies

Closed

dyf opened this issue
2 months ago

I noticed it hit a point where all the requests started failing despite re-runs.

curious_coder avatar

The issue was due to using urls instead of domain names, I've sent an update to convert the urls into domains automatically

DY

dyf

2 months ago

Ahhh okay, gotcha. So you're saying the ideal format would have been "whatever.com" vs "https://whatever.com" ?

Thanks so much for the quick update!

Developer
Maintained by Community

Actor Metrics

  • 80 monthly users

  • 15 stars

  • >99% runs succeeded

  • 25 days response time

  • Created in Jul 2023

  • Modified 6 days ago

Categories