Similarweb Quick Scraper
Try for free
Pay $10.00 for 1,000 results
View all Actors
Similarweb Quick Scraper
mscraper/similarweb-quick-scraper
Try for free
Pay $10.00 for 1,000 results
A quick scraper for Similarweb. Get needed data instantly for domains of your choice. Export accumulated data into formats such as HTML, JSON, or Excel.
Failed urls
Closed
world33 opened this issue
a year ago
Hello,
- Are the failed URLs inevitably occuring in each actor run due to bad proxies and the strong bot mitigation protections similarweb is adopting?
- Is there any way to rerun the actor only with the failed URLs once a a run is concluded? Or is there a way to download also the failed urls to easily and quickly identify them and start a new run only with those? It is very time consuming to identify the failed urls otherwise. Maybe when downloading the results it would be possible to add a field called Status where it says Succeded or Failed for each URLs initially added to the input field. At the moment only the succeded URLs are exported if I am not wrong. Thanks
a year ago
- Now, they only block IPs. I recommend to try Oxylabs or Brightdata, and pass their proxy string as a custom proxy.
- I will save all failed keywords to the key-value store to simplify the process. I can't add it to the dataset because it will cost the same as a successful result.
I will let you know here when I add it.
a year ago
Ok, now an actor saves the failed-websites.json file to the key-value store if any websites fail. I added it as the last step of the actor before the end.
world33
a year ago
Thank you very much!
mscraper
