Web Scraper
Pricing
Pay per usage
Web Scraper
Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
4.5 (22)
Pricing
Pay per usage
737
Total users
83.6k
Monthly users
4.3k
Runs succeeded
>99%
Issues response
32 days
Last modified
a day ago
Shopee scraper problem
Closed
I'm trying to use the scraper to extract information about searching for items in the store, but without success. After some research, I noticed that the information on the page only loads after a few seconds. Any tips on how to resolve this?
URL example: https://shopee.com.br/search?keyword=chinelo
Hello and thank you for your interest in this Actor!
Usually with dynamic-loaded websites, it should be enough to set the Performace and Limits > Navigations waits until to networkidle0 - this way, the scraper should wait until there are no active (asynchronous) requests between the client and the server.
Unfortunately, there seems to be some issue with the Actor itself - even locally, the website just doesn't load properly. We already have an idea about what might be the culprit, but we'll need to run some tests first to be absolutely sure.
I'll keep you posted about any updates regarding this issue. Thank you! (and sorry for the inconvenience).
lojahub
Hi Jindřich Bär, first of all thank you for the message and interest helping. I'm waiting for news. Thank you !
lojahub
Hello, Jindřich Bär hoping you're doing great, do you have any news regarding the problem ?
Hello and sorry for the delay.
Upon closer examination, the Actor seems to be working just fine. The networkidle0 / networkidle2 option might not be the best choice for this website, as there might be some long-running HTTP requests (think polling the API for new items etc.), which cause the Actor to never stop loading. When you switch the Performace and Limits > Navigations waits until option to load, it seems to load the page alright - check out my example run.
It's possible that over the past 2 months, the web site might have changed (so the original issues aren't there anymore).
I'll close this issue now, but feel free to reopen it (or open a new one), if you encounter any other issue with this Actor.
Again, sorry for the wait, and thank you for your patience. Cheers!
