Web Scraper
No credit card required
Web Scraper
No credit card required
Crawls arbitrary websites using the Chrome browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.
Crawling takes much longer
every time i run actor, it takes different execution time. from 9 sec to 5 min!
Hello and thank you for your interest in this Actor!
It looks like the navigation is timing out. Under Performance & Limits > Navigation wait until, you're using networkidle2. This means the Actor waits until (almost) all network requests have finished. However, on pages with long-polling scripts or very large resources, this can cause the Actor to take much longer — or even never finish.
Switching this setting from networkidle2 to domcontentloaded seems to unblock the Actor. With this setting, the Actor will start processing the page immediately after the initial load.
I ran a test with this adjustment, and the execution time improved. You can check the fixed run here:
https://console.apify.com/view/runs/2lMzfm26K81mM5oSq
I'll go ahead and close this issue, but feel free to open a new one if you need further assistance. Cheers!
ApifyActor Metrics
3.3k monthly users
-
456 bookmarks
>99% runs succeeded
4.8 days response time
Created in Mar 2019
Modified a month ago
