Web Scraper avatar
Web Scraper

Pricing

Pay per usage

Go to Store
Web Scraper

Web Scraper

Developed by

Apify

Apify

Maintained by Apify

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (22)

Pricing

Pay per usage

691

Total users

82.3k

Monthly users

4k

Runs succeeded

>99%

Issue response

32 days

Last modified

16 days ago

SL

Crawling takes much longer

Closed

SallaR opened this issue
3 months ago

every time i run actor, it takes different execution time. from 9 sec to 5 min!

jindrich.bar avatar

Hello and thank you for your interest in this Actor!

It looks like the navigation is timing out. Under Performance & Limits > Navigation wait until, you're using networkidle2. This means the Actor waits until (almost) all network requests have finished. However, on pages with long-polling scripts or very large resources, this can cause the Actor to take much longer — or even never finish.

Switching this setting from networkidle2 to domcontentloaded seems to unblock the Actor. With this setting, the Actor will start processing the page immediately after the initial load.

I ran a test with this adjustment, and the execution time improved. You can check the fixed run here:
https://console.apify.com/view/runs/2lMzfm26K81mM5oSq

I'll go ahead and close this issue, but feel free to open a new one if you need further assistance. Cheers!