Web Scraper avatar
Web Scraper

Pricing

Pay per usage

Go to Store
Web Scraper

Web Scraper

Developed by

Apify

Apify

Maintained by Apify

Crawls arbitrary websites using a web browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

4.5 (23)

Pricing

Pay per usage

920

Total users

90K

Monthly users

4.9K

Runs succeeded

>99%

Issues response

7.8 days

Last modified

2 months ago

RE

Crawler took a long time to complete, way more than average

Closed

remitaffin opened this issue
a month ago

It was just hanging, so I aborted it

MU

Mustafa99

a month ago

same with me.

jindrich.bar avatar

Hello @remitaffin, and thank you for your interest in this Actor.

From the log, the Actor was waiting to load the page but hit a navigation timeout after 60 seconds. This can happen for various reasons - for example, slow response from the target server or proxy-related delays.

To shorten the navigation stage, you can change the "waitUntil" option from "networkidle2" to "load" or "domcontentloaded".

I also see that you've since successfully scraped the target site, so I'll go ahead and close this issue. Feel free to reach out again if you run into anything else.