Fails very often and takes quite awhile 2-4 minutes each run.

retrying the same query by running the actor again works, but hoping for a better solution thats more reliable and ideally faster too. as cost incur for failing is not low as well.

Thanks for report the issue! Seem they change their site, just now I fixed it. Please try again with latest version 0.0.24.

Thanks it works better now, runs are now ~1 min+. Most of my initial runs work but at the end there's some weird behavior where it claims multiple results are being returned but viewing it only shows 1. there are a few failures near the end as well not sure why,

Here's an example of one of the fails below 2024-08-22T18:24:06.523Z ACTOR: Pulling Docker image of build HMr6kFCNLcnClmFPF from repository. 2024-08-22T18:24:08.559Z ACTOR: Creating Docker container. 2024-08-22T18:24:12.155Z ACTOR: Starting Docker container. 2024-08-22T18:24:13.535Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1920x1080x24+32 -nolisten tcp 2024-08-22T18:24:13.541Z Executing main command 2024-08-22T18:24:14.534Z INFO System info {"apifyVersion":"3.2.4","apifyClientVersion":"2.9.4","crawleeVersion":"3.11.1","osType":"Linux","nodeVersion":"v18.20.4"} 2024-08-22T18:24:18.459Z INFO Configuring Web Scraper. 2024-08-22T18:24:42.187Z WARN 2024-08-22T18:24:42.188Z ***************************************************************** 2024-08-22T18:24:42.190Z * Web Scraper is running in DEVELOPMENT MODE! * 2024-08-22T18:24:42.192Z * Concurrency is limited, sessionPool is not available, * 2024-08-22T18:24:42.193Z * timeouts are increased and debugger is enabled. * 2024-08-22T18:24:42.195Z * If you want full control and performance switch * 2024-08-22T18:24:42.196Z * Run type to PRODUCTION! * 2024-08-22T18:24:42.197Z ***************************************************************** 2024-08-22T18:24:42.198Z 2024-08-22T18:24:42.199Z INFO Configuration completed. Starting the scrape. 2024-08-22T18:24:42.561Z INFO PuppeteerCrawler: Starting the crawler. 2024-08-22T18:24:43.564Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:24:53.562Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:24:59.587Z devtools-server starting. 2024-08-22T18:24:59.589Z devtools-server listening on port: 4321 2024-08-22T18:25:03.607Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:25:13.591Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:25:23.598Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:25:33.603Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:25:42.562Z INFO PuppeteerCrawler:Statistics: PuppeteerCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":0,"requestTotalDurationMillis":0,"requestsTotal":0,"crawlerRuntimeMillis":62662,"retryHistogram":[]} 2024-08-22T18:25:43.605Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:25:43.699Z INFO PuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":1,"desiredConcurrency":1,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0.079},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0.368},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}} 2024-08-22T18:25:53.488Z WARN Puppeteer Utils: An error occurred during the script injection! {"error":{"name":"TypeError","message":"Cannot read properties of undefined (reading 'createElement')","stack":"TypeError: Cannot read properties of undefined (reading 'createElement')\n at assert (evaluate at /home/myuser/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:83:14, :978:19)\n at evaluate (evaluate at /home/myuser/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:83:14, :2694:23)\n at evaluate (evaluate at /home/myuser/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:83:14, :2722:3)\n at evaluate (evaluate at /home/myuser/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:83:14, :32:2)\n at evaluate (evaluate at /home/myuser/node_modules/@crawlee/puppeteer/internals/utils/puppeteer_utils.js:83:14, :36:3)\n at #evaluate (/home/myuser/node_modules... [line-too-long] 2024-08-22T18:25:53.617Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:26:03.610Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:26:23.609Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:26:23.915Z INFO PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down. 2024-08-22T18:26:33.325Z INFO PuppeteerCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":83168,"requestsFinishedPerMinute":1,"requestsFailedPerMinute":0,"requestTotalDurationMillis":83168,"requestsTotal":1,"crawlerRuntimeMillis":113425} 2024-08-22T18:26:33.326Z INFO PuppeteerCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true} 2024-08-22T18:26:33.609Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:26:34.326Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:26:34.328Z INFO Web Scraper finished.

2024-08-22T18:21:56.533Z ACTOR: Pulling Docker image of build HMr6kFCNLcnClmFPF from repository. 2024-08-22T18:22:16.275Z ACTOR: Creating Docker container. 2024-08-22T18:22:23.205Z ACTOR: Starting Docker container. 2024-08-22T18:22:24.626Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1920x1080x24+32 -nolisten tcp 2024-08-22T18:22:24.627Z Executing main command 2024-08-22T18:22:26.556Z INFO System info {"apifyVersion":"3.2.4","apifyClientVersion":"2.9.4","crawleeVersion":"3.11.1","osType":"Linux","nodeVersion":"v18.20.4"} 2024-08-22T18:22:27.326Z INFO Configuring Web Scraper. 2024-08-22T18:22:39.090Z WARN 2024-08-22T18:22:39.092Z ***************************************************************** 2024-08-22T18:22:39.093Z * Web Scraper is running in DEVELOPMENT MODE! * 2024-08-22T18:22:39.095Z * Concurrency is limited, sessionPool is not available, * 2024-08-22T18:22:39.096Z * timeouts are increased and debugger is enabled. * 2024-08-22T18:22:39.097Z * If you want full control and performance switch * 2024-08-22T18:22:39.098Z * Run type to PRODUCTION! * 2024-08-22T18:22:39.099Z ***************************************************************** 2024-08-22T18:22:39.102Z 2024-08-22T18:22:39.103Z INFO Configuration completed. Starting the scrape. 2024-08-22T18:22:39.225Z INFO PuppeteerCrawler: Starting the crawler. 2024-08-22T18:22:41.047Z devtools-server starting. 2024-08-22T18:22:41.049Z devtools-server listening on port: 4321 2024-08-22T18:22:50.226Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:23:10.226Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:23:20.226Z WARN PuppeteerCrawler: Setting status message timed out after 1s 2024-08-22T18:23:29.890Z INFO PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down. 2024-08-22T18:23:31.008Z INFO PuppeteerCrawler: Final request statistics: {"requestsFinished":1,"requestsFailed":0,"retryHistogram":[1],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":48477,"requestsFinishedPerMinute":1,"requestsFailedPerMinute":0,"requestTotalDurationMillis":48477,"requestsTotal":1,"crawlerRuntimeMillis":54274} 2024-08-22T18:23:31.009Z INFO PuppeteerCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true} 2024-08-22T18:23:31.733Z INFO Web Scraper finished.

Hi! Could you please give the inputs that make the tasks failed, so that I can check the issue easily.

I just do some test and it work fine. Some tips:

1. Add multiple search terms into one task: this will run faster then do one by one. Remember to increase timeout if need
2. Always using residential proxy

Sure, here are the tasks with fails and weird number of results below.

Failed https://console.apify.com/actors/OWZduFLjIazgQu9Id/runs/8Vlt1aHCa0P8Zmplq#log https://console.apify.com/actors/OWZduFLjIazgQu9Id/runs/o2dZYBDkEI16rbfcd https://console.apify.com/actors/OWZduFLjIazgQu9Id/runs/IsnF8wqHzBNNjaao6 https://console.apify.com/actors/OWZduFLjIazgQu9Id/runs/jvc9JcoScmfqrYhd8

Weird multiple results mismatch claims 3 results but clicking in shows 1 - https://console.apify.com/actors/OWZduFLjIazgQu9Id/runs/aSrEjpnDSlv3kRnEB claims 2 results but clciking in shows 1 - https://console.apify.com/actors/OWZduFLjIazgQu9Id/runs/uiJHAkRJ05ZyNb2lx