No credit card required
Puppeteer Scraper
No credit card required
Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.
about waitforresponse
Closed
i run these code in page function, but i got the error : "Timeout exceeded while waiting for even" const correctResponse = await page.waitForResponse((response) => { return response.url.includes('/crm/web/bid/xbb/bidding/detail'); }); const data = await correctResponse.json();
how fo fix it ?
Hey there!
You're waiting for response inside pageFunction, while in the "Performance & Limits" section you have "Navigation wait until" set to ["networkidle0"] (default). pageFunction is executed after the page is loaded in accordance with the set params (I.e. in this case - when the network is idle, meaning all requests are received by the page). In other words - in this case the error is valid as the response you are waiting was probably already received while loading the page.
First thing you could try - set "Navigation wait until" to ["domcontentloaded"]. Although if response will be received before the navigation is complete - you will get the same error.
Second option - you could try to move your code to "Pre-navigation hooks" in "Advanced configuration" and save json you need to request.userData, and further process it in pageFunction (it should be available in request.userData).
Alternatively if the above would not work - you could listen for the responses inside the "Pre-navigation hooks", save json to request.userData the same way, while in pageFunction you will add some check which will wait for request.userData.json to be defined.
Something like the following (should be checked/adjusted to your specific use-case):
Inside "Pre-navigation hooks": try { page.on('response', async (response) => { const url = response.url(); if (url.includes('/crm/web/bid/xbb/bidding/detail')) request.userData.response = await response.json(); }); } catch (e) {}
Inside "Page function": while (!request.userData.response) await sleep(1000);
Hope this helps.
Apify- 255 monthly users
- 99.8% runs succeeded
- 13 days response time
- Created in Apr 2019
- Modified about 1 month ago
