Puppeteer Scraper
Pricing
Pay per usage
Puppeteer Scraper
Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.
5.0 (5)
Pricing
Pay per usage
116
Monthly users
605
Runs succeeded
>99%
Response time
30 days
Last modified
10 months ago
about waitforresponse
i run these code in page function, but i got the error : "Timeout exceeded while waiting for even" const correctResponse = await page.waitForResponse((response) => { return response.url.includes('/crm/web/bid/xbb/bidding/detail'); }); const data = await correctResponse.json();
how fo fix it ?
Hey there!
You're waiting for response inside pageFunction, while in the "Performance & Limits" section you have "Navigation wait until" set to ["networkidle0"] (default). pageFunction is executed after the page is loaded in accordance with the set params (I.e. in this case - when the network is idle, meaning all requests are received by the page). In other words - in this case the error is valid as the response you are waiting was probably already received while loading the page.
First thing you could try - set "Navigation wait until" to ["domcontentloaded"]. Although if response will be received before the navigation is complete - you will get the same error.
Second option - you could try to move your code to "Pre-navigation hooks" in "Advanced configuration" and save json you need to request.userData, and further process it in pageFunction (it should be available in request.userData).
Alternatively if the above would not work - you could listen for the responses inside the "Pre-navigation hooks", save json to request.userData the same way, while in pageFunction you will add some check which will wait for request.userData.json to be defined.
Something like the following (should be checked/adjusted to your specific use-case):
Inside "Pre-navigation hooks": try { page.on('response', async (response) => { const url = response.url(); if (url.includes('/crm/web/bid/xbb/bidding/detail')) request.userData.response = await response.json(); ... [trimmed]
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
