Dun & Bradstreet Scraper

2024-03-09T12:03:07.554Z ACTOR: Pulling Docker image of build et0crKtIX0LfR2a24 from repository. 2024-03-09T12:03:16.369Z ACTOR: Creating Docker container. 2024-03-09T12:03:16.549Z ACTOR: Starting Docker container. 2024-03-09T12:03:18.788Z INFO System info {"apifyVersion":"3.1.12","apifyClientVersion":"2.8.0","crawleeVersion":"3.5.7","osType":"Linux","nodeVersion":"v18.19.1"} 2024-03-09T12:03:18.792Z INFO PHASE -- STARTING ACTOR. 2024-03-09T12:03:18.937Z INFO 2024-03-09T12:03:18.940Z ========================================================== 2024-03-09T12:03:18.942Z REMINDER: US-based proxies are required to use this actor 2024-03-09T12:03:18.944Z ========================================================== 2024-03-09T12:03:18.946Z 2024-03-09T12:03:19.093Z INFO PHASE -- SETTING UP CRAWLER. 2024-03-09T12:03:19.577Z INFO CRAWLER STARTED. 2024-03-09T12:03:19.906Z INFO PuppeteerCrawler: Starting the crawler. 2024-03-09T12:03:22.700Z INFO CRAWLER: -- Searching companies with keyword: web development, page: 1 2024-03-09T12:03:22.740Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. We got blocked. Retrying. 2024-03-09T12:03:22.742Z {"id":"xi9vvcpFIjWkRar","url":"https://www.dnb.com/sitemap.xml","retryCount":1} 2024-03-09T12:03:27.030Z INFO CRAWLER: -- Searching companies with keyword: web development, page: 1 2024-03-09T12:03:27.069Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. We got blocked. Retrying. 2024-03-09T12:03:27.071Z {"id":"xi9vvcpFIjWkRar","url":"https://www.dnb.com/sitemap.xml","retryCount":2} 2024-03-09T12:03:31.145Z INFO CRAWLER: -- Searching companies with keyword: web development, page: 1 2024-03-09T12:03:31.198Z WARN PuppeteerCrawler: Reclaiming failed request back to the list or queue. We got blocked. Retrying. 2024-03-09T12:03:31.201Z {"id":"xi9vvcpFIjWkRar","url":"https://www.dnb.com/sitemap.xml","retryCount":3} 2024-03-09T12:03:35.646Z INFO CRAWLER: -- Searching companies with keyword: web development, page: 1 2024-03-09T12:03:38.239Z ERROR PuppeteerCrawler: Request failed and reached maximum retries. We got blocked. Retrying. 2024-03-09T12:03:38.241Z at PuppeteerCrawler._handleFailedRequestHandler (/home/myuser/node_modules/@crawlee/basic/internals/basic-crawler.js:951:30) 2024-03-09T12:03:38.244Z at PuppeteerCrawler._requestFunctionErrorHandler (/home/myuser/node_modules/@crawlee/basic/internals/basic-crawler.js:937:20) 2024-03-09T12:03:38.246Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2024-03-09T12:03:38.248Z at async wrap (/home/myuser/node_modules/@apify/timeout/index.js:52:21) {"id":"xi9vvcpFIjWkRar","url":"https://www.dnb.com/sitemap.xml","method":"GET","uniqueKey":"COMPANIES-1"} 2024-03-09T12:03:38.773Z INFO PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down. 2024-03-09T12:03:39.479Z INFO PuppeteerCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":1,"retryHistogram":[null,null,null,1],"requestAvgFailedDurationMillis":3558,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":3,"requestTotalDurationMillis":3558,"requestsTotal":1,"crawlerRuntimeMillis":19904} 2024-03-09T12:03:39.507Z INFO PuppeteerCrawler: Error analysis: {"totalErrors":1,"uniqueErrors":1,"mostCommonErrors":["1x: We got blocked. Retrying. (missing stack trace)"]} 2024-03-09T12:03:39.509Z INFO PuppeteerCrawler: Finished! Total 1 requests: 0 succeeded, 1 failed. {"terminal":true}