Zoopla Scraper

2023-12-10T23:55:05.920Z ACTOR: Pulling Docker image of build Xf4lDUyzzJcW0lJUf from repository. 2023-12-10T23:55:07.868Z ACTOR: Creating Docker container. 2023-12-10T23:55:07.946Z ACTOR: Starting Docker container. 2023-12-10T23:55:09.829Z INFO System info {"apifyVersion":"3.1.11","apifyClientVersion":"2.8.0","crawleeVersion":"3.5.7","osType":"Linux","nodeVersion":"v18.18.2"} 2023-12-10T23:55:09.836Z INFO PHASE -- STARTING ACTOR. 2023-12-10T23:55:10.070Z INFO PHASE -- SETTING UP CRAWLER. 2023-12-10T23:55:10.072Z INFO CRAWLER STARTED. 2023-12-10T23:55:10.260Z INFO BasicCrawler: Starting the crawler. 2023-12-10T23:55:13.496Z WARN BasicCrawler: Reclaiming failed request back to the list or queue. TypeError: Cannot read properties of null (reading 'listings') 2023-12-10T23:55:13.498Z at Object.scrapeList (file:///usr/src/app/src/extractors.js:4:27) 2023-12-10T23:55:13.500Z at file:///usr/src/app/src/routes.js:20:30 2023-12-10T23:55:13.502Z at func (/usr/src/app/node_modules/@crawlee/core/router.js:172:44) 2023-12-10T23:55:13.504Z at BasicCrawler.requestHandler (file:///usr/src/app/src/main.js:64:15) 2023-12-10T23:55:13.506Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-12-10T23:55:13.508Z at async BasicCrawler._runRequestHandler (/usr/src/app/node_modules/@crawlee/basic/internals/basic-crawler.js:645:9) 2023-12-10T23:55:13.510Z at async wrap (/usr/src/app/node_modules/@apify/timeout/index.js:52:21) {"id":"3K6zLIbaWO4ovMM","url":"https://api-graphql-lambda.prod.zoopla.co.uk/graphql","retryCount":1} 2023-12-10T23:55:19.952Z WARN BasicCrawler: Reclaiming failed request back to the list or queue. TypeError: Cannot read properties of null (reading 'listings') 2023-12-10T23:55:19.954Z at Object.scrapeList (file:///usr/src/app/src/extractors.js:4:27) 2023-12-10T23:55:19.956Z at file:///usr/src/app/src/routes.js:20:30 2023-12-10T23:55:19.958Z at func (/usr/src/app/node_modules/@crawlee/core/router.js:172:44) 2023-12-10T23:55:19.960Z at BasicCrawler.requestHandler (file:///usr/src/app/src/main.js:64:15) 2023-12-10T23:55:19.962Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-12-10T23:55:19.964Z at async BasicCrawler._runRequestHandler (/usr/src/app/node_modules/@crawlee/basic/internals/basic-crawler.js:645:9) 2023-12-10T23:55:19.966Z at async wrap (/usr/src/app/node_modules/@apify/timeout/index.js:52:21) {"id":"3K6zLIbaWO4ovMM","url":"https://api-graphql-lambda.prod.zoopla.co.uk/graphql","retryCount":2} 2023-12-10T23:55:25.415Z WARN BasicCrawler: Reclaiming failed request back to the list or queue. TypeError: Cannot read properties of null (reading 'listings') 2023-12-10T23:55:25.418Z at Object.scrapeList (file:///usr/src/app/src/extractors.js:4:27) 2023-12-10T23:55:25.420Z at file:///usr/src/app/src/routes.js:20:30 2023-12-10T23:55:25.422Z at func (/usr/src/app/node_modules/@crawlee/core/router.js:172:44) 2023-12-10T23:55:25.424Z at BasicCrawler.requestHandler (file:///usr/src/app/src/main.js:64:15) 2023-12-10T23:55:25.427Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-12-10T23:55:25.429Z at async BasicCrawler._runRequestHandler (/usr/src/app/node_modules/@crawlee/basic/internals/basic-crawler.js:645:9) 2023-12-10T23:55:25.431Z at async wrap (/usr/src/app/node_modules/@apify/timeout/index.js:52:21) {"id":"3K6zLIbaWO4ovMM","url":"https://api-graphql-lambda.prod.zoopla.co.uk/graphql","retryCount":3} 2023-12-10T23:55:32.044Z ERROR BasicCrawler: Request failed and reached maximum retries. TypeError: Cannot read properties of null (reading 'listings') 2023-12-10T23:55:32.046Z at Object.scrapeList (file:///usr/src/app/src/extractors.js:4:27) 2023-12-10T23:55:32.048Z at file:///usr/src/app/src/routes.js:20:30 2023-12-10T23:55:32.050Z at func (/usr/src/app/node_modules/@crawlee/core/router.js:172:44) 2023-12-10T23:55:32.052Z at BasicCrawler.requestHandler (file:///usr/src/app/src/main.js:64:15) 2023-12-10T23:55:32.054Z at process.processTicksAndRejections (node:internal/process/task_queues:95:5) 2023-12-10T23:55:32.056Z at async BasicCrawler._runRequestHandler (/usr/src/app/node_modules/@crawlee/basic/internals/basic-crawler.js:645:9) 2023-12-10T23:55:32.058Z at async wrap (/usr/src/app/node_modules/@apify/timeout/index.js:52:21) {"id":"3K6zLIbaWO4ovMM","url":"https://api-graphql-lambda.prod.zoopla.co.uk/graphql","method":"GET","uniqueKey":"https://www.zoopla.co.uk/for-sale/map/property/bethnal-green/?is_auction=false&is_retirement_home=false&is_shared_ownership=false&q=Bethnal%20Green%2C%20London&results_sort=newest_listings&search_source=for-sale"} 2023-12-10T23:55:32.092Z INFO BasicCrawler: All requests from the queue have been processed, the crawler will shut down. 2023-12-10T23:55:32.283Z INFO BasicCrawler: Final request statistics: {"requestsFinished":0,"requestsFailed":1,"retryHistogram":[null,null,null,1],"requestAvgFailedDurationMillis":3465,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":2,"requestTotalDurationMillis":3465,"requestsTotal":1,"crawlerRuntimeMillis":22209} 2023-12-10T23:55:32.286Z INFO BasicCrawler: Error analysis: {"totalErrors":1,"uniqueErrors":1,"mostCommonErrors":["1x: Cannot read properties of null (reading 'listings') (file:///usr/src/app/src/extractors.js:4:27)"]} 2023-12-10T23:55:32.288Z INFO BasicCrawler: Finished! Total 1 requests: 0 succeeded, 1 failed. {"terminal":true}