Shopee Api Scraper

Until yesterday, I was able to crawl the Search and Category pages. However, it stopped functioning today. Here’s a sample log for you to examine.

2024-03-26T07:36:01.758Z ACTOR: Pulling Docker image of build Hxt9I5HGeNpZglgpG from repository. 2024-03-26T07:36:33.719Z ACTOR: Creating Docker container. 2024-03-26T07:36:33.959Z ACTOR: Starting Docker container. 2024-03-26T07:36:35.940Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1920x1080x24+32 -nolisten tcp 2024-03-26T07:36:35.942Z Executing main command 2024-03-26T07:36:37.335Z INFO System info {"apifyVersion":"3.1.16","apifyClientVersion":"2.9.3","crawleeVersion":"3.8.2","osType":"Linux","nodeVersion":"v20.11.1"} 2024-03-26T07:36:38.345Z 2024-03-26 07:36:38.344 INFO SinglePagePuppeteerCrawler: Starting the crawler. 2024-03-26T07:37:02.871Z 2024-03-26 07:37:02.871 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:37:02.873Z Website not loaded properly 2024-03-26T07:37:02.875Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":1} 2024-03-26T07:37:23.301Z 2024-03-26 07:37:23.300 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:37:23.302Z Website not loaded properly 2024-03-26T07:37:23.304Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":2} 2024-03-26T07:37:38.344Z 2024-03-26 07:37:38.344 INFO Statistics: SinglePagePuppeteerCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":0,"requestTotalDurationMillis":0,"requestsTotal":0,"crawlerRuntimeMillis":60148,"retryHistogram":[]} 2024-03-26T07:37:38.385Z 2024-03-26 07:37:38.384 INFO SinglePagePuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":1,"desiredConcurrency":20,"systemStatus":{"isSystemIdle":false,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0.038},"cpuInfo":{"isOverloaded":true,"limitRatio":0.4,"actualRatio":0.468},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}} 2024-03-26T07:37:51.801Z 2024-03-26 07:37:51.800 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:37:51.803Z Website not loaded properly 2024-03-26T07:37:51.804Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":3} 2024-03-26T07:38:04.225Z 2024-03-26 07:38:04.224 WARN SinglePagePuppeteerCrawler: Unable to cache the response https://shopee.co.id/api/v2/authentication/qrcode_status?qrcode_id=wnnrKXGFtj2cBnmvoMKy9ROfTxIr%2Be%2FhQzttFNz%2FA8U%3D: ProtocolError: Could not load body for this request. This might happen if the request is a preflight request. 2024-03-26T07:38:11.895Z 2024-03-26 07:38:11.894 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:38:11.897Z Website not loaded properly 2024-03-26T07:38:11.898Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":4} 2024-03-26T07:38:33.230Z 2024-03-26 07:38:33.229 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:38:33.232Z Website not loaded properly 2024-03-26T07:38:33.233Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":5} 2024-03-26T07:38:38.375Z 2024-03-26 07:38:38.374 INFO Statistics: SinglePagePuppeteerCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":0,"requestTotalDurationMillis":0,"requestsTotal":0,"crawlerRuntimeMillis":120178,"retryHistogram":[]} 2024-03-26T07:38:38.416Z 2024-03-26 07:38:38.388 INFO SinglePagePuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":1,"desiredConcurrency":20,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0.038},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0.355},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}} 2024-03-26T07:38:45.817Z 2024-03-26 07:38:45.816 WARN SinglePagePuppeteerCrawler: Unable to cache the response https://shopee.co.id/api/v2/authentication/qrcode_status?qrcode_id=Y7WaJGWxIYk2%2F%2FmrFAoeXHorX0SC2vYdSCGRbFyNGsE%3D: ProtocolError: Could not load body for this request. This might happen if the request is a preflight request. 2024-03-26T07:38:53.404Z 2024-03-26 07:38:53.403 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:38:53.405Z Website not loaded properly 2024-03-26T07:38:53.406Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":6} 2024-03-26T07:39:06.595Z 2024-03-26 07:39:06.594 WARN SinglePagePuppeteerCrawler: Unable to cache the response https://shopee.co.id/api/v2/authentication/qrcode_status?qrcode_id=GnKi8Rp6AYz8hd6eYVEetIyVwE66NEecPge3N9Bi5fM%3D: ProtocolError: Could not load body for this request. This might happen if the request is a preflight request. 2024-03-26T07:39:14.999Z 2024-03-26 07:39:14.998 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:39:15.000Z Website not loaded properly 2024-03-26T07:39:15.001Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":7} 2024-03-26T07:39:36.825Z 2024-03-26 07:39:36.824 WARN SinglePagePuppeteerCrawler: Reclaiming failed request back to the list or queue. Detected a session error, rotating session... 2024-03-26T07:39:36.826Z Website not loaded properly 2024-03-26T07:39:36.827Z at SinglePagePuppeteerCrawler.postNavigationHooks (file:///home/myuser/dist/actor.js:117:27) {"id":"knmwoQzcSXFXDcj","url":"https://shopee.co.id/api/v4/search/search_items?by=relevancy&keyword=laptop","retryCount":8} 2024-03-26T07:39:38.377Z 2024-03-26 07:39:38.376 INFO Statistics: SinglePagePuppeteerCrawler request statistics: {"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":null,"requestsFinishedPerMinute":0,"requestsFailedPerMinute":0,"requestTotalDurationMillis":0,"requestsTotal":0,"crawlerRuntimeMillis":180180,"retryHistogram":[]} 2024-03-26T07:39:38.399Z 2024-03-26 07:39:38.399 INFO SinglePagePuppeteerCrawler:AutoscaledPool: state {"currentConcurrency":0,"desiredConcurrency":20,"systemStatus":{"isSystemIdle":true,"memInfo":{"isOverloaded":false,"limitRatio":0.2,"actualRatio":0},"eventLoopInfo":{"isOverloaded":false,"limitRatio":0.6,"actualRatio":0},"cpuInfo":{"isOverloaded":false,"limitRatio":0.4,"actualRatio":0.369},"clientInfo":{"isOverloaded":false,"limitRatio":0.3,"actualRatio":0}}} 2024-03-26T07:39:44.691Z ACTOR: The Actor run was aborted by the user.