Similarweb Scraper

Hello, it does not work I got this erorr: 2022-11-21T12:55:47.942Z ERROR CheerioCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue {"url":"https://similarweb.com/website/https://www.zeta-shoes.com/en","retryCount":1,"id":"P44n9ZdbHI7VWb6"} 2022-11-21T12:55:47.945Z TypeError: Cannot read properties of null (reading 'substr') 2022-11-21T12:55:47.948Z at parseAppData (/usr/src/app/src/routes.js:13:22) 2022-11-21T12:55:47.949Z at exports.handleStart (/usr/src/app/src/routes.js:26:21) 2022-11-21T12:55:47.949Z at CheerioCrawler.handlePageFunction [as userProvidedHandler] (/usr/src/app/main.js:38:28) 2022-11-21T12:55:47.950Z at /usr/src/app/node_modules/apify/build/crawlers/cheerio_crawler.js:516:62 2022-11-21T12:55:47.951Z at wrap (/usr/src/app/node_modules/@apify/timeout/index.js:73:33) 2022-11-21T12:55:47.952Z at /usr/src/app/node_modules/@apify/timeout/index.js:88:13 2022-11-21T12:55:47.952Z at AsyncLocalStorage.run (node:async_hooks:319:14) 2022-11-21T12:55:47.953Z at /usr/src/app/node_modules/@apify/timeout/index.js:87:25 2022-11-21T12:55:47.954Z at new Promise (

I guess we waiting the fix from Lukas, but it's like 21 days he don't show up

Hi, sorry for the huge delay, I'll work on a fix now

Turns out they implemented aggressive bot detection, so I'll have to rework the entire scraper, so it might take a while.. I will try to figure out a possible solution.

Hi Lukas were you able to fix the issue?

I still have the same issue as this this thread currently.

2023-01-11T07:22:40.659Z DEBUG CheerioCrawler:SessionPool: Created new Session - session_XO6v8cYNuF 2023-01-11T07:22:41.032Z DEBUG Page opened. {"url":"https://similarweb.com/website/maybank.com"} 2023-01-11T07:22:41.151Z ERROR CheerioCrawler: Request failed and reached maximum retries {"id":"amRDy7Ft4GmPm3D","url":"https://similarweb.com/website/maybank.com","method":"GET","uniqueKey":"https://similarweb.com/website/maybank.com"} 2023-01-11T07:22:41.153Z TypeError: Cannot read properties of null (reading 'substr') 2023-01-11T07:22:41.155Z at parseAppData (/usr/src/app/src/routes.js:13:22) 2023-01-11T07:22:41.156Z at exports.handleStart (/usr/src/app/src/routes.js:26:21) 2023-01-11T07:22:41.158Z at CheerioCrawler.handlePageFunction [as userProvidedHandler] (/usr/src/app/main.js:38:28) 2023-01-11T07:22:41.160Z at /usr/src/app/node_modules/apify/build/crawlers/cheerio_crawler.js:516:62 2023-01-11T07:22:41.161Z at wrap (/usr/src/app/node_modules/@apify/timeout/index.js:73:33) 2023-01-11T07:22:41.163Z at /usr/src/app/node_modules/@apify/timeout/index.js:88:13 2023-01-11T07:22:41.165Z at AsyncLocalStorage.run (node:async_hooks:319:14) 2023-01-11T07:22:41.166Z at /usr/src/app/node_modules/@apify/timeout/index.js:87:25 2023-01-11T07:22:41.168Z at new Promise (

I just tried, I get same error as above an no results

Hi everyone, it's Misa from Apify. We're currently working on an update of this scraper and will inform you once it's done. Sorry for any inconvenience you've experienced with the scraper malfunctioning in the meantime.