No credit card required
Website Content Crawler
No credit card required
Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.
Failed to process HTML with 'readableText', falling back to 'none'
Closed
Hello, I am using the CheerioCrawler and for some websites it gives the following warnings:
12024-03-26T11:50:31.118Z INFO System info {"apifyVersion":"3.1.16","apifyClientVersion":"2.9.0","crawleeVersion":"3.8.1","osType":"Linux","nodeVersion":"v18.19.1"} 22024-03-26T11:50:31.261Z WARN A non-zero value of the `dynamicContentWaitSecs` option is not supported by the Cheerio crawler 32024-03-26T11:50:31.741Z INFO CheerioCrawler: Starting the crawler. 42024-03-26T11:50:33.341Z WARN Failed to process HTML with 'readableText', falling back to 'none' for URL _URL_ 52024-03-26T11:50:33.344Z WARN No text parsed from _URL_. 62024-03-26T11:50:33.413Z INFO No links found on _URL_.
There are no issues if I use Playwright. What could be the reason for this? The Website (I don't want to mention it here) seems normal, nothing unusual from my point of view.
Thanks!
Hello and thank you for your interest in this Actor!
The website from your run uses client-side JS to render the content. This means, in the initial HTTP request, the server sends just a bunch of links to Javascript files that the client has to execute, render the DOM elements dynamically and perhaps load more content with other requests.
While regular browsers can do all this (and therefore can load the webpage properly), Cheerio cannot - it only sends the initial request to the server (let's say, example.com) and finishes once it gets the response to this request (but at that point, there is just a bunch of links to scripts and stylesheets in there, no real content).
This is also why we're offering different crawler types - while Cheerio is blazing fast, it simply cannot load all websites. You can read more about dynamic pages and SPAs in our Academy.
If you're unsure about what crawler type you should use, you can always try the Adaptive crawler type - it automatically switches between Playwright and Cheerio based on the content of the current webpage. Just keep in mind that it's still experimental - it works just fine 99% of the time, but we're still gathering feedback for it. If you decide to give it a try, let us know how it went!
I'll close this issue now, but feel free to reopen it if you have any additional questions. Cheers!
Apify- 1.9k monthly users
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago
