Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.9 (41)
Pricing
Pay per usage
1545
Total users
60K
Monthly users
7.8K
Runs succeeded
>99%
Issues response
7.9 days
Last modified
3 days ago
Failed to process HTML with 'readableText', falling back to 'none'
Closed
Hello, I am using the CheerioCrawler and for some websites it gives the following warnings:
There are no issues if I use Playwright. What could be the reason for this? The Website (I don't want to mention it here) seems normal, nothing unusual from my point of view.
Thanks!
Hello and thank you for your interest in this Actor!
The website from your run uses client-side JS to render the content. This means, in the initial HTTP request, the server sends just a bunch of links to Javascript files that the client has to execute, render the DOM elements dynamically and perhaps load more content with other requests.
While regular browsers can do all this (and therefore can load the webpage properly), Cheerio cannot - it only sends the initial request to the server (let's say, example.com) and finishes once it gets the response to this request (but at that point, there is just a bunch of links to scripts and stylesheets in there, no real content).
This is also why we're offering different crawler types - while Cheerio is blazing fast, it simply cannot load all websites. You can read more about dynamic pages and SPAs in our Academy.
If you're unsure about what crawler type you should use, you can always try the Adaptive crawler type - it automatically switches between Playwright and Cheerio based on the content of the current webpage. Just keep in mind that it's still experimental - it works just fine 99% of the time, but we're still gathering feedback for it. If you decide to give it a try, let us know how it went!
I'll close this issue now, but feel free to reopen it if you have any additional questions. Cheers!
