Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
4.6 (38)
Pricing
Pay per usage
1.1k
Monthly users
5.8k
Runs succeeded
>99%
Response time
2.4 days
Last modified
10 days ago
Decode non-UTF-8 text in crawlerType cheerio
When using 'cheerio' mode to crawl a non-UTF-8 encoded page, the page content is not decoded correctly (the encoding can be obtained from the html header charset part). However, the decoding is correct in playwright:* mode. For example: http://finance.ce.cn/stock/gsgdbd/202207/01/t20220701_37824007.shtml
Hello and thank you for your interest in this Actor!
This seems to be an issue with our underlying open-source library Crawlee and the way it determines document encoding in case the server doesn't send a Content-Encoding HTTP header.
I already submitted an issue in Crawlee's GitHub repo here. While we exactly know where the issue is (and how to fix it), the "correct" way of fixing it is quite hard.
I'll take a look at it soon and will let you know once there are any changes.
Thank you again (and sorry for the inconvenience)!
Pricing
Pricing model
Pay per usageThis Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.
