Website Content Crawler avatar
Website Content Crawler
Try for free

No credit card required

View all Actors
Website Content Crawler

Website Content Crawler

apify/website-content-crawler
Try for free

No credit card required

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

User avatar

Decode non-UTF-8 text in crawlerType cheerio

Open

consoling_knock opened this issue
2 months ago

When using 'cheerio' mode to crawl a non-UTF-8 encoded page, the page content is not decoded correctly (the encoding can be obtained from the html header charset part). However, the decoding is correct in playwright:* mode. For example: http://finance.ce.cn/stock/gsgdbd/202207/01/t20220701_37824007.shtml

User avatar

Hello and thank you for your interest in this Actor!

This seems to be an issue with our underlying open-source library Crawlee and the way it determines document encoding in case the server doesn't send a Content-Encoding HTTP header. I already submitted an issue in Crawlee's GitHub repo here. While we exactly know where the issue is (and how to fix it), the "correct" way of fixing it is quite hard.

I'll take a look at it soon and will let you know once there are any changes.

Thank you again (and sorry for the inconvenience)!

Developer
Maintained by Apify
Actor metrics
  • 2k monthly users
  • 99.6% runs succeeded
  • 2.7 days response time
  • Created in Mar 2023
  • Modified 6 days ago