Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.9 (41)
Pricing
Pay per usage
1546
Total users
60K
Monthly users
7.8K
Runs succeeded
>99%
Issues response
7.9 days
Last modified
4 days ago
New rust http client failing on valid SSL config: SelectedUnusableCipherSuiteForVersion
Open
All urls from site recently started failing. No changes to their SSL config: WARN CheerioCrawler: Reclaiming failed request back to the list or queue. impit error: The internal HTTP library has thrown an error: 2025-07-01T16:11:29.722Z reqwest::Error { 2025-07-01T16:11:29.724Z kind: Request, 2025-07-01T16:11:29.726Z url: "https://faq.interlink.or.jp/faq2/View/wcDisplayContent.aspx?id=64", 2025-07-01T16:11:29.727Z source: hyper_util::client::legacy::Error( 2025-07-01T16:11:29.729Z Connect, 2025-07-01T16:11:29.730Z Custom { 2025-07-01T16:11:29.732Z kind: InvalidData, 2025-07-01T16:11:29.734Z error: PeerMisbehaved( 2025-07-01T16:11:29.735Z SelectedUnusableCipherSuiteForVersion, 2025-07-01T16:11:29.737Z ), 2025-07-01T16:11:29.739Z }, 2025-07-01T16:11:29.741Z ), 2025-07-01T16:11:29.742Z }
I ran an SSL labs test and it gets an A-, not the latest but should work fine. I also tried crawling with ignore HTTPS errors turned on and it's still failing.
Hello, and thank you for your interest in this Actor!
Indeed, this issue seems to be connected with our recent switch to impit as the HTTP client. I already filed an issue in the GitHub repository and we'll look into this.
In the meantime, you can switch to one of the older builds that are still using got-scraping. To do so, you simply type in the build version (0.3.66) into the Run options > Build field in the Input tab (or pass the build version through API). Note that you can pin the build version for any saved tasks as well.
I'll help you posted in case of any changes. Cheers!
