Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.7 (41)
Pricing
Pay per usage
1531
Total users
59K
Monthly users
7.9K
Runs succeeded
>99%
Issues response
7.6 days
Last modified
5 days ago
Remove Browser depreciation warnings
Closed
Hello, for some web pages the crawler returns browser depreciation warnings such as "Your browser is not supported anymore". This happens with Cheerio. Is there any way of removing these? Is this a Cheerio thing? Unfortunately, there are no decent css selectors, just plain h2 and p tags. Thanks!
Hi, thank you for using the Website Content Crawler.
In the runId you attached, there is a warning about dynamicContentWaitSecs:
But I haven't seen the issues you are mentioning. I randomly checked a few of your runs but didn’t see this issue recurring.
Could you please provide a runId where you’re encountering the problem?
Thank you, Jiri
cirez_d
I should have elaborated: These warnings appear in the html, not in any logs. You can see it at the bottom of the html. I know this may be out of scope, but I have seen a couple of websites do this and it is annoying
Hi, thank you for the clarification
The warnings like "Your browser is not supported anymore" are part of the HTML and are captured by Cheerio since it processes raw page content. The challenge is that these messages vary by website and language (e.g., "Ihr Browser wird nicht mehr unterstützt Mehr"), making them difficult to filter out automatically.
I've logged this at our GitHub for tracking, but I'm afraid, it might not get prioritized soon. I'm sorry for the inconvenience. Thank you, Jiri
