Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Remove Browser depreciation warnings
Hello, for some web pages the crawler returns browser depreciation warnings such as "Your browser is not supported anymore". This happens with Cheerio. Is there any way of removing these? Is this a Cheerio thing? Unfortunately, there are no decent css selectors, just plain h2 and p tags. Thanks!
Hi, thank you for using the Website Content Crawler.
In the runId you attached, there is a warning about dynamicContentWaitSecs:
2025-01-21T09:20:48.943Z WARN A non-zero value of the `dynamicContentWaitSecs` option is not supported by the Cheerio crawler.
But I haven't seen the issues you are mentioning. I randomly checked a few of your runs but didn’t see this issue recurring.
Could you please provide a runId where you’re encountering the problem?
Thank you, Jiri
cirez_d
I should have elaborated: These warnings appear in the html, not in any logs. You can see it at the bottom of the html. I know this may be out of scope, but I have seen a couple of websites do this and it is annoying
Hi, thank you for the clarification
The warnings like "Your browser is not supported anymore" are part of the HTML and are captured by Cheerio since it processes raw page content. The challenge is that these messages vary by website and language (e.g., "Ihr Browser wird nicht mehr unterstützt Mehr"), making them difficult to filter out automatically.
I've logged this at our GitHub for tracking, but I'm afraid, it might not get prioritized soon. I'm sorry for the inconvenience. Thank you, Jiri
ApifyActor Metrics
5.4k monthly users
-
990 bookmarks
>99% runs succeeded
1 days response time
Created in Mar 2023
Modified 13 days ago
