Website Content Crawler
No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoTesting the actor but get this result
Closed
"This question is for testing whether you are a human visitor and to prevent automated spam submission. What code is in the image?"
Hi, thank you for using the Website Content Crawler.
In your case, the crawler is encountering a CAPTCHA. To overcome this issue, you need to use residential proxies and specify Denmark as the country. Please see my example run here
However, I attempted to crawl the website using a residential proxy and was blocked in half of the cases. Typically, the crawler works for most websites, but in this case, the CAPTCHA occurs very frequently, and we are unable to scrape the content successfully.
Unfortunately, there isn’t much we can do for now. I will go ahead and close this issue. If you have any other questions, please don’t hesitate to ask!
ApifyActor Metrics
3.9k monthly users
-
718 stars
>99% runs succeeded
2.2 days response time
Created in Mar 2023
Modified 15 hours ago
