Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.9 (41)
Pricing
Pay per usage
1544
Total users
60K
Monthly users
7.8K
Runs succeeded
>99%
Issues response
7.9 days
Last modified
3 days ago
Testing the actor but get this result
Closed
"This question is for testing whether you are a human visitor and to prevent automated spam submission. What code is in the image?"
Hi, thank you for using the Website Content Crawler.
In your case, the crawler is encountering a CAPTCHA. To overcome this issue, you need to use residential proxies and specify Denmark as the country. Please see my example run here
However, I attempted to crawl the website using a residential proxy and was blocked in half of the cases. Typically, the crawler works for most websites, but in this case, the CAPTCHA occurs very frequently, and we are unable to scrape the content successfully.
Unfortunately, there isn’t much we can do for now. I will go ahead and close this issue. If you have any other questions, please don’t hesitate to ask!
