![Website Content Crawler avatar](https://images.apifyusercontent.com/1VrdawICnxIwM4X5JzRJHPBmLx0OpmiNxtHGGLmxdu8/rs:fill:92:92/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
Website Content Crawler
Try for free
No credit card required
View all Actors![Website Content Crawler](https://images.apifyusercontent.com/1VrdawICnxIwM4X5JzRJHPBmLx0OpmiNxtHGGLmxdu8/rs:fill:92:92/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
![janbuchar avatar](https://apify-image-uploads-prod.s3.us-east-1.amazonaws.com/4yfxkhP2QSajkpybN/Z6gszzr1SgoYXNz8H-DSCF1317_crop.jpg)
![Website Content Crawler](https://images.apifyusercontent.com/1VrdawICnxIwM4X5JzRJHPBmLx0OpmiNxtHGGLmxdu8/rs:fill:92:92/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
Website Content Crawler
apify/website-content-crawler
Try for free
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Cloudflare challenge failed
Closed
mihai opened this issue
5 months ago
Crawl failed due to cloudflare, what can be done?
![janbuchar avatar](https://apify-image-uploads-prod.s3.us-east-1.amazonaws.com/4yfxkhP2QSajkpybN/Z6gszzr1SgoYXNz8H-DSCF1317_crop.jpg)
5 months ago
Hello, and thank you for your interest in the actor!
I see that you already used the stealthy firefox crawler type and have a residential proxy enabled. The only recommendation I can give you is to change the proxy group to something in Europe, instead of the US, in the actor input settings.
Let us know if that helps!
Developer
Maintained by Apify
Actor metrics
- 2.8k monthly users
- 317 stars
- 100.0% runs succeeded
- 4 days response time
- Created in Mar 2023
- Modified 1 day ago
Categories