Website Content Crawler avatar
Website Content Crawler
Try for free

No credit card required

View all Actors
Website Content Crawler

Website Content Crawler

apify/website-content-crawler
Try for free

No credit card required

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

User avatar

Cloudflare challenge failed

Closed

mihai opened this issue
2 months ago

Crawl failed due to cloudflare, what can be done?

User avatar

Hello, and thank you for your interest in the actor!

I see that you already used the stealthy firefox crawler type and have a residential proxy enabled. The only recommendation I can give you is to change the proxy group to something in Europe, instead of the US, in the actor input settings.

Let us know if that helps!

Developer
Maintained by Apify
Actor metrics
  • 2k monthly users
  • 99.7% runs succeeded
  • 3.0 days response time
  • Created in Mar 2023
  • Modified 3 days ago