Website Content Crawler
Pricing
Pay per usage
Website Content Crawler
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.
3.7 (41)
Pricing
Pay per usage
1499
Total users
58K
Monthly users
8.1K
Runs succeeded
>99%
Issues response
7.6 days
Last modified
an hour ago
Getting 403 from public page
Open
`Recently when run this for source it was working without any issue. even with resedential proxy the crawler is being blocked.
Hello, and thank you for your interest in this Actor!
You're getting blocked by Cloudflare Bot Management — their current settings appear to be quite strict, which is why WCC isn't able to get through anymore, even with residential proxies. Unfortunately, there’s not much that can be done directly in this case with your current setup.
I’d recommend trying our new Camoufox Scraper, which is specifically designed to handle these types of challenges. You can check out my example run here - it seems to have bypassed the bot filter and scraped the full content of the target page. Let me know if you need help setting it up.
I'll close this issue now, but feel free to ask additional questions if you have any. Cheers!
formidable_quagmire
Yes we will definitely need help to setup couple of things in new scraper. we use lot of features of WCC such as css remover, file downloaded. will this still work ? markdown. if yes i don;t seem to find documentation around that if you can share that that will be great.
formidable_quagmire
could you build "Camoufox Scraper" inside the WCC ? or atleast similar input and output parity with WCC ?
Hello again - sorry for the delay in my response.
We're currently considering adding Camoufox as one of the Crawler types for Website Content Crawler (and perhaps, in the long run, the default one). We'll keep you updated once we have a build including Camoufox ready for testing.
Cheers!
formidable_quagmire
That will definitely help us.
