No credit card required
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
Do you want to learn more about this Actor?
Get a demoBasic Auth in Header not working
Closed
Hello, the crawler needs to access a website behind basic auth. Unfortunately, it does not work, I receive a 401 error. Using the same header and URL in a simple python script it works without issues. I saw that there is a similar issue already, but I wanted to be able to provide the run id to help debug this.
Hello and thank you for your interest in this Actor - and the reproduction scenario.
We've found out that there is indeed a problem with the way we're processing the custom headers. Currently, we're working on a fix and will let you know once the patch is out and ready (most likely sometime next week).
Thanks again!
Hello - just letting you know that we've just released a new version of Website Content Crawler (0.3.37) where this is fixed - the Actor now correctly processes the passed HTTP headers, methods, and payloads.
I'll close this issue now, but feel free to ping us in case of any other issues or questions.
Cheers!
Apify- 2.8k monthly users
- 434 stars
- 99.9% runs succeeded
- 2.9 days response time
- Created in Mar 2023
- Modified 3 days ago
