![Website Content Crawler avatar](https://images.apifyusercontent.com/1VrdawICnxIwM4X5JzRJHPBmLx0OpmiNxtHGGLmxdu8/rs:fill:92:92/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
No credit card required
![Website Content Crawler](https://images.apifyusercontent.com/1VrdawICnxIwM4X5JzRJHPBmLx0OpmiNxtHGGLmxdu8/rs:fill:92:92/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMuYW1hem9uYXdzLmNvbS9hWUcwbDlzN2RiQjdqM2diUy9QZlRvRU5rSlp4YWh6UER1My1DbGVhblNob3RfMjAyMy0wMy0yOF9hdF8xMC40MC4yMF8yeC5wbmc.webp)
Website Content Crawler
No credit card required
Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗LangChain, LlamaIndex, and the wider LLM ecosystem.
KV Store isn't downloading file attachments correctly
Closed
We recently initiated a crawl and noticed that some of the file attachments aren't being saved to the KV Store correctly.
For example, the file encountered at the URL below is 50 MB but the corresponding KV store entry is reporting 1.5 MB. Do we need to increase our download timeout in order to download the entire file in time?
- https://barbercosmo.ca.gov/schools/healthsafety_textbook_ko.pdf
- dc9f565-healthsafety_textbook_ko.pdf
Hello and thank you for your interest in this Actor!
I don't think there is anything you can do with this right now - this is most likely caused by the way we handle file downloads in this Actor. I started looking into this and already have a promising proof-of-concept for the patch, but before releasing it, I need to test this thoroughly.
I'll keep you posted with updates!
Thank you again for the report and sorry for the momentary inconvenience. Cheers!
This is fixed in the latest build of this Actor (0.3.33
). Give it a try and let us know how it went.
Thank you again for your patience!
- 2.8k monthly users
- 317 stars
- 100.0% runs succeeded
- 4 days response time
- Created in Mar 2023
- Modified 1 day ago