Website Content Crawler avatar
Website Content Crawler
Try for free

No credit card required

View all Actors
Website Content Crawler

Website Content Crawler

apify/website-content-crawler
Try for free

No credit card required

Automatically crawl and extract text content from websites with documentation, knowledge bases, help centers, or blogs. This Actor is designed to provide data to feed, fine-tune, or train large language models such as ChatGPT or LLaMA.

User avatar

KV Store isn't downloading file attachments correctly

Open

embrace_ai opened this issue
15 days ago

We recently initiated a crawl and noticed that some of the file attachments aren't being saved to the KV Store correctly.

For example, the file encountered at the URL below is 50 MB but the corresponding KV store entry is reporting 1.5 MB. Do we need to increase our download timeout in order to download the entire file in time?

User avatar

Hello and thank you for your interest in this Actor!

I don't think there is anything you can do with this right now - this is most likely caused by the way we handle file downloads in this Actor. I started looking into this and already have a promising proof-of-concept for the patch, but before releasing it, I need to test this thoroughly.

I'll keep you posted with updates!

Thank you again for the report and sorry for the momentary inconvenience. Cheers!

Developer
Maintained by Apify
Actor metrics
  • 2.1k monthly users
  • 99.6% runs succeeded
  • 3.2 days response time
  • Created in Mar 2023
  • Modified 7 days ago