Feature Suggestion: Automated Retry Mechanism for Failed Downloads

Hey there!

Thank you very much for reaching out, and letting us know about your inquiry. The actor already has a retry mechanism in place where it is following the Request Queue's default retry. However, we can enable maxConcurrency and maxRetries as an additional property to the input. Do you think that can help you out?

Regarding manually analyzing, and retrying it, do you have a Run ID that we can check? We are also checking the fail reasons.

Best

Thanks for the quick response (just saw). Yes, here's one of the Run IDs (this one has 3014 results and 365 failed requests): Rp3tARCAo0vOYWLFu

If I compile a list of the 365 failed requests' links and try again, 90% will work, and then I repeat this process on the 10% failed until there's 0-1 fails left.

I'm not fully sure how maxConcurrency and maxRetries ties into this. But if your understanding determines that it helps, then I'm all for it! :)

Currently, your Scraper is my #1 favorite for my use-case. If this stays the case, you can expect 6-figures+ worth of usage in the future

I think the issue stems from the Database Proxies Apify provides. However, if simply re-trying on a separate run solves the issue somehow.. maybe there's a different way to retry internally (via Actor code)?

I'm unfamiliar with Apify's proxy systems (Idk if they run a separate/new batch for each Run, or how it works exactly)

Hey again,

Thank you very much for the detailed explanation. We investigated the run and it seems like the issue that you are facing is completely related to the Apify proxies as you mentioned. The actor already has the IP-rotation and retry mechanisms in place to increase its success rate.

There are a couple of possible options to increase the success rate. We can also support with some new input properties:

1. Sometimes, advanced protection systems like YT can identify an actor even if it is behind one or multiple proxies. Therefore, splitting the run into smaller chunks can help you out here.
2. We added a new field called Max Concurrency. This option allows the actor to limit its concurrency. Slowing the actor, or defining an upper limit, can help the success rate to increase and have a more stabilized output retrieval. By default, this field is empty and lets the actor arrange itself.
3. We added a new field called Maximum Request Retries. This option allows how many times a video should be retried. By increasing this number, you can achieve better results but please keep in mind that you might be caught already by YouTube (point 1). By default, this field is empty and lets the actor arrange itself.

I hope this helps. Please let us know if you need anything else. Best