My runs are suddenly being rate limited by API Dojo

Hello,

When I check your history, I see that you got rate-limited automatically by our rate-limiter. Here are the possible reasons:

• You may be executing too many runs in a very short period of time. You should only have 1 (or max 2) concurrent runs at the same time, above this is damaging our actor so it rate-limits automatically.
• You are fetching few items. When I check your runs, I see that you are fetching 10 items for 5 keywords. You need to fetch 10 items per keyword, otherwise we make a negative margin from your runs. The cost of those runs become greater than the profit.

Do you think you can follow these rules? You need to increase maxItems to 50 (if you continue to batch 5 keywords each) and also lower your concurrent runs to 1 or 2.

Let us know if you can follow these and we will be happy to lift the rate-limit.

Cheers

1. Apify rules state I can have 100+ concurrent runs, this is because im a premium member paying $500+ a month. I've also been using many concurrent runs for months now. You can't enforce arbitrary changes like this without telling customers. I will contact Apify platform to tell them about this because it goes against customer expectations.

2. This rule I can follow but you need to do a better job of clarifying that on your scraper page so all others users know too. Nobody can see this limitation today... Also this is not a "rate limit", that would be API design and you need to clarify that in your docs.

If you look at my run and purchase history with your scraper you can see that i've been using your scraper heavily and my 10 items for 5 keywords is by design. I need to do this for my business need. I will change this to fetch 10 items for 1 keyword instead of 10 for 5 keywords.

I will not lower my concurrent runs because I get 128+ concurrent runs as a premium customer. If you don't lift the rate limit I will contact Apify platform team.

More than 2 runs can't be 'damaging' to your actor, you obviously need to design your system to support these needs if you are offering a service that wants to be taken seriously

Hello,

These limitations are not from Apify but from us as the actor developer. They were there in the first place but somehow your runs didn't get noticed.

Unfortunately technical complexity of scraping Youtube limits us here. Why would we limit you if that's not something mandatory for us? We'd want you to execute more runs and profit more if that's not the case.

Also please don't forget that this is not an API, it is a scraper. So you shouldn't expect the same things that you get from an API. There are other technical challenges here.

Currently you are doing burst of runs. I think instead of doing bursts of runs, you can spread your runs into the day and comply with our terms. Let me know if you can lower your requests and we will be happy to remove this.

Cheers

How about this, I will make sure we are querying 10 items for every keyword, but can you let me have at least 32 concurrent runs? I will no longer do 128+ concurrent as before but I need at least 32+.

Tell me soon, if i only get 1 concurrent run then I may need to move to another vendor, i need these results quickly for my application usecase.

Hello again,

Can you please get in touch via Discord or email to talk further about this? You can find us on the Apify Discord server as apidojo or you can email directly to apidojo10@gmail.com

Cheers

I've DMed you on Discord just now.

Please reply soon, i've messaged you already.