Usage cost inconcistency

Hi, thanks for opening this issue!

This is the expected behavior, as in both cases, the actual work is pretty much the same. Let me explain:

Basically, this just has to do with the pagination of the videos, which come in batches of around 20-30 at once. As an example, when opening a new profile page, we get like 20 videos. To get 1 or 4 videos will cost the same, because both counts require the same amount of web requests.

In your case then, if you want to scrape thousands of profiles, feel free to set the number of videos to something like 15 and the cost will still stay mostly the same. Just a note, pinned posts count towards the initial videos batch, so consider turning excludePinnedPosts off to not go over the batch limit and to not enqueue the next one request.

I hope this helps, thanks!

Hey Lukas! Thanks for your reply.

Sorry for the follow up question but I just want to confirm for sure before I go ahead and overspend. You're telling me that if I set the amount of videos to scrape per creator to 15, I'll pay the same per creator than if I set this to 5?

TDLR: Almost yes

Checking it out further, it turns out this uses more requests on profiles with fewer videos than the desired amount, in this case 15. The scraper basically has to make sure that there are no more videos possible to scrape, even though I have an idea of how this can be optimized.

This means that most profiles will get scraped for the same price, but profiles with only few videos will use additional requests, and hence cost more. Also, the stuff that I said previously is still relevant. You still have to get under one batch, which 15 should be good enough for 95% of the time, but you can easily experiment with that on your own.

I will reopen this issue and keep you updated here with the optimization, thanks!

By the way, we also have this actor https://console.apify.com/actors/OtzYfK1ndEGdwWFKQ/input, which has a predictable pricing model based on the number of results and it would be cheaper in the second case (but not the first one)

Hi again, I'm happy to announce that we've updated the scraper with the fixed pagination!

It will now correctly identify profiles with only a few videos early and not waste additional requests on them.

Try it out and let me know how it works now, thanks and happy scraping!