Hello Alex, and thank you for your interest in this Actor!

Indeed, Website Content Crawler cannot do this. This is because WCC (like any other web crawler) only follows the links on web pages, discovering one new web page at a time - and stops only once all the links lead to already discovered pages.

The only idea I have right now is sitemap discovery - by parsing the website's sitemap, you should get a list of all the URLs on the website, and therefore get the count of all the pages on the website. However, Website Content Crawler doesn't do this, as we didn't see the need for it yet.

Could you share your ideas about this feature - most importantly, what's your use case for this? Thanks again!

Hello Jindrich Bär, thanks for the quick response.

I imagined it would be like this. Your question about the use case is in fact the most important and I will make sure that the next time I use this space I make it clear why I have this question.

Use case: Estimate the cost and duration of scrapping considering a given infrastructure resource.

Maybe this should be my question, is there any way I can predict the cost and execution time of scrapping considering 1GB of CPU and 4GB of memory?

I know there are more variables that can impact this forecast, but as I am going to offer the service to the end public, I wanted to have a certain predictability to make more assertive pricing.

Alright, now it makes perfect sense!

The cost and scraping time depend on the following three variables:

• The website size (more pages = longer and more pricy crawl)
  • this is quite hard to estimate, see above
• The amount of RAM available (more RAM = more space for concurrent processes, so faster processing, but you're paying more $ per hour)
• The crawler type (Firefox and Chrome take some time to load the page, Cheerio loads the pages almost instantly, but doesn't execute client-side JS)
  • you want to go for Cheerio whenever it's possible... but it's not always possible (e.g. on pages with dynamic content loading).

On our testing account, the average price per result is $0.011 (or 12 seconds) and the median is $0.006 (or 3 seconds). The majority of the runs for these statistics are Firefox with 8 GB of RAM - and they are from reasonably long crawls (the Actor takes a few seconds to spin up, so the efficiency of short runs will be worse).

You can always check out the "Monitoring" tab (https://console.apify.com/actors/aYG0l9s7dbB7j3gbS/monitoring), which shows you statistics about your runs - In the Stats per run section, you can find the Duration/Cost per result chart, which shows you the average price for crawling one website (based on your last 200 runs).

Feel free to ask any additional questions. Thanks!