Apify's proxy hasn't been good lately. Select "no proxy" will get you better results in this case.

I've run 1 more with "No Proxy" setup as well as with Proxy options, unfortunately the same thing happens

I believe my old ones were run with "No Proxy" as well as that seems to be the default option

https://console.apify.com/actors/runs/CfQS5DKPLviiugcok https://console.apify.com/actors/runs/kygQD2OlJ8OgHnPfm

oh you used the product url directly. The url input is actually for product list urls, not product detail url. So if you use the product listing page that includes that product, you should be fine.

Sorry if this is too much, could you clarify?

Ex: https://www.costco.com/%e2%80%8belvie-wearable-hands-free-electric-breast-pump-kit-with-elvie-catch-and-3-in-1-carry-bag.product.4000092738.html

What am I supposed to use for that instead?

I think I got it, it is currently running so not sure what the result is going to be, but I need to specify the category URL instead of the actual URL, and it will try yo get all the products below that, right?

if so, I have questions:

1. How deep does it go since some categories have their sub categories and then products. Does it go over all the sub-categories to get to the actual product URL? - Just tested this, it does not go more than 1 level, so categories with sub-categories FAIL right away, so it will be super useful to make the tool run by specifying the exact URLs.
2. Can you maybe modify your tool to work with exact URLs? I have a tool that gets me all the product URLs, so that would be very useful, and you will get customers for that too?

sure I can modify it so it'll work with exact product detail urls. I didn't put in much effort for this particular one because I got 0 paid users for this costco scraper.

It is also struggling to run and freezes, could you please have a look: https://console.apify.com/actors/xnpapdVIHyown497f/runs/cZyC0kVKn1ll0igUJ https://console.apify.com/actors/xnpapdVIHyown497f/runs/iKSKt5PupB8IEYBLW

LOL, this is a great tool you got there. I wrote my own through SELENIUM, but super slow. Would love to use your tool if it were open-source.

Could you actually have that tool separately, I like this one too and would love to use it. Think I found a way to get all the categories from Costco, writing a tool for it now.

Apify's proxy has been pretty useless lately. That's the reason for all the 403s you were getting. Happy to build a new one for product detail page and keep this for product listing page.

Please do, and let me know. Would love to work with you on this. Just FYI, I have over 10k product URLs I need to go through, so would love to do that in 1 run without any FAILS since each URL is very important on its own.

And I'll be doing that repeatedly 2-4 or maybe once everyday

Also would love to talk to you over slack,discord or even an email if that does not break apify's policies.