More exactly it's been 5 days that I don't get the results.

It's an issue with the apify proxy, most likely it has been blacklisted, so you'll need to try with a different country code proxy. Recently the french ones seem to have gotten blocked as well. I tried with DE for the moment and it worked.

I think i'll add in a possibility to specify various proxy countries and the option to retry with new proxy countries automatically if one fails

Also you might want to use the key "&per_page=50" at the end of your URL, so that you get 50 requests at once, and won't have to check other pages and thereby reduce the risk of getting flagged by cloudflare/upwork

Thx Artur! Is there any other way to use some kind of residential proxies or something similar to make sure I don't get into this problem again?

The scraper does use residential proxies. If you're using automation software you could check for failure and then try querying again with another country code.

Thanks for the update. I tried changing the country, but after only a few executions, I started getting 403 errors again: https://monosnap.com/file/QXcYGl7c5jdkY3BVBFalqi9uJlitwy

It looks like Upwork is detecting the scraping pattern and blocking the IPs fairly quickly. Does this mean they are actively identifying and banning the proxies? If so, what can be done to avoid this?

Also, is there a list of available countries I can use for proxy rotation? That would help in managing the retries more effectively.

Thanks!

I also implemented what you recommended, so I'm changing the country and separating each url to be sent with another country. There's still no consistency: https://monosnap.com/file/hwCKe7UNYGNDczNtAvB7yT5YdhnD1J

Heya Artur, when could we expect this to be fixed? Cheers!

Looking into it right now, a quick fix is to try with another proxy. Sadly apify doesn't provide a list of what countries are available. In the docs they've only provided the two letter country code list: https://laendercode.net/en/2-letter-list.html

legend. Good luck! "AI" worked btw.

Hey Artur,

Any news on this? Any progress?

Best,

Hey.

I added an User Cookies field. You should match it with the proxy country code. So if you're located in Sweden for example, you should add SE for the proxy country.

To get the cookie:

-Open Upwork & Navigate to Job Search Open Brave/Chrome/Chromium and go to: https://www.upwork.com/nx/search/jobs/?nbs=1&q=stripe&page=4 Make sure you're logged in to Upwork. -Open Developer Tools Right-click anywhere on the page and select "Inspect" OR Press F12 or Ctrl + Shift + I (Windows/Linux) Press Cmd + Option + I (Mac)

• Capture the Network Request In Developer Tools, go to the "Network" tab. In the search bar, filter by "jobs" Click on the request that matches: jobs/?nbs=1&q=stripe&page=4 -Copy Cookies Click on the "Headers" tab. Scroll down until you see "Cookie" under Request Headers. Select and copy everything after Cookie:.
• Paste Cookies in Your Scraper Use the copied cookies for the input

Just tried with the cookie, most of the searches are still blocked, but is successfully retrieving some of the pages! Thanks for keeping up on this, Artur.