Retrying Fetch

Yeah, might have to go to residential IP's. Zillows increasing their security and not messin around. Gonna check it out today. Was working on it the majority of yesterday. Trying to make it work with regular IPs, but might just have to bite the bullet.

Hmm, will us providing you IPs be of any use?

Can you add functionality for that? We can set that up ourselves.

Yeah looking into right now. I think you can buy residential IPs from Apify. But yeah hold off until I can check it out.

Okay. I've used IP royal in the past and never had issues.

Can you send over the csv from this run when you have a chance?

Yeah working on it

Still trying to get it, it looks like I can use regular IP's, I just have to slow it down. Looks like they rate limit pretty severely.

Done

Thanks, any updates on the IPs getting implemented?

So far I'm circumventing that with slowing down the requests, but am going to try and see if I can use the Apify IP's to speed things up.

Hey Adrian,

We don't really care so much about speed just that we actually get the data. A scrape taking 30 minutes is fine to us.

Ok cool, well then seems like it's good to go. You might have to increase the timeout of the actor so it doesn't timeout.

[image: Screenshot 2023-06-05 at 3.13.26 PM.png]

No results. Hmmmm. Zillow still IP blocked us. Is there a way to add external residential proxies into appify?

https://console.apify.com/actors/tOr3TP5NDY7gpbiJh/runs/UZ9KWg3o0OKhJqzmT#log

Dang. Let me check it out.

Check out the attached results for Fort Worth. Working on a solution.

Its missing a ton of stuff. I think the old code was fine (unless zillow changed their front end or something) We just need a way to integrate a proxy service. I cant seem to figure out the proxy thing. We may need you to add some code from this page to allow it. https://docs.apify.com/platform/proxy/datacenter-proxy/examples

Unfortunately it’s not quite that easy. Looks like they are still severely rate limiting residential IPs, so working on a way around it.

Oh sorry misunderstood that last email.

Yeah the proxy thing is indeed a pain, trying a couple other things to work around it.

Lets massively lower our requests then and just do it slower. We dont really care about it taking 15 minutes to scrape, just that it gets done. Does that make sense?

Yeah, I can try that also, looks like Apify restarts the server if it takes too long, but then I have to keep track of state, which is a little tricky.

That's why the latest one you did never finished ( https://docs.apify.com/platform/actors/development/state-persistence)

So both options are a pain lol. So yeah, trying to figure something out 🤘