1 day trial then $45.00/month - No credit card required now
Reddit Scraper
1 day trial then $45.00/month - No credit card required now
Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.
I am only able to get up to 600 or so posts
Closed
Hi there,
I would like to run this for 4 subreddits and get data from Nov 2022 until now. Is this possible using this?
I have only been able to get 600 or so posts so far which go back about a week. Any help would be very much apreciated
resilient_ladybug
Willing to pay for actor of it can provide all the data i require
It should be possible. Can you share the run ID so I can check why only 600 posts were returned?
resilient_ladybug
dov1BJUXQRzBmrrrs
Thanks
Using the URL to sort posts by new worked for me. You can try with this as a start URL: https://www.reddit.com/r/webdev/new
resilient_ladybug
Thanks, I tried this. Now i'm getting 945 posts but still only back to 06-2023. Any idea why it wont let me get data past this? Is my input for the actor wrong? Were you able to get posts back to Nov 2022? The one I just run was mAQZZVhAVJ2T1PU1a
thanks
I limited the items to 1000 so my scraper stopped when it reached that number of results but it seems to me like it would continue. The only apparent reason is that I selected the US as the country in my proxy configurations.
resilient_ladybug
Hmm, i set mine to 10k. So wonder why this would stop before 1000 even. I'll check the proxy configurations again and see. Would yours scrape past 1000 if you set it?
Never mind, I was scraping comments also, that is why my number is greater. Seems like your run went to the last page that Reddit provides using the web interface.
resilient_ladybug
Right, okay.
Is there a way past that? Surely more previous posts than last month can be seen on their web pages? The posts go back to last month. If i scroll back I can see posts from before this.
resilient_ladybug
Could it be due to the API limit? I coded a scraper that would not allow after a similar point due to limits
It can be, Reddit is changing a lot of its internal workings. I am trying to find a way to pass those limits if required
resilient_ladybug
Thanks, let me know if you manage to fix it. I will purchase actor then. This is ideally exactly what I need so appreciate you looking to fix it
resilient_ladybug
Any luck?
tasteful_island
I have the same issue with my trial. Can only get results to up to 2200 and in the last 2-3 weeks only. Would be keen if I can get the data I am looking for!
resilient_ladybug
It might be that this is where the webpage stop refreshing. This is the issue I have found so far
exciting_jacktree
I have tried kinds of methods to break through the limited number 1000 of posts, but they didn't work. I start toward this API and hope it can fix this problem.
Closing the ticket since the scrape is able to get all data that is available without the Reddit API.
exciting_jacktree
Please more details for closing the ticket. How did it?
I will have to create a different scraper to get all the data and it will probably require authenticated login with reddit
exciting_jacktree
OK, I can't wait to try the scraper that does not limit the number of posts.
Gustavo Rudiger- 286 monthly users
- 99.2% runs succeeded
- 0.5 days response time
- Created in Feb 2022
- Modified 4 days ago
