Reddit Scraper avatar
Reddit Scraper
Try for free

1 day trial then $45.00/month - No credit card required now

View all Actors
Reddit Scraper

Reddit Scraper

trudax/reddit-scraper
Try for free

1 day trial then $45.00/month - No credit card required now

Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.

User avatar

I am only able to get up to 600 or so posts

Closed

resilient_ladybug opened this issue
10 months ago

Hi there,

I would like to run this for 4 subreddits and get data from Nov 2022 until now. Is this possible using this?

I have only been able to get 600 or so posts so far which go back about a week. Any help would be very much apreciated

User avatar

resilient_ladybug

10 months ago

Willing to pay for actor of it can provide all the data i require

User avatar

It should be possible. Can you share the run ID so I can check why only 600 posts were returned?

User avatar

resilient_ladybug

10 months ago

dov1BJUXQRzBmrrrs

Thanks

User avatar

Using the URL to sort posts by new worked for me. You can try with this as a start URL: https://www.reddit.com/r/webdev/new

User avatar

resilient_ladybug

10 months ago

Thanks, I tried this. Now i'm getting 945 posts but still only back to 06-2023. Any idea why it wont let me get data past this? Is my input for the actor wrong? Were you able to get posts back to Nov 2022? The one I just run was mAQZZVhAVJ2T1PU1a

thanks

User avatar

I limited the items to 1000 so my scraper stopped when it reached that number of results but it seems to me like it would continue. The only apparent reason is that I selected the US as the country in my proxy configurations.

User avatar

resilient_ladybug

10 months ago

Hmm, i set mine to 10k. So wonder why this would stop before 1000 even. I'll check the proxy configurations again and see. Would yours scrape past 1000 if you set it?

User avatar

Never mind, I was scraping comments also, that is why my number is greater. Seems like your run went to the last page that Reddit provides using the web interface.

User avatar

resilient_ladybug

10 months ago

Right, okay.

Is there a way past that? Surely more previous posts than last month can be seen on their web pages? The posts go back to last month. If i scroll back I can see posts from before this.

User avatar

resilient_ladybug

10 months ago

Could it be due to the API limit? I coded a scraper that would not allow after a similar point due to limits

User avatar

It can be, Reddit is changing a lot of its internal workings. I am trying to find a way to pass those limits if required

User avatar

resilient_ladybug

10 months ago

Thanks, let me know if you manage to fix it. I will purchase actor then. This is ideally exactly what I need so appreciate you looking to fix it

User avatar

resilient_ladybug

10 months ago

Any luck?

User avatar

tasteful_island

10 months ago

I have the same issue with my trial. Can only get results to up to 2200 and in the last 2-3 weeks only. Would be keen if I can get the data I am looking for!

User avatar

resilient_ladybug

10 months ago

It might be that this is where the webpage stop refreshing. This is the issue I have found so far

User avatar

exciting_jacktree

10 months ago

I have tried kinds of methods to break through the limited number 1000 of posts, but they didn't work. I start toward this API and hope it can fix this problem.

User avatar

Closing the ticket since the scrape is able to get all data that is available without the Reddit API.

User avatar

exciting_jacktree

6 months ago

Please more details for closing the ticket. How did it?

User avatar

I will have to create a different scraper to get all the data and it will probably require authenticated login with reddit

User avatar

exciting_jacktree

6 months ago

OK, I can't wait to try the scraper that does not limit the number of posts.

Developer
Maintained by Community
Actor metrics
  • 281 monthly users
  • 100.0% runs succeeded
  • 12 hours response time
  • Created in Feb 2022
  • Modified 7 days ago
Categories