Reddit Scraper avatar
Reddit Scraper
Try for free

1 day trial then $45.00/month - No credit card required now

View all Actors
Reddit Scraper

Reddit Scraper

trudax/reddit-scraper
Try for free

1 day trial then $45.00/month - No credit card required now

Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.

User avatar

Crawling subreddit posts stops after reaching ~500 posts

Closed

grown_axe opened this issue
3 months ago

I'm trying to scrape all the posts from a given subreddit in the past year, I've set the total posts limit to a very high number (1 million). Unfortunately the scraper stops after finishing ~500 posts, and it didn't return any visible error

User avatar

By default, the root subreddit URL loads the post in the Hot category. Try scrapping https://www.reddit.com/r/worldnews/new/ to get more posts.

User avatar

I was able to get 845 results going until the the last page available on the worldnews subreddit. You should get something similar.

User avatar

grown_axe

3 months ago

Thank you for the quick response! Is there a way to scrape all the posts historically on this subreddit?

User avatar

Reddit limits the maximum posts that you can get on their website to 1000. I don't know any other way to surpass this at the moment besides to constantly scrape the subreddit and store the history yourself.

Developer
Maintained by Community
Actor metrics
  • 281 monthly users
  • 100.0% runs succeeded
  • 12 hours response time
  • Created in Feb 2022
  • Modified 7 days ago
Categories