1 day trial then $45.00/month - No credit card required now
Reddit Scraper
1 day trial then $45.00/month - No credit card required now
Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.
Crawling subreddit posts stops after reaching ~500 posts
Closed
I'm trying to scrape all the posts from a given subreddit in the past year, I've set the total posts limit to a very high number (1 million). Unfortunately the scraper stops after finishing ~500 posts, and it didn't return any visible error
By default, the root subreddit URL loads the post in the Hot category. Try scrapping https://www.reddit.com/r/worldnews/new/ to get more posts.
I was able to get 845 results going until the the last page available on the worldnews subreddit. You should get something similar.
grown_axe
Thank you for the quick response! Is there a way to scrape all the posts historically on this subreddit?
Reddit limits the maximum posts that you can get on their website to 1000. I don't know any other way to surpass this at the moment besides to constantly scrape the subreddit and store the history yourself.
Gustavo Rudiger- 286 monthly users
- 99.1% runs succeeded
- 0.5 days response time
- Created in Feb 2022
- Modified 4 days ago
