Reddit Scraper
Pricing
$45.00/month + usage
Reddit Scraper
Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.
3.9 (2)
Pricing
$45.00/month + usage
120
Total users
7.1K
Monthly users
463
Runs succeeded
>99%
Issues response
2.4 days
Last modified
15 days ago
Crawling subreddit posts stops after reaching ~500 posts
Closed
I'm trying to scrape all the posts from a given subreddit in the past year, I've set the total posts limit to a very high number (1 million). Unfortunately the scraper stops after finishing ~500 posts, and it didn't return any visible error
By default, the root subreddit URL loads the post in the Hot category. Try scrapping https://www.reddit.com/r/worldnews/new/ to get more posts.
I was able to get 845 results going until the the last page available on the worldnews subreddit. You should get something similar.
grown_axe
Thank you for the quick response! Is there a way to scrape all the posts historically on this subreddit?
Reddit limits the maximum posts that you can get on their website to 1000. I don't know any other way to surpass this at the moment besides to constantly scrape the subreddit and store the history yourself.
