1 day trial then $45.00/month - No credit card required now

Reddit Scraper

trudax/reddit-scraper

1 day trial then $45.00/month - No credit card required now

Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.

Back to issues Create new issue

won't collect beyond a certain limit

Closed

lemon_normalcy opened this issue

I've tried two different runs now of trying to collect a significant portion of a subreddit's activity (2+ years), but the scraper stops after about 50,000 lines of data

runs: https://li2vbaesoevb.runs.apify.net and https://cargcpdn5cbn.runs.apify.net

one further detail or question - it'd be great to pick up where a previous run stopped. is there some way to use the previous pagination marker to start collecting where it left off?

Gustavo Rudiger (trudax)

can you share the run ID?

lemon_normalcy

I think they're both in the original post aren't they? Those 2 links?

Gustavo Rudiger (trudax)

I don't think so, there is no usefull information when I click on those links. Ff you go to the run you should be able to see a share button on the right upper side that will provide you with the correct link.

lemon_normalcy

Sorry about that. Here you go:

https://console.apify.com/view/runs/dP9XSbh7cxCArL5c8 (first run)

https://console.apify.com/view/runs/GkjTkfGQ9cNnN9JpF (second run)

lemon_normalcy

what do you think?

Gustavo Rudiger (trudax)

I will try to replicate the run here but at first glance, I don't see anything wrong besides some blocked request, but that is expected.

lemon_normalcy

okay - what about picking up a new run where the old one left off though? currently there isn't an option to do that.

lemon_normalcy

what do you think?

Gustavo Rudiger (trudax)

I have got the same results, seems like it is returning everything that it can.

lemon_normalcy

right - but my question is why we can't start the scraping at somewhere other than the beginning of the subreddit?

Gustavo Rudiger (trudax)

You can, you just need to know the last URL used for paginating.

lemon_normalcy

Okay. How do I implement that within this actor then?

Gustavo Rudiger (trudax)

I will add a change to log the last page so you can copy from the logs and use it.

lemon_normalcy

okay - where do I paste in the last URL used?

lemon_normalcy

hello?

Gustavo Rudiger (trudax)

the last usedr url is on the logs

Add comment

Developer

Gustavo Rudiger

Actor metrics

365 monthly users
37 stars
99.9% runs succeeded
1.2 days response time
Created in Feb 2022
Modified 9 days ago

Categories

Social media

Reddit Scraper Lite

trudax/reddit-scraper-lite

Pay Per Result, unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.

Gustavo Rudiger

3.4k

Reddit Scraper For Posts & Comments

creative_tablecloth/reddit-scraper-for-posts

Access Reddit data freely without authentication. Quickly extract detailed information from Reddit posts and comments, both efficiently and cost-effectively. (approx $0.015 for 1,000 results)

Jinny Kim

(New) Reddit Scraper Pro - Fast, Affordable, Supported

harshmaur/reddit-scraper-pro

Reddit Scraper Pro is a powerful, user-friendly tool for extracting data from Reddit without API limitations. Offers scraping of posts, users, comments, and communities, advanced search capabilities, and multiple export options. Perfect for brand monitoring, trend tracking, and competitor research.

Harsh Maur

Reddit API Scraper

comchat/reddit-api-scraper

Reddit Scraper is a powerful tool that allows you to extract data from Reddit such as posts by keyword. With Reddit Scraper, you can easily gather valuable information from Reddit without the need to log in. You can easily use this Reddit scraper as an alternative API.

Comchat

384

Ultimate Reddit Profile Scraper (Lite)

potatopeeler/reddit-account-scraper-lite

Pay per result. Seamlessly download full Reddit user accounts, capturing posts, images, activity, and historical data, including URLs and media comments. Export detailed insights to CSV, JSON, XML, EXCEL formats, or effortlessly import them into your email for comprehensive analysis and easy access.

Jamie Potato

Reddit Email Scraper

bhansalisoft/reddit-email-scraper

Reddit Email Scraper- Scrap Emails from Reddit specific profile using google search engine

bhansalisoft

Reddit Explorer 3.0

jupri/reddit

🤖 Scrape Reddit.com ⬛🟪🟦🟩🟨🟧🟥🟫

cat

TikTok Data Extractor

clockworks/free-tiktok-scraper

Extract data about videos, users, and channels based on hashtags or scrape full user profiles including posts, total likes, name, nickname, numbers of comments, shares, followers, following, and more.

Clockworks

14.1k

109

Facebook Posts Scraper

apify/facebook-posts-scraper

Extract data from hundreds of Facebook posts from one or multiple Facebook pages and profiles. Get post URL, post text, page or profile URL, timestamp, number of likes, shares, comments, and more. Download the data in JSON, CSV, and Excel and use it in apps, spreadsheets, and reports.

Apify

11.5k

115

Instagram Scraper

apify/instagram-scraper

Scrape and download Instagram posts, profiles, places, hashtags, photos, and comments. Get data from Instagram using one or more Instagram URLs or search queries. Export scraped data, run the scraper via API, schedule and monitor runs or integrate with other tools.

Apify

50k

282