Reddit Scraper avatar

Reddit Scraper

Try for free

1 day trial then $45.00/month - No credit card required now

Go to Store
Reddit Scraper

Reddit Scraper

trudax/reddit-scraper
Try for free

1 day trial then $45.00/month - No credit card required now

Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.

ND

Can't seem to be able to scrape comments, only posts

Open

andreea.elisabeta.pomirleanu opened this issue
22 days ago

Hi Gustavo, awesome tool you have developed! I am trying to scrape a specific subreddit for some specific keywords.

Specifically, I am using a URL like this: https://www.reddit.com/r/askscience/search/?q=radioactive&cId=4d6df564-c307-413f-a048-d9d7cdf25365&iId=044b33fa-000b-48c2-b532-363a94b24888

However, the results I'm getting back only include posts that mention the keyword in their body and the comments of these post. But on Reddit itself, if i do a similar search I can switch to Comments at the top and it will show me the comments that contain the keyword regardless if the keyword is mentioned in the post body.

Am I doing something wrong? How can I make sure I am getting the comments as well, not just the posts?

Let me know if there's additional information you need from me, thank you so much!

PS(I'm including the actual actor run so you can see what I mean)

trudax avatar

Looks like a bug on the actor. I am working on a fix

trudax avatar

You might be reaching the limit with only posts. You can try to use one run to get only posts and another to get only comments.

ND

andreea.elisabeta.pomirleanu

21 days ago

Hi Gustavo, thanks so much for looking into this. I tried running a scrape just for comments, but it doesnt seem to work properly or maybe I'm not setting it up properly. From the main screen I activate Skip user posts and Skip community. But when I check what the run actually did I find these parameters in the json input: "searchComments": true, "searchCommunities": false, "searchPosts": true, "searchUsers": false,

As you can see, search Posts is still set to true. And when I check the results I still have results that do not include the keyword. I understand that the URL i'm giving it it's a bit more special perhaps since it includes both a subreddit and a search query parameter, since what i want is to search for that keyword in that specific subreddit. Is it not possible to search in this way and get only comments?

For example, if you want to search on r/news the term "Shakira" the comments you see on this link they all contain the keyword: https://www.reddit.com/r/news/search/?q=shakira&type=comments&cId=704ffb7c-0ffe-4a09-9c00-63fdf5f3f6a1&iId=2816285f-f7b6-4a14-91ed-203a5259e0e2 Is there a way for the scraper to achieve this with the scraper?

Thank you so much for your time!

trudax avatar

Hi Andreea,

Sorry for the delayed response. "Skip user posts" and "Skip community" only prevent the actor from getting those while doing a broad search. What you need to check is the Search settings. You will find the toggle to disable search for posts, users, and communities there.

Developer
Maintained by Community

Actor Metrics

  • 317 monthly users

  • 60 stars

  • >99% runs succeeded

  • 12 hours response time

  • Created in Feb 2022

  • Modified 21 days ago

Categories