Reddit Scraper avatar
Reddit Scraper
Try for free

1 day trial then $45.00/month - No credit card required now

View all Actors
Reddit Scraper

Reddit Scraper

trudax/reddit-scraper
Try for free

1 day trial then $45.00/month - No credit card required now

Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.

IH

Scraping incomplete

Closed

Ismael_Hayden opened this issue
6 months ago

I'm trying to scrape the following url:

https://www.reddit.com/r/autism/comments/164ggvt/i_havent_told_my_daughter_that_she_has_autism/

It's not pulling all of the data, it tops out at 257 or approx 540 comments

IH

Ismael_Hayden

6 months ago

Also, I tried it three times to no avail.

trudax avatar

Can you share the Run ID with me?

trudax avatar

I have found the issue and I am working on a solution.

trudax avatar

Can you try it again?

IH

Ismael_Hayden

6 months ago

Hi Gustavo, sorry for the delay and thanks for the speed of your response, the issue now is that it returned 1001 comments, I thinks it's separating them but it is still incomplete. Also, it's saying that I have run out of credits, it's a big task and trying it four times I think has used a lot of credits. I'm looking for an excel spreadsheet, I'll try send you the run ids

IH

Ismael_Hayden

6 months ago

No. 1 9ecrcz45rta8ut8Ix No. 2 o8BvmrPQUySKjITLu No. 3 qny0H1HtL0EPQhAyI No. 4 Qp1PAe7Ehe5ikXTSS

IH

Ismael_Hayden

6 months ago

Thanks :)

IH

Ismael_Hayden

6 months ago
trudax avatar

You are limiting the maximum number of results to 1000 using the maxItems property on your input. You should increase that so you will be able to get more results.

IH

Ismael_Hayden

5 months ago

But there are only 540 comments. Can you try to run it and see if it returns the correct results on your end? Apologies, I'm not sure how troubleshooting works, I'm new to this and your's is the only tool I've used. Thank you so much by the way, it's been great!

IH

Ismael_Hayden

5 months ago

Or is there anyway that you could attach the excel file output in this chat? It's for a piece of research one of 11 (the others were successful).

IH

Ismael_Hayden

5 months ago

This file of the most recent attempt the data is not in order and has 5000 rows instead of 540, looking at the cells I can see that rows are being repeated

trudax avatar

My run got 530 results

trudax avatar

Can you share the run ID with 5000 rows?

IH

Ismael_Hayden

5 months ago

xguPeBuAm0c9T7VZR

Here you go

IH

Ismael_Hayden

5 months ago

Could you send me your excel spreadsheet for the 530, though the page says 540? Thanks a mil

trudax avatar

Ran two times with the same number of resuts.

trudax avatar

Can you confirm that you received the spreadsheet? The Apify page seems to have a bug and it is not displaying the attached file to me.

trudax avatar

From the logs of this run (xguPeBuAm0c9T7VZR) seems like the actor is storing results so fast reached Apify's rate limit when tried to store more than 200 results per second. That has never happened to me which is very weird. This can be a bug on Apify side, if it is something related to the actor I should be able to replicate the issue here.

IH

Ismael_Hayden

5 months ago

Hi, no I didn't receive it, I noticed a similar bug when I tried to send to you.

my email address is: ismaelihayden@gmail.com

Thank you so much!

IH

Ismael_Hayden

5 months ago

Hello, I wonder has there been any movement on this? I didn't receive an email in case you sent it.

Thanks a mil!

Ismael

trudax avatar

I have sent it with the subject: Reddit actor results Can you check your spam folder?

trudax avatar

I have found the issue with your run and created a fix for it. But it still returns 530 results. Regarding the order, since the actor performs parallel requests to increase performance, the order is not guaranteed.

IH

Ismael_Hayden

5 months ago

Ok that's understood, so long as they're in order etc, I can go through and see. So it will work if I run it or can you send me an excel file? Best wishes, Ismael

trudax avatar

You can run it now

Developer
Maintained by Community
Actor metrics
  • 274 monthly users
  • 23 stars
  • 100.0% runs succeeded
  • 23 hours response time
  • Created in Feb 2022
  • Modified 16 days ago
Categories