Reddit Scraper avatar
Reddit Scraper
Try for free

1 day trial then $45.00/month - No credit card required now

View all Actors
Reddit Scraper

Reddit Scraper

trudax/reddit-scraper
Try for free

1 day trial then $45.00/month - No credit card required now

Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.

User avatar

No items scraped

Closed

bravura opened this issue
a year ago

I first tried the free-reddit-scraper and then this. Both of them return 0 items!

I am trying to find posts and comments with particular search terms.

Here is my config:

{ "debugMode": false, "maxComments": 0, "maxCommunitiesAndUsers": 2, "maxItems": 0, "maxLeaderBoardItems": 2, "maxPostCount": 0, "proxy": { "useApifyProxy": true }, "scrollTimeout": 40, "searchComments": true, "searchCommunities": false, "searchPosts": true, "searchUsers": false, "searches": [ "keyword1", "keyword2", "keyword3", "keyword4", "keyword5", ] }

(I want to search for any of the search keywords, not all.

Here is my log

2023-06-19T22:23:06.521Z ACTOR: Pulling Docker image from repository. 2023-06-19T22:23:06.676Z ACTOR: Creating Docker container. 2023-06-19T22:23:06.780Z ACTOR: Starting Docker container. 2023-06-19T22:23:08.850Z INFO System info {"apifyVersion":"3.0.2","apifyClientVersion":"2.7.1","osType":"Linux","nodeVersion":"v16.20.0"} 2023-06-19T22:23:09.616Z ACTOR: Actor run will metamorph 2023-06-19T22:23:09.619Z ACTOR: Sending Docker container SIGTERM signal. 2023-06-19T22:23:09.644Z ACTOR: Pulling Docker image from repository. 2023-06-19T22:23:09.747Z ACTOR: Creating Docker container. 2023-06-19T22:23:10.026Z ACTOR: Starting Docker container. 2023-06-19T22:23:11.254Z Starting X virtual framebuffer using: Xvfb :99 -ac -screen 0 1280x720x16 -nolisten tcp 2023-06-19T22:23:11.256Z Executing main command 2023-06-19T22:23:12.532Z INFO System info {"apifyVersion":"3.1.4","apifyClientVersion":"2.7.1","crawleeVersion":"3.3.1","osType":"Linux","nodeVersion":"v16.15.0"} 2023-06-19T22:23:12.612Z INFO Search param found. Performing search... 2023-06-19T22:23:13.285Z WARN PuppeteerCrawler: "handlePageFunction" has been renamed to "requestHandler", and will be removed in a future version. 2023-06-19T22:23:13.286Z The provided value will be used, but you should rename "handlePageFunction" to "requestHandler" in your crawler options. 2023-06-19T22:23:13.286Z WARN PuppeteerCrawler: "handleFailedRequestFunction" has been renamed to "failedRequestHandler", and will be removed in a future version. 2023-06-19T22:23:13.287Z The provided value will be used, but you should rename "handleFailedRequestFunction" to "failedRequestHandler" in your crawler options. 2023-06-19T22:23:13.669Z INFO Starting the crawl. 2023-06-19T22:23:13.771Z INFO PuppeteerCrawler: Starting the crawl 2023-06-19T22:23:22.567Z WARN Actor reached the max items limit. Crawler is going to halt and abort ongoing requests... 2023-06-19T22:23:22.568Z WARN Actor reached the max items limit. Crawler is going to halt and abort ongoing requests... 2023-06-19T22:23:29.718Z WARN Actor reached the max items limit. Crawler is going to halt and abort ongoing requests... 2023-06-19T22:23:31.515Z WARN Actor reached the max items limit. Crawler is going to halt and abort ongoing requests... 2023-06-19T22:23:31.590Z INFO PuppeteerCrawler: All requests from the queue have been processed, the crawler will shut down. 2023-06-19T22:23:31.838Z INFO PuppeteerCrawler: Crawl finished. Final request statistics: {"requestsFinished":4,"requestsFailed":0,"retryHistogram":[4],"requestAvgFailedDurationMillis":null,"requestAvgFinishedDurationMillis":8364,"requestsFinishedPerMinute":13,"requestsFailedPerMinute":0,"requestTotalDurationMillis":33455,"requestsTotal":4,"crawlerRuntimeMillis":18554} 2023-06-19T22:23:31.840Z INFO PuppeteerCrawler: Terminal status message: Finished! Total 4 requests: 4 succeeded, 0 failed. 2023-06-19T22:23:34.279Z INFO Crawl finished. 2023-06-19T22:23:34.280Z INFO [Status message]: Actor finished successfully (exit code 0)

User avatar

You need to provide a value greater than zero for maxItems. maxPostCount and maxComments.

User avatar

bravura

a year ago

I tried this (making the values non-zero), and enabled the trial, but get this log:

2023-06-20T07:44:16.248Z ACTOR: Pulling Docker image from repository. 2023-06-20T07:44:16.378Z ACTOR: Creating Docker container. 2023-06-20T07:44:16.456Z ACTOR: Starting Docker container. 2023-06-20T07:44:18.367Z INFO System info {"apifyVersion":"3.0.2","apifyClientVersion":"2.7.1","osType":"Linux","nodeVersion":"v16.20.0"} 2023-06-20T07:44:18.513Z ERROR Metamorphing to an Actor that is paid per dataset item is not yet possible. 2023-06-20T07:44:18.516Z ApifyApiError: Metamorphing to an Actor that is paid per dataset item is not yet possible.(details: type=cannot-metamorph-to-paid-actor) 2023-06-20T07:44:18.518Z clientMethod: RunClient.metamorph 2023-06-20T07:44:18.520Z statusCode: 403 2023-06-20T07:44:18.521Z type: cannot-metamorph-to-paid-actor 2023-06-20T07:44:18.523Z attempt: 1 2023-06-20T07:44:18.525Z httpMethod: post 2023-06-20T07:44:18.527Z path: /v2/actor-runs/FDnf679nLWzszzDzm/metamorph 2023-06-20T07:44:18.529Z stack: 2023-06-20T07:44:18.531Z at makeRequest (/usr/src/app/node_modules/apify-client/dist/http_client.js:184:30) 2023-06-20T07:44:18.533Z at processTicksAndRejections (node:internal/process/task_queues:96:5) 2023-06-20T07:44:18.535Z at async RunClient.metamorph (/usr/src/app/node_modules/apify-client/dist/resource_clients/run.js:76:26) 2023-06-20T07:44:18.537Z at async Actor.metamorph (/usr/src/app/node_modules/apify/actor.js:372:9) 2023-06-20T07:44:18.539Z at async file:///usr/src/app/main.mjs:6:3 2023-06-20T07:44:18.541Z at async /usr/src/app/node_modules/apify/actor.js:132:23 2023-06-20T07:44:18.543Z at async file:///usr/src/app/main.mjs:3:1 2023-06-20T07:44:18.545Z ERROR Actor finished with an error (exit code 91)

User avatar

This error started today and should be fixed now.

Developer
Maintained by Community
Actor metrics
  • 281 monthly users
  • 100.0% runs succeeded
  • 12 hours response time
  • Created in Feb 2022
  • Modified 7 days ago
Categories