Reddit Scraper
Pricing
$45.00/month + usage
Reddit Scraper
Unlimited Reddit web scraper to crawl posts, comments, communities, and users without login. Limit web scraping by number of posts or items and extract all data in a dataset in multiple formats.
3.9 (2)
Pricing
$45.00/month + usage
106
Total users
6.5k
Monthly users
380
Runs succeeded
>99%
Issue response
2.2 days
Last modified
20 days ago
The duration for scraping more than trippled
Closed
For 500 posts, the duration jumped up from 5(ish) minutes to 15+ minutes consistently. attaching logs
input: { "searches": [ "gaming headset recommendations", "best gaming headset", "gaming headset", "carbon fiber headset" ], "sort": "new", "maxItems": 500, "maxPostCount": 500, "searchPosts": true, "skipComments": true, "scrollTimeout": 90, "timeoutSecs": 900, "customerId": "550e8400-e29b-41d4-a716-446655440000", "projectId": "25c4d7ca-9ad0-4745-8ca1-2377dcdcb5a6", "runMode": "LITE", "skipUserPosts": false, "skipCommunity": false, "searchComments": false, "searchCommunities": false, "searchUsers": false, "includeNSFW": true, "maxComments": 10, "maxCommunitiesCount": 2, "maxUserCount": 2, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] }, "debugMode": false }
I have removed a bug, can you try again and let me know if speed improved?
alex.af.junior
It's hard to say because the max number of posts it scrapes now caps at 35ish. Even if I put 300-500 posts to scrape. I tried it on different keywords, different sorting. Example input that returns 30ish posts: { "customerId": "550e8400-e29b-41d4-a716-446655440000", "debugMode": false, "includeNSFW": true, "maxComments": 10, "maxCommunitiesCount": 2, "maxItems": 300, "maxPostCount": 300, "maxUserCount": 2, "projectId": "f21ae5d6-1ee0-4a4b-8d51-3285f2d0aeec", "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] }, "runMode": "LITE", "scrollTimeout": 90, "searchComments": false, "searchCommunities": false, "searchPosts": true, "searchUsers": false, "searches": [ "Boston apparel", "street fashion Boston", "Boston Terrier clothing", "Boston sports team gear", "high quality Boston clothes" ], "skipComments": true, "skipCommunity": false, "skipUserPosts": false, "sort": "relevance", "timeoutSecs": 900 }
