Reddit Scraper

What does Reddit Scraper do?

Our unofficial Reddit API will get data from Reddit with no limitations or authentication. It enables you to extract posts and comments together with some user info without login. It is built on top of Apify SDK, and you can run it both on the Apify platform.

Reddit Scraper allows you to:

• scrape subreddits (communities) with top posts
• scrape Reddit posts with title and text, username, number of comments, votes, media elements.
• get Reddit comments, timestamps, points, usernames, post and comment URLs.
• scrape user details, their most recent posts and comments.
• sort scraped data by categories of Relevance, Hot, Top, and New.
• scrape data using a specific URL or by keyword.

What data can I extract from Reddit?

📌 Popular subreddits	🔍 Subreddit details
📋 Subreddit name	👥 Number of members
🌐 Community URL	📚 Category
📌 Reddit posts	💬 Reddit comments
📃 Title and text	⏱ Timestamps
👤 Username	🔗 Post and comment URLs
👍 Votes	📷 Media elements
👤 User details	📄 Recent posts and comments

How much will it cost to scrape Reddit?

Reddit Scraper on the Apify platform will give you 1,000 results for less than $4 in platform usage credits. That should be covered by the free $5 in monthly credits you get on every Apify Free plan.

But if you need to get more data regularly from Reddit, you should grab an Apify subscription. We recommend our $49/month Starter plan - with that one, you can get well over 10,000 results every month! Watch this video for a few helpful tips on how to pick a plan.

How to scrape Reddit?

Reddit Scraper doesn't require any coding skills to start using it.

1. Create a free Apify account using your email.
2. Open the Reddit Scraper.
3. Add one or more subreddits, users or post URLs to scrape their information.
4. Click "Start" and wait for the scraper to extract the data.
5. Download your data in JSON, XML, CSV, Excel, or HTML format.

If you're unsure where to start, just follow our step-by-step guide or see our short video tutorial. The tutorial steps can be also be used for Reddit Scraper Lite.

How to use scraped Reddit data

• Keep track of discussions about your brand or product across Reddit communities.
• Research the topics that interest you and get a wide range of opinions.
• Keep an eye on debates over high stakes subjects such as finance, politics, new technology, and news in general.
• Watch out for new trends, attitudes, and PR opportunities.
• Automatically track mentions of the business or topic that interests you.
• Scrape Reddit comments to kick off and support your sentiment analysis.

Input parameters

If this Actor is run on the Apify platform, there are two ways you can scrape Reddit:

1. by Start URLs field - this will get you all details from any Reddit URL, no matter whether it's a post, a user, or a community.
2. or by Search Term field - this will scrape all data from Reddit in Communities, Posts, and People for a specific keyword.

How to scrape Reddit by URLs

Almost any URL from Reddit will return a dataset. If the URL is not supported, the scraper will display a message before scraping the page.

Input examples

Here are some examples of URLs that can be scraped.

• scraping communities: https://www.reddit.com/r/worldnews/

• scraping channels within communities: https://www.reddit.com/r/worldnews/hot

• scraping popular communities: https://www.reddit.com/subreddits/leaderboard/crypto/

• scraping users: https://www.reddit.com/user/lukaskrivka

• scraping user comments: https://www.reddit.com/user/lukaskrivka/comments/

• scraping posts: https://www.reddit.com/r/learnprogramming/comments/lp1hi4/is_webscraping_a_good_skill_to_learn_as_a_beginner/

• scraping popular posts: https://www.reddit.com/r/popular/

• scraping search results:

• for users/communities: https://www.reddit.com/search/?q=news&type=sr%2Cuser

• for posts: https://www.reddit.com/search/?q=news

Note: if you use a search URL as a parameter for startUrls, it will only scrape for posts. If you want to search for communities and users, use the search field or a specific URL instead.

How to scrape Reddit by search term

• Search Term or searches - the keywords you want to search via the Reddit's search engine. You can keep one field or add as many as you want. Don't use this field if you're using the startUrls parameter.

• Search type or type - indicates which part of Reddit you're scraping: "Posts" or "Communities and users".

• Sort search or sort - will sort search results by Relevance, Hot, Top, New or most amount of Comments.

• Filter by date or time - will filter the search by the last hour, day, week, month or year. Only available if you're scraping Posts.

To see the full list of parameters, their default values, and how to set the values of your own, head over to Input Schema tab.

Input example

This is an example of how your input will look like if you decide to scrape all Reddit communities that contain the keyword parrot. Results will be sorted by the newest first.

1{
2  "maxItems": 10,
3  "maxPostCount": 10,
4  "maxComments": 10,
5  "maxCommunitiesCount": 10,
6  "maxUserCount": 10,
7  "maxLeaderBoardItems": 10,
8  "scrollTimeout": 40,
9  "proxy": {
10    "useApifyProxy": true
11  },
12  "searches": ["parrots"],
13  "type": "community",
14  "sort": "new",
15  "time": "all"
16}

Results

The output from scraping Reddit is stored in the dataset. Each post, comment, user or community is stored as an item inside the dataset. After the run is finished, you can download the scraped data onto your computer or export to any web app in various data formats (JSON, CSV, XML, RSS, HTML Table). Here's a few examples of the outputs you can get for different types of inputs:

📝 Example Reddit post

1{
2  "id": "t3_144w7sn",
3  "parsedId": "144w7sn",
4  "url": "https://www.reddit.com/r/HonkaiStarRail/comments/144w7sn/my_luckiest_10x_pull_yet/",
5  "username": "YourKingLives",
6  "title": "My Luckiest 10x Pull Yet",
7  "communityName": "r/HonkaiStarRail",
8  "parsedCommunityName": "HonkaiStarRail",
9  "body": "URL: https://i.redd.it/yod3okjkgx4b1.jpg\nThumbnail: https://b.thumbs.redditmedia.com/lm9KxS4laQWgx4uOoioM3N7-tBK3GLPrxb9da2hGtjs.jpg\nImages:\n\thttps://preview.redd.it/yod3okjkgx4b1.jpg?auto=webp&v=enabled&s=be5faf0250e19138b82c7bbe5e7406fa46da4e73\n",
10  "html": null,
11  "numberOfComments": 0,
12  "upVotes": 1,
13  "isVideo": false,
14  "isAd": false,
15  "over18": false,
16  "createdAt": "2023-06-09T05:23:15.000Z",
17  "scrapedAt": "2023-06-09T05:23:28.409Z",
18  "dataType": "post"
19},

💬 Example Reddit comment

1{
2  "id": "t1_jnhqrgg",
3  "parsedId": "jnhqrgg",
4  "url": "https://www.reddit.com/r/NewsWithJingjing/comments/144v5c3/theres_no_flag_large_enough/jnhqrgg/",
5  "parentId": "t3_144v5c3",
6  "username": "smokecat20",
7  "category": "NewsWithJingjing",
8  "communityName": "r/NewsWithJingjing",
9  "body": "A true patriot.",
10  "createdAt": "2023-06-09T05:00:00.000Z",
11  "scrapedAt": "2023-06-09T05:23:32.025Z",
12  "upVotes": 3,
13  "numberOfreplies": 0,
14  "html": "<div class=\"md\"><p>A true patriot.</p>\n</div>",
15  "dataType": "comment"
16}

👥 Example Reddit community

1{
2  "id": "2qlhq",
3  "name": "t5_2qlhq",
4  "title": "Pizza",
5  "headerImage": "https://b.thumbs.redditmedia.com/jq9ytPEOecwd5bmGIvNQzjTPE9hdd0kB9XGa--wq55A.png",
6  "description": "The home of pizza on reddit. An educational community devoted to the art of pizza making.",
7  "over18": false,
8  "createdAt": "2008-08-26T00:03:48.000Z",
9  "scrapedAt": "2023-06-09T05:16:55.443Z",
10  "numberOfMembers": 569724,
11  "url": "https://www.reddit.com/r/Pizza/",
12  "dataType": "community"
13}

👤 Example Reddit user

1{
2  "id": "c3h2qmv",
3  "url": "https://www.reddit.com/user/jancurn/",
4  "username": "jancurn",
5  "userIcon": "https://www.redditstatic.com/avatars/defaults/v2/avatar_default_7.png",
6  "postKarma": 4,
7  "commentKarma": 10,
8  "description": "",
9  "over18": false,
10  "createdAt": "2018-09-10T15:13:39.000Z",
11  "scrapedAt": "2023-06-09T05:21:14.409Z",
12  "dataType": "user"
13}

Only need a few Reddit results?

Use our super fast dedicated Reddit Scraper Lite if you want to scrape Reddit data on a smaller scale. Just enter one or more Reddit URLs or keywords and click to scrape.

Notes for developers

Limiting results with maxItems

If you need to limit the scope of your search, you can do that by setting the max number of posts you want to scrape inside a community or user. You can also set a limit to the number of comments for each post. You can limit the number of communities and the number of leaderboards by using the following parameters:

1{
2  "maxItems": 100,
3  "maxPostCount": 50,
4  "maxComments": 10,
5  "maxCommunitiesCount": 5,
6  "maxUserCount": 5,
7  "maxLeaderBoardsItems": 5
8}

You can also set maxItems to prevent a very long run of the Actor. This parameter will stop your scraper when it reaches the number of results you've indicated, so you need to be careful not to trim your results.

See the Input Schema tab for the full list of the ways to restrict Reddit Scraper using these parameters: maxItems, maxPostCount, maxComments, maxCommunitiesCount, maxLeaderBoardItems

Extend output function

You can use this function to update the result output of this Actor. You can choose what data from the page you want to scrape. The output from this function will get merged with the result output.

The return value of this function has to be an object!

You can return fields to achieve 3 different things:

• Add a new field - Return object with a field that is not in the result output
• Change a field - Return an existing field with a new value
• Remove a field - Return an existing field with a value undefined

1async () => {
2  return {
3    pageTitle: document.querySelector('title').innerText,
4  };
5};

This example will add the title of the page to the final object:

1{
2  "id": "2qlhq",
3  "name": "t5_2qlhq",
4  "title": "Pizza",
5  "headerImage": "https://b.thumbs.redditmedia.com/jq9ytPEOecwd5bmGIvNQzjTPE9hdd0kB9XGa--wq55A.png",
6  "description": "The home of pizza on reddit. An educational community devoted to the art of pizza making.",
7  "over18": false,
8  "createdAt": "2008-08-26T00:03:48.000Z",
9  "scrapedAt": "2023-06-09T05:16:55.443Z",
10  "numberOfMembers": 569724,
11  "url": "https://www.reddit.com/r/Pizza/",
12  "dataType": "community"
13  "pageTitle": "homemade chicken cheese masala pasta"
14}

FAQ

Is Reddit scraping legal?

While scraping publicly available data from Reddit is generally allowed, it's important to comply with Reddit's terms of service and respect the site's usage policies. It's recommended to use the scraper responsibly, avoid excessive requests, and ensure that the scraped data is used in compliance with applicable laws and regulations. You can read more about compliance with ToS in our blogpost.

Can I use Reddit API to scrape Reddit?

The Reddit API is currently free. However, Reddit has specific API rules, and free access to the API will be restricted in the future due to concerns over data usage. In addition, Reddit API has been recently announced to become a paid service. Using a Reddit web scraper such as this one as an API has advantages such over the official one such as not requiring authentication, special authorization for commercial use, or registration for a token.

How can I scrape Reddit comments?

Reddit Scraper allows scraping specific parts of Reddit, including comments. You can extract posts and comments along with user information, such as timestamps, number of votes, usernames, post URL, and comment URLs. This enables you to gather comprehensive commment data from subreddits and Reddit users.

Is it necessary to use cookies for accessing logged-in content when scraping Reddit?

No, it is not required. As of May 2023, Reddit maintains its data publicly accessible and does not enforce a login barrier.

Do you need proxies for scraping Reddit?

It is highly recommended. Subreddits are open for access and do not require a login to retrieve information. Typically, using proxies is necessary to ensure successful Reddit scraping. While some results can be obtained with datacenter proxies, residential proxies are preferred for Reddit scraping. Fortunately, our Free plan offers a trial of Apify Proxy, which get you started.

Can I export or import scraped Reddit data using API?

Yes. The Apify API gives you programmatic access to the Apify platform. The API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run any Apify Actor, including this one. The API also lets you access any datasets, monitor Actor performance, fetch results, create and update versions, and more.

To access the API using Node.js, use the apify-client NPM package. To access the API using Python, use the apify-client PyPi package.

Check out the Apify API reference docs for full details or click on the API tab for code examples.

How can I build a Reddit scraper in Python?

You can create your own Reddit web scraper using a Python scraper template directly on the Apify platform and keep production there. Alternatively, you can develop it locally on your computer and only push it to the Apify cloud during deployment.

What is the difference between Reddit Scraper and Reddit Scraper Lite?

The functionality of both actor are the same. The difference is that the Lite version charges based on the number of results it extracts and it has a custom proxy configuration integrated with the actor that is cheaper than then the Residential proxies used by Apify. The Reddit Scraper version allows you to use either one of Apify`s proxies or a custom one.