Unlock valuable insights from Skool.com discussions. Extract comprehensive post data and nested comments with ease, empowering your community analysis and content strategy.

Overview

This actor allows you to scrape posts and their associated comments from Skool.com. It extracts detailed information about each post, including content, metadata, user information, and nested comments, providing a complete picture of discussions within Skool communities.

Features

• Detailed Post Information: Extract comprehensive data about each post, including title, content, and metadata.
• User Details: Retrieve information about post authors and commenters.
• Nested Comments: Capture full comment threads, including replies to comments.
• Customizable Depth: Option to include or exclude comments on each post.
• Flexible Input: Support for specific Skool.com community URLs.
• Engagement Metrics: Capture upvotes, comment counts, and other relevant statistics.
• Proxy Support: Built-in proxy configuration to enhance scraping reliability and avoid blocks.

How to Use

1. Set Up: Ensure you have an Apify account and access to the Apify platform.
2. Install Cookie-Editor (there is option to Export cookies as JSON) or EditThisCookie chrome extension.
3. Login to your https://www.skool.com/ account
4. While you are on Skool tab, Click on the extension and export the cookies
5. Paste the cookies to this actor's Cookie input field (delete everything before pasting in it)
6. Specify the Skool's group URL you want to scrape. Input your URLs in the format https://www.skool.com/{group-name} (see Input Configuration section). You can add multiple URLs for a broader scraping scope
7. Configure Member Details: Choose whether to include comments for each group by setting the includeComments option.
8. (Optional) Customize Settings: Adjust the scraper settings to your preference, including the max number of listings to scrape, maximum concurrency, minimum concurrency, max request retries and any specific data fields you wish to collect.
9. (Optional)Bypass Site Protection: In order to bypass the protection of the site, use residential proxies from the country where you are scraping from. This approach helps in mimicking the behavior of a regular user from that region, minimizing the risk of detection and blocking.
10. Run the Scraper: Launch the scraper on the Apify platform. You can monitor its progress and adjust settings as needed.
11. Data Collection: Extracted data will be available in your preferred format, with support for JSON, HTML, CSV, Excel, and other formats provided by Apify.

Input Configuration

Here's an example of how to set up the input for the Skool Posts With Comments Scraper:

1{
2    "startUrls": [
3        {
4            "url": "https://www.skool.com/ai-automation-mastery"
5        }
6    ],
7    "includeComments": true,
8    "maxItems": 1000,
9    "maxConcurrency": 100,
10    "minConcurrency": 1,
11    "maxRequestRetries": 10
12}

Input Fields Explanation

• startUrls: Array of Skool.com community URLs to scrape posts and comments from.
• includeComments: Boolean to determine whether to scrape comments for each post (default: true).
• maxItems: Maximum number of posts to scrape (default: 1000).
• maxConcurrency: Maximum number of pages processed simultaneously (default: 100).
• minConcurrency: Minimum number of pages processed simultaneously (default: 1).
• maxRequestRetries: Number of retries for failed requests (default: 10).

Output Structure

The output data includes detailed information about each post and its comments. Here's a sample of the structure:

1{
2    "id": "01e8b631b67347b082c95056160df60f",
3    "name": "ever-heard-of-prompt-chaining-with-human-in-the-loop",
4    "metadata": {
5      "action": 0,
6      "attachmentsEdit": 1725675290488096300,
7      "comments": 2,
8      "content": "AI workflow with human in the loop is another effective way to automate tasks in high-quality with AI.\n\nExample? Generating a blog post. Every step in this process, from brainstorming ideas, researching, and outlining, to writing the final post, requires humans for confirmation and feedback.\n\nThe same goes for a lot of other use cases. How are you implementing this human-in-the-loop thing with AI? 🤔\n\nThere are many platforms to build AI automation. However, I do not think there are many that allows human-in-the-loop yet.",
9      "title": "Ever heard of AI multi-agent workflows with human in the loop?",
10      "upvotes": 1
11    },
12    "createdAt": "2024-09-07T02:10:39.028Z",
13    "updatedAt": "2024-09-17T17:07:53.029Z",
14    "groupId": "084f62bdfd304b36a03cdc3dd0e55ea2",
15    "userId": "43e8f80d28a4459daab6f2677f8439e9",
16    "postType": "generic",
17    "user": {
18      "id": "43e8f80d28a4459daab6f2677f8439e9",
19      "name": "ha-my-tran-7705",
20      "metadata": {
21        "bio": "Entrepreneur",
22        "pictureBubble": "https://assets.skool.com/f/43e8f80d28a4459daab6f2677f8439e9/3383230567c64d5eb2bb537d28250207ebef9e78bfd1420a939bce7487817615-sm.jpg",
23        "pictureProfile": "https://assets.skool.com/f/43e8f80d28a4459daab6f2677f8439e9/3383230567c64d5eb2bb537d28250207ebef9e78bfd1420a939bce7487817615-md.jpg"
24      },
25      "firstName": "Ha My",
26      "lastName": "Tran"
27    },
28    "comments": [
29      {
30        "post": {
31          "id": "ba1d97a1b7104a308eca09b7a33a13d3",
32          "metadata": {
33            "content": "For me personally, I often build my AI workflows with human in the loop, like these [https://youtu.be/N-5lKDHQOMM](https://youtu.be/N-5lKDHQOMM) Super easy and fast to get started yet customizable enough to fit my exact use case",
34            "upvotes": 1
35          },
36          "user": {
37            "id": "43e8f80d28a4459daab6f2677f8439e9",
38            "name": "ha-my-tran-7705",
39            "firstName": "Ha My",
40            "lastName": "Tran"
41          }
42        },
43        "children": [
44          {
45            "post": {
46              "id": "b86df88d5fda4682b5e1d3146e5cb226",
47              "metadata": {
48                "content": "[@Ha My Tran](obj://user/43e8f80d28a4459daab6f2677f8439e9) it's nice to know about it",
49                "upvotes": 1
50              },
51              "user": {
52                "id": "f72c8a41c5e64f2c85abc890a88aa0c9",
53                "name": "bolla-jagadish-3451",
54                "firstName": "Bolla",
55                "lastName": "Jagadish"
56              }
57            }
58          }
59        ]
60      }
61    ]
62}

Output Fields Explanation

• id: Unique identifier for the post.
• name: URL-friendly name of the post.
• metadata: Object containing various metadata about the post, including content, title, and engagement metrics.
• createdAt: Timestamp of when the post was created.
• updatedAt: Timestamp of the last update to the post.
• groupId: Identifier for the group or community the post belongs to.
• userId: Identifier of the post author.
• postType: Type of the post (e.g., "generic").
• user: Object containing information about the post author.
• comments: Array of comment objects, each containing:
  • post: Object with comment details and author information.
  • children: Array of nested replies to the comment.

Related scrapers

Skool Members Scraper - gather extensive users details such as first and last name, email, user's links to: facebook, instagram, linkedin, twitter, website, youtube and etc.

Support

• For issues or feature requests, please use the Issues section of this actor.
• If you need customization or have questions, feel free to contact the author:
  • Author's website: https://muhamed-didovic.github.io/
  • Email: muhamed.didovic@gmail.com

Additional Services

• Request customization or whole dataset: muhamed.didovic@gmail.com
• If you need anything else scraped, or this actor customized, email: muhamed.didovic@gmail.com
• For API services of this scraper (no Apify fee, just usage fee for the API), contact: muhamed.didovic@gmail.com