GitHub Discussions Scraper

Extract public GitHub Discussions, comments, and replies from repository and organization communities—without requiring a GitHub token.

Use GitHub Discussions Scraper to turn community questions, product feedback, announcements, and support conversations into structured JSON, CSV, Excel, or API-ready datasets.

• 🔎 Discover discussions from public index pages
• 💬 Preserve bodies, comments, and nested replies
• ✅ Capture answer state, categories, labels, and reactions
• 🕐 Run incremental monitoring with an update cutoff
• 🤖 Feed support analytics, research, and RAG pipelines

What does GitHub Discussions Scraper do?

The actor accepts public GitHub discussion index URLs or individual discussion URLs. It discovers discussion pages, downloads the server-rendered HTML, and emits one normalized record per discussion.

Supported sources include:

• Organization indexes such as https://github.com/orgs/community/discussions
• Repository indexes such as https://github.com/OWNER/REPOSITORY/discussions
• Individual organization discussions
• Individual repository discussions

No GitHub login or API token is required for the public scope.

Who is it for?

Developer relations teams

Track community questions, recurring friction, feature requests, and engagement around releases.

Support and success teams

Export solved and unanswered conversations into a searchable support corpus.

Product researchers

Analyze discussion themes, categories, reactions, authors, and response activity over time.

AI and RAG teams

Collect discussion bodies and comment trees as grounded, attributable knowledge-base documents.

Open-source maintainers

Monitor multiple repositories for newly active conversations without checking each page manually.

Why use this GitHub Discussions scraper?

GitHub's interface is designed for reading one conversation at a time. Repeated analysis across communities requires structured, stable records.

This actor provides:

• A credential-free HTTP workflow
• Stable discussion IDs and canonical URLs
• Text and rendered HTML bodies
• Provenance through the original input URL
• Configurable row and comment limits
• Apify scheduling, webhooks, datasets, and integrations
• Pay-per-event billing aligned with produced records

What data can you extract?

Field	Type	Description
id	string	Stable owner/repository and discussion number
url	string	Canonical discussion URL
sourceUrl	string	Input index or detail URL
owner	string	Repository owner or organization
repository	string/null	Repository name; null for organization discussions
number	number	GitHub discussion number
title	string	Discussion title
bodyText	string	Plain-text discussion body
bodyHtml	string	Rendered body HTML
category	string/null	GitHub Discussions category
author	string/null	Author login
authorUrl	string/null	Public author profile URL
createdAt	string/null	Creation timestamp
updatedAt	string/null	Latest rendered activity timestamp
isAnswered	boolean	Whether GitHub marks the question answered
labels	array	Rendered labels and category tags
reactions	array	Reaction labels and counts
commentCount	number	Full rendered comment count
comments	array	Bounded comments with reply arrays
scrapedAt	string	Extraction timestamp

How to scrape GitHub Discussions

1. Open the actor input page.
2. Add one or more public Discussions URLs.
3. Choose the maximum number of discussions.
4. Keep comments enabled for complete conversations.
5. Optionally set an update cutoff for incremental monitoring.
6. Run the actor.
7. Export the dataset or consume it through the API.

Start with a small run to verify the community and output shape. Increase limits only when the results match your workflow.

Input options

startUrls

Required list of public github.com Discussions URLs. Mix index and direct discussion URLs in one run when needed.

maxDiscussions

Global maximum number of discussion records. The actor deduplicates discovered URLs before fetching details.

updatedAfter

Optional ISO 8601 date or timestamp. Discussions whose latest rendered timestamp is older are not emitted.

category

Optional GitHub category name for index filtering. Use the category wording shown by the target community.

sort

Choose latest activity, top, newest, or oldest ordering.

includeComments and includeReplies

Disable conversation extraction when only discussion-level metadata is needed. Replies remain nested beneath their parent when GitHub renders that relationship.

maxCommentsPerDiscussion

Limits extracted comments per discussion to control runtime and record size.

proxyConfiguration

Optional Apify Proxy configuration for larger workloads or network-specific routing. Direct HTTP is the recommended default.

Example input

{
  "startUrls": [
    { "url": "https://github.com/orgs/community/discussions" }
  ],
  "maxDiscussions": 20,
  "maxCommentsPerDiscussion": 100,
  "includeComments": true,
  "includeReplies": true,
  "sort": "latest"
}

For an incremental run:

{
  "startUrls": [
    { "url": "https://github.com/orgs/community/discussions" }
  ],
  "maxDiscussions": 100,
  "updatedAfter": "2026-01-01T00:00:00Z"
}

Example output

{
  "id": "community/discussions#175571",
  "url": "https://github.com/orgs/community/discussions/175571",
  "owner": "community",
  "repository": null,
  "number": 175571,
  "title": "Sub-accounts of a primary GitHub Account",
  "category": "Other Feature Feedback, Questions, & Ideas",
  "author": "BenjamenMeyer",
  "createdAt": "2025-10-02T14:47:08Z",
  "updatedAt": "2026-03-20T08:44:01Z",
  "isAnswered": false,
  "commentCount": 2,
  "comments": [
    {
      "id": "14575338",
      "author": "softwarecreations",
      "bodyText": "Example comment text",
      "createdAt": "2025-10-02T15:29:14Z",
      "replies": []
    }
  ]
}

How much does it cost to scrape GitHub Discussions?

The actor uses pay-per-event pricing:

• A $0.005 start event is charged once per run.
• Each discussion pushed to the dataset is charged as one result event.
• Subscription tiers receive progressively lower per-result prices. For example, the current BRONZE price is $0.00018286 per discussion; always confirm the active price in Apify Console before running.

Example actor charges at the BRONZE rate:

Results	Calculation	Estimated charge
20	$0.005 + 20 × $0.00018286	$0.0087
100	$0.005 + 100 × $0.00018286	$0.0233
1,000	$0.005 + 1,000 × $0.00018286	$0.1879

On the FREE tier ($0.00021028 per result), $5 of platform credits covers about 23,750 discussions in one run after the start fee. Multiple runs each incur the start fee, and your available Apify credits and active prices may differ. You can control cost directly with maxDiscussions and comment-depth settings.

Incremental monitoring workflow

Schedule a daily or hourly actor task. Store the last successful run timestamp in your automation. Pass that value as updatedAfter on the next run.

Then:

1. Read the new dataset.
2. Compare stable id values with your destination.
3. Upsert changed discussions.
4. Notify the owning support or product team.
5. Advance the cutoff only after successful ingestion.

This avoids repeatedly processing older conversations downstream.

Tips for reliable results

• Use canonical public GitHub URLs.
• Start with 10–20 discussions.
• Limit comments when monitoring very large communities.
• Keep direct HTTP enabled unless your network requires a proxy.
• Use stable id rather than title as the database key.
• Retain sourceUrl for auditability.
• Treat body HTML as untrusted user-generated content in your UI.
• Respect GitHub's terms and avoid excessive scheduling frequency.

Integrations

Google Sheets and Airtable

Send newly updated discussions to a triage table for tagging and ownership.

Slack and Microsoft Teams

Trigger alerts for unanswered questions or discussions in priority categories.

Webhooks

Run a workflow when the actor finishes and consume the dataset URL from the event payload.

Zapier and Make

Create tickets, update a CRM, or append records to a research workspace.

Data warehouses

Load stable discussion rows into BigQuery, Snowflake, PostgreSQL, or object storage.

Vector databases

Chunk title, bodyText, and comment bodies while preserving url metadata for citations.

API usage

Run the actor programmatically with the JavaScript client, Python client, or direct HTTP API. The examples below use the stable actor ID automation-lab/github-discussions-scraper and return results through the run's default dataset.

Run with the Apify API using JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/github-discussions-scraper').call({
  startUrls: [{ url: 'https://github.com/orgs/community/discussions' }],
  maxDiscussions: 20,
  includeComments: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Run with the Apify API using Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("automation-lab/github-discussions-scraper").call(run_input={
    "startUrls": [{"url": "https://github.com/orgs/community/discussions"}],
    "maxDiscussions": 20,
    "includeComments": True,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Run with cURL

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~github-discussions-scraper/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://github.com/orgs/community/discussions"}],"maxDiscussions":20}'

Fetch dataset items after the run succeeds through the run's defaultDatasetId.

Use with Apify MCP and AI agents

Connect Claude Code or another remote-MCP client to:

https://mcp.apify.com?tools=automation-lab/github-discussions-scraper

Claude Code

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/github-discussions-scraper"

Complete the Apify authorization flow when Claude prompts you. Verify the connection with claude mcp list.

Claude Desktop, Cursor, and VS Code

Add this server to the client's MCP JSON configuration (Claude Desktop: Settings → Developer → Edit Config; Cursor: .cursor/mcp.json; VS Code: .vscode/mcp.json):

{
  "mcpServers": {
    "apify-github-discussions": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=automation-lab/github-discussions-scraper"
    }
  }
}

For VS Code versions that use the servers key, use:

{
  "servers": {
    "apify-github-discussions": {
      "type": "http",
      "url": "https://mcp.apify.com?tools=automation-lab/github-discussions-scraper"
    }
  }
}

Restart the client if needed, select the apify-github-discussions server, and complete Apify authentication in the browser. Do not put an Apify token directly in a committed JSON file.

Example prompts:

• “Extract the latest 20 discussions from this GitHub organization and summarize unresolved themes.”
• “Collect this repository discussion with comments and prepare a cited support answer.”
• “Monitor discussions updated this week and group them by category.”

The actor's structured URLs make source citation straightforward.

Error handling

Invalid or non-GitHub URLs fail early with an actionable message. A failed individual detail page is logged and skipped so other discussions can continue. GitHub 4xx and 5xx responses are retried conservatively before being skipped.

A successful run with zero records usually means:

• The index contained no matching public discussions.
• The category filter did not match GitHub's category wording.
• The update cutoff excluded all discovered discussions.
• The target community changed visibility.

Legality and responsible use

The actor accesses publicly available pages without bypassing authentication. Public availability does not remove your responsibilities.

You should:

• Follow GitHub's Terms of Service.
• Respect applicable privacy and database laws.
• Avoid collecting unnecessary personal data.
• Use reasonable run frequency and limits.
• Secure exported datasets.
• Honor deletion or compliance obligations applicable to your use case.

This documentation is not legal advice.

FAQ

Does it require a GitHub token?

No. The baseline public scope is credential-free.

Can it scrape private discussions?

No. It supports only pages publicly visible without authentication.

Does it support repository and organization discussions?

Yes. Both index types and direct discussion URLs are supported.

Why is repository null?

Organization-level discussions do not belong to one repository, so repository is intentionally null.

Why are some reaction arrays empty?

GitHub may not render reaction controls when there are no reactions, or markup may vary by conversation state.

Why did my incremental run return no records?

Check that updatedAfter is valid ISO 8601 and earlier than the latest activity shown on GitHub.

Can I export CSV or Excel?

Yes. Use the dataset export controls or API format parameter. Nested comments are best preserved in JSON exports.

How should I deduplicate records?

Use the stable id or canonical url, not the title.

Related scrapers

Combine this actor with other automation-lab GitHub tools when your workflow also needs repository, contributor, issue, or release metadata. Browse the automation-lab Apify profile for currently available related actors.

Use GitHub Discussions Scraper specifically when conversation bodies, answer state, comments, and replies are the core records you need.

Support

If a public discussion page fails to parse, include:

• The public input URL
• The run ID
• The expected field
• A short description of the observed result

Do not include private credentials or sensitive data in a support request.