GitHub Discussions Scraper

What does GitHub Discussions Scraper do?

GitHub Discussions Scraper collects public discussion pages from one or more GitHub repositories. It follows the public listing's next-page link, opens a bounded number of detail pages, and extracts discussion content, authors, categories, labels, answer state, timestamps, reactions, comments, and optional replies.

This is a public HTML scraper and API alternative for exporting discussion data. It does not log in, access private repositories, call undocumented private APIs, bypass challenges, or fabricate blocked/empty results. Requests are direct; proxy configuration is not supported or accepted.

Why use GitHub Discussions Scraper?

Use it for community research, support-topic analysis, product-feedback discovery, and public repository monitoring. Listing pagination preserves the query and records the producing searchUrl, pageNum, and link position. Discussion URLs and discussion numbers are deduplicated across pages and repositories. Detail records have a stable recordId and recordType=discussion, while run-level failures remain in the OUTPUT_SUMMARY key-value artifact instead of polluting the dataset.

What data can it extract?

Group	Fields
Identity	recordType, recordId, repository, discussionNumber, discussionId, discussionUrl
Content	title, body, authorLogin, authorUrl, state, isAnswered, categoryName, categoryUrl, labels
Engagement	commentCount, replyCount, bounded replies, reactions, reactionCount
Provenance	searchUrl, pageNum, position, sourceUrl, extractionMethod, accessStatus, scrapedAt

How to scrape GitHub Discussions

1. Open the Actor's Input tab.
2. Enter public owner/repository slugs or full GitHub repository URLs.
3. Optionally provide a GitHub discussions_q query.
4. Set per-repository discussion and listing-page limits, reply limits, and pacing.
5. Run the Actor and inspect normal rows plus OUTPUT_SUMMARY for pagination and errors.

How much will it cost to scrape GitHub Discussions?

Cost depends on the Apify plan and compute used. Each listing page and detail page is a public HTTP request. More repositories, listing pages, detail records, replies, retries, and longer timeouts increase work. Sequential requests and bounded arrays keep runs predictable; maxPagesPerRepository limits listing coverage and maxDiscussionsPerRepository limits successful detail records per repository.

Input

See the input tab for the complete configuration. The input is a JSON object.

Default smoke test

{
  "repositories": ["community/community"],
  "query": "sort:date_created_desc",
  "maxDiscussionsPerRepository": 3,
  "maxPagesPerRepository": 2,
  "includeReplies": true,
  "maxReplies": 3,
  "requestDelayMs": 100,
  "timeoutSecs": 30,
  "maxRequestRetries": 1
}

Multiple repositories and no replies

{
  "repositories": ["community/community", "octo/demo"],
  "query": "sort:date_created_desc",
  "maxDiscussionsPerRepository": 5,
  "maxPagesPerRepository": 2,
  "includeReplies": false,
  "maxReplies": 0
}

Conservative developer options

{
  "repositories": ["community/community"],
  "maxDiscussionsPerRepository": 2,
  "maxPagesPerRepository": 1,
  "requestDelayMs": 500,
  "timeoutSecs": 30,
  "maxRequestRetries": 0
}

repositories is normalized to at most 20 unique valid public repository slugs. query defaults to sort:date_created_desc and is preserved on every listing page. maxDiscussionsPerRepository is 1–50 per repository; maxPagesPerRepository is 1–10 per repository; maxReplies is 0–100. includeReplies=false produces no replies array. Delay is 0–5000 ms, timeout is 5–120 seconds, and transient retries are 0–3. Unknown properties and invalid repository shapes are rejected.

Output

Discussion record

{
  "recordType": "discussion",
  "recordId": "github-discussion:example%2Fproject:42",
  "dataAvailable": true,
  "accessStatus": "public",
  "extractionMethod": "github_html",
  "detailStatus": "succeeded",
  "repository": "example/project",
  "discussionNumber": 42,
  "discussionId": "DI_42",
  "title": "How do I configure this?",
  "body": "A public question body.",
  "authorLogin": "octocat",
  "authorUrl": "https://github.com/octocat",
  "state": "Answered",
  "isAnswered": true,
  "categoryName": "Q&A",
  "labels": [
    {
      "name": "help",
      "url": "https://github.com/orgs/community/discussions?discussions_q=label%3Ahelp"
    }
  ],
  "commentCount": 2,
  "replyCount": 1,
  "replies": [
    {
      "body": "Use the documented configuration file.",
      "authorLogin": "maintainer",
      "createdAt": "2026-08-24T01:00:00.000Z",
      "reactions": []
    }
  ],
  "reactions": [{ "name": "upvote", "count": 2 }],
  "reactionCount": 2,
  "discussionUrl": "https://github.com/orgs/community/discussions/42",
  "sourceUrl": "https://github.com/orgs/community/discussions/42",
  "searchUrl": "https://github.com/orgs/community/discussions?discussions_q=sort%3Adate_created_desc&page=1",
  "pageNum": 1,
  "position": 1,
  "scrapedAt": "2026-08-24T00:00:00.000Z"
}

Run summary in OUTPUT_SUMMARY

{
  "actorName": "github-discussions-scraper",
  "status": "SUCCEEDED",
  "dataAvailable": true,
  "repositories": ["example/project"],
  "query": "sort:date_created_desc",
  "maxDiscussionsPerRepository": 3,
  "maxPagesPerRepository": 2,
  "includeReplies": true,
  "maxReplies": 3,
  "listPagesFetched": 1,
  "discussionsDiscovered": 3,
  "detailPagesFetched": 3,
  "recordsStored": 3,
  "duplicatesSkipped": 0,
  "sourceBlocked": false,
  "errorCount": 0,
  "errors": [],
  "scrapedAt": "2026-08-24T00:00:00.000Z"
}

When a listing or detail page fails, status becomes PARTIAL if usable records remain, or BLOCKED_OR_EMPTY if none do. The summary includes bounded error entries with stage, page, URL, block classification, and message. Diagnostics are never represented as normal discussions.

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Tips and advanced options

Use a small maxPagesPerRepository for a quick run and increase it only when you need older discussions. maxDiscussionsPerRepository counts successful detail records, so a failed detail page does not become a placeholder. The Actor follows only observed public pagination links, stops on repeated/no-progress pages, and keeps includeReplies and maxReplies independent of the page-reported comment count.

FAQ, support, and responsible use

If the dataset is empty, inspect OUTPUT_SUMMARY for sourceBlocked, errorCount, and the per-stage errors. The API tab exposes the dataset and summary; for support, include the Apify run ID and summary in the Issues tab. Do not treat a blocked or empty result as evidence that a repository has no discussions.

Use public data lawfully and follow GitHub's terms, rate limits, robots guidance where applicable, and all privacy/data-protection laws. Public discussions may contain personal data; do not use this Actor to bypass access controls or collect private information. This Actor is not affiliated with GitHub.

Local validation

npm ci
npm run check
npm test
apify validate-schema
apify run --purge --input-file test/inputs/live-smoke.json
npm run validate