# Discourse Forum Scraper (`bindler/discourse-forum-scraper`) Actor

Extract topics and full post text from any Discourse forum. Clean plain-text output for LLM training, RAG corpora and community research.

- **URL**: https://apify.com/bindler/discourse-forum-scraper.md
- **Developed by:** [Neil Sangwaiya](https://apify.com/bindler) (community)
- **Categories:** AI, Developer tools, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Discourse Forum Scraper

Extract every topic and full post text from **any Discourse forum**, as clean plain text. Built for LLM training data, RAG corpora, and community research.

Discourse powers thousands of developer, product and support communities: Discourse Meta, Rust, Elixir, Home Assistant, OpenAI, Hugging Face, Figma, McNeel, Ubuntu, and countless niche forums. This Actor works on all of them without configuration. Just paste the forum URL.

### What you get

| Field | Description |
|---|---|
| `topicId` | Discourse topic ID |
| `title` | Topic title |
| `url` | Direct link to the thread |
| `category` | Category ID |
| `tags` | Topic tags |
| `createdAt` | When the thread started |
| `lastPostedAt` | Most recent reply |
| `replyCount` | Number of replies |
| `postCount` | Total posts |
| `views` | View count |
| `likeCount` | Likes on the thread |
| `pinned` / `closed` | Thread state |
| `posts[]` | Every post: author, date, plain text, likes, post number |
| `fullText` | The whole thread as one clean text block, ready for embedding |
| `forum` | Source forum |
| `scrapedAt` | ISO timestamp |

### Why `fullText` matters

Discourse returns posts as HTML. Feeding that to a model wastes tokens on markup. This Actor strips it to plain text and joins the thread into a single `fullText` field, so each record drops straight into a vector store or fine-tuning set with no preprocessing.

It also handles Discourse's post pagination. Discourse only returns the first chunk of posts in a topic payload, so most scrapers silently truncate long threads. This one fetches the remaining posts and returns the complete conversation.

### Three modes

- **Latest topics** — the forum's recent activity
- **A category** — pass a category slug like `feature` or `support/33`
- **Search results** — pass any Discourse search query

### Example input

```json
{
  "forumUrl": "https://meta.discourse.org",
  "mode": "latest",
  "maxTopics": 500,
  "includePosts": true,
  "maxPostsPerTopic": 100
}
```

### Example output

```json
{
  "topicId": 1,
  "title": "New to Discourse? Start here!",
  "url": "https://meta.discourse.org/t/new-to-discourse-start-here/1",
  "replyCount": 5,
  "views": 118812,
  "posts": [
    {
      "postNumber": 1,
      "author": "Discourse",
      "createdAt": "2023-05-15T10:00:00.000Z",
      "text": "We're so glad you're here! This is our official community...",
      "likes": 42,
      "isOriginalPost": true
    }
  ],
  "fullText": "Discourse: We're so glad you're here!...",
  "scrapedAt": "2026-09-09T19:19:33.000Z"
}
```

### Cost control

`maxTopics` and `maxPostsPerTopic` cap every run. You are charged per topic returned, so a run's cost is knowable before you start it.

### Notes

- Uses only public, unauthenticated Discourse JSON endpoints. No login required, no private content accessed.
- Rate-limited and retried politely, with backoff on HTTP 429.
- Works on self-hosted Discourse instances as well as hosted ones.

# Actor input Schema

## `forumUrl` (type: `string`):

Base URL of any Discourse forum, e.g. https://meta.discourse.org

## `mode` (type: `string`):

Latest topics, a single category, or search results.

## `categorySlug` (type: `string`):

Only used when mode is 'category'. Example: 'feature' or 'support/33'.

## `searchQuery` (type: `string`):

Only used when mode is 'search'.

## `maxTopics` (type: `integer`):

Stop after this many topics. Controls your cost.

## `includePosts` (type: `boolean`):

Fetch every reply, not just topic metadata. Needed for LLM and RAG use.

## `maxPostsPerTopic` (type: `integer`):

Caps very long threads.

## Actor input object example

```json
{
  "forumUrl": "https://meta.discourse.org",
  "mode": "latest",
  "maxTopics": 100,
  "includePosts": true,
  "maxPostsPerTopic": 50
}
```

# Actor output Schema

## `topics` (type: `string`):

One record per topic, including every post as plain text and a combined fullText field ready for embedding.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "forumUrl": "https://meta.discourse.org"
};

// Run the Actor and wait for it to finish
const run = await client.actor("bindler/discourse-forum-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "forumUrl": "https://meta.discourse.org" }

# Run the Actor and wait for it to finish
run = client.actor("bindler/discourse-forum-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "forumUrl": "https://meta.discourse.org"
}' |
apify call bindler/discourse-forum-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bindler/discourse-forum-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Jqo4j7TROHGpcOhSh/builds/Dq579CXXAGja2fgEi/openapi.json