# Discourse Forum Scraper: Topics, Posts & Users (`arman-bd/discourse-forum-scraper`) Actor

Scrape Discourse forums: topics, posts, categories, tags, view counts, reply counts and user activity. Every Discourse instance exposes JSON by appending .json to any URL.

- **URL**: https://apify.com/arman-bd/discourse-forum-scraper.md
- **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community)
- **Categories:** Social media, Marketing, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.56 / 1,000 topic scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Discourse Forum Scraper: Topics, Posts & Users

![Discourse Forum Scraper: Topics with categories, tags, views, replies, likes and posters, replies as clean plain text when you want them](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/discourse-forum-scraper.jpg)

**Discourse Forum Scraper** exports topics from any **Discourse**-powered community, title, category, tags, view count, reply count, like count, timestamps, the list of people who posted, the excerpt, and optionally every reply body as clean plain text.

Discourse publishes a structured representation of every page it renders, and reading it is a deliberate platform feature rather than a loophole. The Actor needs nothing configured to reach it: no browser, no proxy, no login.

**Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/discourse-forum-scraper.md)**

```
https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/discourse-forum-scraper.md
```

### What you get

| Output field | Meaning |
|---|---|
| `forum` | The normalised forum host the topic came from |
| `topicId`, `slug`, `url` | Discourse topic ID, URL slug and the direct link |
| `title` | Topic title |
| `categoryId`, `categoryName` | Category ID and its resolved human name, sub-categories included |
| `tags` | Every tag on the topic |
| `postsCount`, `replyCount` | Total posts, and replies excluding the opening post |
| `views`, `likeCount` | Engagement metrics as Discourse reports them |
| `createdAt`, `lastPostedAt` | Opened and last-activity timestamps |
| `posters` | Everyone who posted, with username, display name and their role in the thread |
| `excerpt` | Discourse's own summary of the opening post, HTML stripped |
| `posts` | Every reply, id, number, author, timestamps, reply count, reads, score and plain text, only when `includePosts` is on |
| `scrapedAt` | Run timestamp |

A `RUN_SUMMARY` record in the key-value store holds per-run counts, the filters used, and any forum or category that failed.

### Common use cases

**Mine product communities for feature requests**, pull the feature category from a forum and rank by `likeCount` to find what users actually want.

```json
{
 "forums": ["forum.obsidian.md"],
 "categories": ["feature-requests"],
 "sortBy": "top",
 "maxTopicsPerForum": 500
}
```

**Track sentiment in open-source forums**, take the newest topics with full bodies and feed them to whatever classifier you already run.

```json
{
 "forums": ["discuss.python.org", "meta.discourse.org"],
 "sortBy": "new",
 "includePosts": true,
 "maxTopicsPerForum": 100
}
```

**Build a support knowledge base**, the support category, most-viewed first, with every reply in plain text is a ready-made retrieval corpus.

```json
{
 "forums": ["meta.discourse.org"],
 "categories": ["support"],
 "sortBy": "top",
 "includePosts": true,
 "maxTopicsPerForum": 1000
}
```

### Quick start

One forum, latest 100 topics:

```json
{
 "forums": ["meta.discourse.org"]
}
```

Several forums, newest topics, capped and cheap:

```json
{
 "forums": ["meta.discourse.org", "https://discuss.python.org/", "forum.obsidian.md"],
 "sortBy": "new",
 "maxTopicsPerForum": 50
}
```

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `forums` | array | - | **Required.** Forum addresses. `https://`, `www.` and trailing paths or slashes are all stripped. |
| `categories` | array | `[]` | Category slugs. Empty = the whole forum. |
| `sortBy` | string | `latest` | `latest` (bumped by activity), `top` (all-time most engaged) or `new` (newest first). |
| `includePosts` | boolean | `false` | Fetch every reply body. One extra request per topic. |
| `maxTopicsPerForum` | integer | `100` | Cap per forum, applied across all its categories. `0` = no limit. |

**Which combinations make sense.** `categories` changes *which route is crawled*, not just what is kept, each slug is resolved to its Discourse category ID and that category's own topic list is paged directly. That matters on large forums, where filtering the global feed would need dozens of pages to surface a quiet category. `maxTopicsPerForum` is shared across a forum's categories and consumed in the order you list them.

`sortBy: "top"` with a generous cap gives you the best evergreen content; `sortBy: "new"` with a small cap is the right shape for a schedule that only wants what appeared since last run. `includePosts` multiplies both run time and cost of time, turn it on for corpus building, off for monitoring.

### Output example

```json
{
 "forum": "meta.discourse.org",
 "topicId": 1,
 "title": "New to Discourse? Start here!",
 "slug": "new-to-discourse-start-here",
 "url": "https://meta.discourse.org/t/new-to-discourse-start-here/1",
 "categoryId": 67,
 "categoryName": "Announcements",
 "tags": [],
 "postsCount": 1,
 "replyCount": 5,
 "views": 117628,
 "likeCount": 435,
 "createdAt": "2013-02-01T04:52:28.073Z",
 "lastPostedAt": "2013-02-01T04:52:28.397Z",
 "posters": [
 { "username": "Discourse", "name": null, "role": "Original Poster, Most Recent Poster" }
 ],
 "excerpt": "We're so glad you're here! This is our official community, where you can connect, ask questions, and share ideas about Discourse…",
 "posts": [
 {
 "postId": 2019829,
 "postNumber": 1,
 "username": "Discourse",
 "name": null,
 "createdAt": "2013-02-01T04:52:28.073Z",
 "updatedAt": "2025-09-23T19:13:50.543Z",
 "replyCount": 0,
 "reads": 16,
 "score": 23,
 "text": "We're so glad you're here! …"
 }
 ],
 "scrapedAt": "2026-08-06T11:39:37.084Z"
}
```

`RUN_SUMMARY` looks like this:

```json
{
 "forumsRequested": 4,
 "forumsFailed": 2,
 "failures": [
 { "forum": "apify.com", "kind": "forum", "error": "not found (404)" },
 { "forum": "community.cloudflare.com", "kind": "forum", "error": "access denied (403), this forum blocks anonymous JSON reads" }
 ],
 "topicsSaved": 80,
 "postsSaved": 0,
 "filters": {
 "forums": ["meta.discourse.org", "discuss.python.org", "apify.com", "community.cloudflare.com"],
 "categories": [],
 "sortBy": "latest",
 "includePosts": false,
 "maxTopicsPerForum": 40
 },
 "finishedAt": "2026-08-06T11:39:39.384Z"
}
```

`kind` distinguishes a forum that could not be reached (`forum`), a category slug that does not exist there (`category`), and a topic list that broke mid-crawl (`list`).

### Finding a forum and a category slug

Any Discourse site works. To confirm one is Discourse before you run, open `https://THAT-HOST/about.json` in a browser, a Discourse forum answers with JSON containing a version like `"2026.8.0-latest.1"`. Anything else is not Discourse, and the Actor will report it exactly that way.

| URL you see | What to pass |
|---|---|
| `https://meta.discourse.org/` | `forums: ["meta.discourse.org"]` |
| `https://meta.discourse.org/c/support/6` | `forums: ["meta.discourse.org"], categories: ["support"]` |
| `https://forum.obsidian.md/c/feature-requests/6` | `forums: ["forum.obsidian.md"], categories: ["feature-requests"]` |

Known-good public examples: `meta.discourse.org`, `discuss.python.org`, `forum.obsidian.md`, `community.openai.com`, `forums.docker.com`.

### Limits and behaviour

- **Every forum is verified first.** Before any crawling, the Actor fetches `/about.json` and requires a Discourse version string. A host that answers 404, returns HTML instead of JSON, or serves JSON without that version is recorded in `RUN_SUMMARY.failures` as *not a Discourse forum* and skipped. That one probe is the difference between "no topics" and "wrong site".
- **Some Discourse forums block anonymous reads.** A few operators put Cloudflare or a login requirement in front of everything, `community.cloudflare.com` is one. Those return HTTP 403 and are reported as such; no proxy or workaround is attempted.
- **`new` is not `/new.json`.** Discourse's `/new.json` means "new *to you*" and requires a session, so it 403s for anonymous clients. `sortBy: "new"` therefore uses the latest list re-ordered by creation date, which is what an anonymous visitor sees under "New".
- **Pagination follows Discourse's own signal.** Lists are paged with `?page=N` from 0 and stop when the forum stops advertising `more_topics_url`, never truncated silently. The per-forum log line reports listed and saved counts.
- **Requests are throttled.** 300 ms between every request, well inside Discourse's default anonymous per-IP limits, so a long crawl does not start collecting 429s. If one appears anyway it is retried with a longer backoff.
- **Post bodies are cleaned, not raw.** Discourse returns `cooked` HTML; entities are decoded, block tags become line breaks, list items become bullets, and image `alt` text is preserved. You get readable text, not markup.
- **Category names are resolved once per forum.** A single `/categories.json` call builds the ID-to-name map, sub-categories included, so `categoryName` is populated without a lookup per topic. If that call fails the crawl still runs and `categoryName` is `null`.
- **One bad forum never kills the run.** The Actor only throws when *every* forum failed.
- **Public data only.** Anonymous, publicly readable topics. No login, no private categories, no personal messages, no access-control bypass.

### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/arman-bd~discourse-forum-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "forums": ["meta.discourse.org", "discuss.python.org"],
 "sortBy": "top",
 "maxTopicsPerForum": 50
 }'
```

### JavaScript example

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/discourse-forum-scraper').call({
 forums: ['forum.obsidian.md'],
 categories: ['feature-requests'],
 sortBy: 'top',
 maxTopicsPerForum: 200,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const t of items) console.log(`${t.likeCount}\t${t.views}\t${t.title}`);
```

**Defaults:** 1 GB memory, 15 minute timeout.

### FAQ

**Do I need a proxy?** No. Proxy configuration is not required to run this Actor.

**Do I need an account on the forum?** No. You supply no credentials.

**What happens if a source is unavailable?** It is logged, recorded in `RUN_SUMMARY.failures`, and the run continues with the remaining forums. The Actor only errors out when every forum fails.

**Can I schedule it?** Yes, it is designed for it. Run `sortBy: "new"` daily with a small cap and diff on `topicId` to get only newly opened threads.

**How do I know a site is really Discourse?** Open `https://THAT-HOST/about.json`. JSON with a version number means yes. The Actor performs the same check itself and reports anything that fails it.

**Why did a forum return 403?** Its operator restricts anonymous access, usually via Cloudflare or a login-required setting. Nothing on this side can change that, and the Actor does not try.

**Does it fetch comments?** Yes, with `includePosts: true`, every post in the thread, in order, as plain text. Without it you still get `postsCount`, `replyCount` and the `posters` list.

**Why is `categoryName` sometimes null?** Only if the forum's `/categories.json` was unavailable during that run. `categoryId` is always present, so you can resolve names yourself later.

**Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

# Actor input Schema

## `forums` (type: `array`):

Forum addresses. 'meta.discourse.org' or 'https://discuss.python.org/'. Scheme, 'www.' and any path are stripped automatically. To check a site is Discourse, open https://THAT-HOST/about.json in a browser: a Discourse forum returns JSON with a version number.

## `categories` (type: `array`):

Restrict the crawl to these categories. The slug is the text after /c/ in a category URL. for https://meta.discourse.org/c/support/6 the slug is 'support'. Sub-categories are matched too. Leave empty to take the whole forum. Slugs that do not exist on a forum are reported, not fatal.

## `sortBy` (type: `string`):

'latest' is the default bumped-by-activity feed. 'top' is the forum's all-time most engaged topics. 'new' is the same feed ordered by creation date. most recently opened first.

## `includePosts` (type: `boolean`):

Fetch every reply inside each topic as clean plain text. Costs one extra request per topic and makes the run several times slower, so leave it off when you only need topic-level metrics.

## `maxTopicsPerForum` (type: `integer`):

Cap the topics saved per forum. Discourse serves 30 topics per page, so 100 costs four requests. Set 0 to page through everything the forum will serve.

## Actor input object example

```json
{
  "forums": [
    "forum.obsidian.md",
    "https://community.openai.com"
  ],
  "categories": [
    "support",
    "feature"
  ],
  "sortBy": "latest",
  "includePosts": false,
  "maxTopicsPerForum": 100
}
```

# Actor output Schema

## `items` (type: `string`):

Every record the run produced.

## `runsummary` (type: `string`):

The RUN_SUMMARY record from the run's key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "forums": [
        "meta.discourse.org",
        "discuss.python.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arman-bd/discourse-forum-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "forums": [
        "meta.discourse.org",
        "discuss.python.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("arman-bd/discourse-forum-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "forums": [
    "meta.discourse.org",
    "discuss.python.org"
  ]
}' |
apify call arman-bd/discourse-forum-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/discourse-forum-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gnxNdFCW719fc4fdr/builds/cJo8TI4MlMtKmegxt/openapi.json