Discourse Forum Scraper: Topics, Posts & Users

Discourse Forum Scraper exports topics from any Discourse-powered community, title, category, tags, view count, reply count, like count, timestamps, the list of people who posted, the excerpt, and optionally every reply body as clean plain text.

Discourse publishes a structured representation of every page it renders, and reading it is a deliberate platform feature rather than a loophole. The Actor needs nothing configured to reach it: no browser, no proxy, no login.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/discourse-forum-scraper.md

What you get

Output field	Meaning
forum	The normalised forum host the topic came from
topicId, slug, url	Discourse topic ID, URL slug and the direct link
title	Topic title
categoryId, categoryName	Category ID and its resolved human name, sub-categories included
tags	Every tag on the topic
postsCount, replyCount	Total posts, and replies excluding the opening post
views, likeCount	Engagement metrics as Discourse reports them
createdAt, lastPostedAt	Opened and last-activity timestamps
posters	Everyone who posted, with username, display name and their role in the thread
excerpt	Discourse's own summary of the opening post, HTML stripped
posts	Every reply, id, number, author, timestamps, reply count, reads, score and plain text, only when includePosts is on
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds per-run counts, the filters used, and any forum or category that failed.

Common use cases

Mine product communities for feature requests, pull the feature category from a forum and rank by likeCount to find what users actually want.

{
 "forums": ["forum.obsidian.md"],
 "categories": ["feature-requests"],
 "sortBy": "top",
 "maxTopicsPerForum": 500
}

Track sentiment in open-source forums, take the newest topics with full bodies and feed them to whatever classifier you already run.

{
 "forums": ["discuss.python.org", "meta.discourse.org"],
 "sortBy": "new",
 "includePosts": true,
 "maxTopicsPerForum": 100
}

Build a support knowledge base, the support category, most-viewed first, with every reply in plain text is a ready-made retrieval corpus.

{
 "forums": ["meta.discourse.org"],
 "categories": ["support"],
 "sortBy": "top",
 "includePosts": true,
 "maxTopicsPerForum": 1000
}

Quick start

One forum, latest 100 topics:

{
 "forums": ["meta.discourse.org"]
}

Several forums, newest topics, capped and cheap:

{
 "forums": ["meta.discourse.org", "https://discuss.python.org/", "forum.obsidian.md"],
 "sortBy": "new",
 "maxTopicsPerForum": 50
}

Input

Field	Type	Default	Notes
forums	array	-	Required. Forum addresses. https://, www. and trailing paths or slashes are all stripped.
categories	array	[]	Category slugs. Empty = the whole forum.
sortBy	string	latest	latest (bumped by activity), top (all-time most engaged) or new (newest first).
includePosts	boolean	false	Fetch every reply body. One extra request per topic.
maxTopicsPerForum	integer	100	Cap per forum, applied across all its categories. 0 = no limit.

Which combinations make sense. categories changes which route is crawled, not just what is kept, each slug is resolved to its Discourse category ID and that category's own topic list is paged directly. That matters on large forums, where filtering the global feed would need dozens of pages to surface a quiet category. maxTopicsPerForum is shared across a forum's categories and consumed in the order you list them.

sortBy: "top" with a generous cap gives you the best evergreen content; sortBy: "new" with a small cap is the right shape for a schedule that only wants what appeared since last run. includePosts multiplies both run time and cost of time, turn it on for corpus building, off for monitoring.

Output example

{
 "forum": "meta.discourse.org",
 "topicId": 1,
 "title": "New to Discourse? Start here!",
 "slug": "new-to-discourse-start-here",
 "url": "https://meta.discourse.org/t/new-to-discourse-start-here/1",
 "categoryId": 67,
 "categoryName": "Announcements",
 "tags": [],
 "postsCount": 1,
 "replyCount": 5,
 "views": 117628,
 "likeCount": 435,
 "createdAt": "2013-02-01T04:52:28.073Z",
 "lastPostedAt": "2013-02-01T04:52:28.397Z",
 "posters": [
 { "username": "Discourse", "name": null, "role": "Original Poster, Most Recent Poster" }
 ],
 "excerpt": "We're so glad you're here! This is our official community, where you can connect, ask questions, and share ideas about Discourse…",
 "posts": [
 {
 "postId": 2019829,
 "postNumber": 1,
 "username": "Discourse",
 "name": null,
 "createdAt": "2013-02-01T04:52:28.073Z",
 "updatedAt": "2025-09-23T19:13:50.543Z",
 "replyCount": 0,
 "reads": 16,
 "score": 23,
 "text": "We're so glad you're here! …"
 }
 ],
 "scrapedAt": "2026-08-06T11:39:37.084Z"
}

RUN_SUMMARY looks like this:

{
 "forumsRequested": 4,
 "forumsFailed": 2,
 "failures": [
 { "forum": "apify.com", "kind": "forum", "error": "not found (404)" },
 { "forum": "community.cloudflare.com", "kind": "forum", "error": "access denied (403), this forum blocks anonymous JSON reads" }
 ],
 "topicsSaved": 80,
 "postsSaved": 0,
 "filters": {
 "forums": ["meta.discourse.org", "discuss.python.org", "apify.com", "community.cloudflare.com"],
 "categories": [],
 "sortBy": "latest",
 "includePosts": false,
 "maxTopicsPerForum": 40
 },
 "finishedAt": "2026-08-06T11:39:39.384Z"
}

kind distinguishes a forum that could not be reached (forum), a category slug that does not exist there (category), and a topic list that broke mid-crawl (list).

Finding a forum and a category slug

Any Discourse site works. To confirm one is Discourse before you run, open https://THAT-HOST/about.json in a browser, a Discourse forum answers with JSON containing a version like "2026.8.0-latest.1". Anything else is not Discourse, and the Actor will report it exactly that way.

URL you see	What to pass
https://meta.discourse.org/	forums: ["meta.discourse.org"]
https://meta.discourse.org/c/support/6	forums: ["meta.discourse.org"], categories: ["support"]
https://forum.obsidian.md/c/feature-requests/6	forums: ["forum.obsidian.md"], categories: ["feature-requests"]

Known-good public examples: meta.discourse.org, discuss.python.org, forum.obsidian.md, community.openai.com, forums.docker.com.

Limits and behaviour

• Every forum is verified first. Before any crawling, the Actor fetches /about.json and requires a Discourse version string. A host that answers 404, returns HTML instead of JSON, or serves JSON without that version is recorded in RUN_SUMMARY.failures as not a Discourse forum and skipped. That one probe is the difference between "no topics" and "wrong site".
• Some Discourse forums block anonymous reads. A few operators put Cloudflare or a login requirement in front of everything, community.cloudflare.com is one. Those return HTTP 403 and are reported as such; no proxy or workaround is attempted.
• new is not /new.json. Discourse's /new.json means "new to you" and requires a session, so it 403s for anonymous clients. sortBy: "new" therefore uses the latest list re-ordered by creation date, which is what an anonymous visitor sees under "New".
• Pagination follows Discourse's own signal. Lists are paged with ?page=N from 0 and stop when the forum stops advertising more_topics_url, never truncated silently. The per-forum log line reports listed and saved counts.
• Requests are throttled. 300 ms between every request, well inside Discourse's default anonymous per-IP limits, so a long crawl does not start collecting 429s. If one appears anyway it is retried with a longer backoff.
• Post bodies are cleaned, not raw. Discourse returns cooked HTML; entities are decoded, block tags become line breaks, list items become bullets, and image alt text is preserved. You get readable text, not markup.
• Category names are resolved once per forum. A single /categories.json call builds the ID-to-name map, sub-categories included, so categoryName is populated without a lookup per topic. If that call fails the crawl still runs and categoryName is null.
• One bad forum never kills the run. The Actor only throws when every forum failed.
• Public data only. Anonymous, publicly readable topics. No login, no private categories, no personal messages, no access-control bypass.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~discourse-forum-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "forums": ["meta.discourse.org", "discuss.python.org"],
 "sortBy": "top",
 "maxTopicsPerForum": 50
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/discourse-forum-scraper').call({
 forums: ['forum.obsidian.md'],
 categories: ['feature-requests'],
 sortBy: 'top',
 maxTopicsPerForum: 200,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const t of items) console.log(`${t.likeCount}\t${t.views}\t${t.title}`);

Defaults: 1 GB memory, 15 minute timeout.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need an account on the forum? No. You supply no credentials.

What happens if a source is unavailable? It is logged, recorded in RUN_SUMMARY.failures, and the run continues with the remaining forums. The Actor only errors out when every forum fails.

Can I schedule it? Yes, it is designed for it. Run sortBy: "new" daily with a small cap and diff on topicId to get only newly opened threads.

How do I know a site is really Discourse? Open https://THAT-HOST/about.json. JSON with a version number means yes. The Actor performs the same check itself and reports anything that fails it.

Why did a forum return 403? Its operator restricts anonymous access, usually via Cloudflare or a login-required setting. Nothing on this side can change that, and the Actor does not try.

Does it fetch comments? Yes, with includePosts: true, every post in the thread, in order, as plain text. Without it you still get postsCount, replyCount and the posters list.

Why is categoryName sometimes null? Only if the forum's /categories.json was unavailable during that run. categoryId is always present, so you can resolve names yourself later.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.