Discourse Community Scraper

Extract topics, discussions, and posts from any public Discourse forum. Supports latest topics, trending discussions, category browsing, tag filtering, full-text search, user activity, and complete post threads — all via the standard Discourse REST API.

Works with meta.discourse.org, community.cloudflare.com, and any self-hosted Discourse instance.

What It Does

• Latest topics — fetch the most recently active discussions
• Top/trending topics — get the most popular topics by time period (daily/weekly/monthly/yearly)
• Browse by category — explore topics within a specific forum category
• Filter by tag — find all topics with a given tag
• Full-text search — search across all forum topics by keyword
• User topics — get all topics created by a specific user
• Get posts — fetch all posts/replies within a specific topic thread

Input Schema

Field	Type	Description	Default
forumUrl	text (required)	Base URL of the Discourse forum	—
mode	select	Scraping mode (see modes below)	latestTopics
period	select	Time period for top topics: daily, weekly, monthly, quarterly, yearly, all	weekly
categorySlug	text	Category slug or path (e.g., support, meta/7)	—
tag	text	Tag name to filter by (e.g., bug, feature-request)	—
searchQuery	text	Search query for finding topics	getting started
username	text	Forum username for byUser mode	—
topicId	integer	Numeric topic ID for getPosts mode	—
apiKey	text (secret)	API key for private forums (optional)	—
apiUsername	text	Username associated with the API key	—
maxItems	integer	Maximum records to return (1–5000)	50

Modes

Mode	Description	Required Fields
latestTopics	Most recently active topics	forumUrl
topTopics	Top topics by engagement	forumUrl, period
byCategory	Topics within a category	forumUrl, categorySlug
byTag	Topics with a specific tag	forumUrl, tag
searchTopics	Full-text search results	forumUrl, searchQuery
byUser	Topics created by a user	forumUrl, username
getPosts	All posts in a topic thread	forumUrl, topicId

Output Schema

Topic Record (recordType: "discourseTopic")

Field	Type	Description
topicId	integer	Unique topic ID
title	string	Topic title
slug	string	URL-friendly topic slug
url	string	Full URL to the topic
postsCount	integer	Total number of posts in the topic
views	integer	Number of views
likeCount	integer	Total likes received
replyCount	integer	Number of replies
createdAt	string	ISO 8601 creation timestamp
lastPostedAt	string	ISO 8601 timestamp of most recent post
categoryId	integer	Category ID
categoryName	string	Human-readable category name
tags	array	List of tag names
authorUsername	string	Username of the original poster
authorName	string	Display name of the original poster
excerpt	string	Short excerpt from the first post
solved	boolean	Whether the topic has an accepted answer
pinned	boolean	Whether the topic is pinned
highestPostNumber	integer	Highest post number in the thread
wordCount	integer	Total word count of the topic
forumUrl	string	Base URL of the source forum
recordType	string	Always discourseTopic
scrapedAt	string	ISO 8601 scrape timestamp

Post Record (recordType: "discoursePost")

Field	Type	Description
postId	integer	Unique post ID
topicId	integer	Parent topic ID
postNumber	integer	Position in the thread
url	string	Direct URL to the post
authorUsername	string	Post author's username
authorName	string	Post author's display name
createdAt	string	ISO 8601 creation timestamp
updatedAt	string	ISO 8601 last update timestamp
cooked	string	Rendered HTML content
raw	string	Raw markdown content
likeCount	integer	Number of likes on this post
replyCount	integer	Number of replies to this post
isAcceptedAnswer	boolean	Whether this post is the accepted solution
recordType	string	Always discoursePost
scrapedAt	string	ISO 8601 scrape timestamp

Example Output

Topic

{
  "topicId": 12345,
  "title": "How to set up email notifications?",
  "slug": "how-to-set-up-email-notifications",
  "url": "https://meta.discourse.org/t/how-to-set-up-email-notifications/12345",
  "postsCount": 8,
  "views": 1250,
  "likeCount": 23,
  "replyCount": 7,
  "createdAt": "2024-03-15T09:30:00.000Z",
  "categoryName": "Support",
  "tags": ["email", "notifications", "configuration"],
  "authorUsername": "john_doe",
  "solved": true,
  "forumUrl": "https://meta.discourse.org",
  "recordType": "discourseTopic",
  "scrapedAt": "2026-05-15T10:00:00+00:00"
}

Post

{
  "postId": 56789,
  "topicId": 12345,
  "postNumber": 3,
  "url": "https://meta.discourse.org/t/how-to-set-up-email-notifications/12345/3",
  "authorUsername": "admin_user",
  "cooked": "<p>Go to Admin → Settings → Email and configure your SMTP server.</p>",
  "likeCount": 15,
  "isAcceptedAnswer": true,
  "recordType": "discoursePost",
  "scrapedAt": "2026-05-15T10:00:00+00:00"
}

FAQ

Does it work with any Discourse forum? Yes. The scraper uses the standard Discourse REST API (all .json endpoints) that is available on every Discourse instance — both hosted and self-hosted. Simply provide the forum's base URL.

Do I need an API key? No API key is needed for public forums. The apiKey and apiUsername fields are only required if the forum restricts access to logged-in users.

How do I find the category slug? Look at the forum URL when browsing a category. For example, https://meta.discourse.org/c/support/6 uses the slug support or the path support/6.

How many posts can I get from a topic? Using mode=getPosts, the scraper fetches all posts in a topic, paginating through the full thread. Set maxItems to control the upper limit.

Can I scrape a private Discourse forum? Yes, if you have valid API credentials. Provide your API key in the apiKey field and your username in apiUsername. The forum administrator generates API keys in the admin panel.

What's the difference between postsCount and replyCount? postsCount is the total number of posts in the thread (including the original post). replyCount counts only replies made to other posts within the thread.

How does the solved/accepted answer work? Discourse has an optional "Solved" plugin that lets users mark a reply as the accepted answer. If a forum uses this plugin, solved=true appears on topics that have an accepted answer, and isAcceptedAnswer=true appears on the specific accepted post.

Can I search across multiple forums in one run? Each run targets one forum URL. To scrape multiple forums, run the actor multiple times with different forumUrl values.