Discourse Forum Scraper

Extract every topic and full post text from any Discourse forum, as clean plain text. Built for LLM training data, RAG corpora, and community research.

Discourse powers thousands of developer, product and support communities: Discourse Meta, Rust, Elixir, Home Assistant, OpenAI, Hugging Face, Figma, McNeel, Ubuntu, and countless niche forums. This Actor works on all of them without configuration. Just paste the forum URL.

What you get

Field	Description
topicId	Discourse topic ID
title	Topic title
url	Direct link to the thread
category	Category ID
tags	Topic tags
createdAt	When the thread started
lastPostedAt	Most recent reply
replyCount	Number of replies
postCount	Total posts
views	View count
likeCount	Likes on the thread
pinned / closed	Thread state
posts[]	Every post: author, date, plain text, likes, post number
fullText	The whole thread as one clean text block, ready for embedding
forum	Source forum
scrapedAt	ISO timestamp

Why fullText matters

Discourse returns posts as HTML. Feeding that to a model wastes tokens on markup. This Actor strips it to plain text and joins the thread into a single fullText field, so each record drops straight into a vector store or fine-tuning set with no preprocessing.

It also handles Discourse's post pagination. Discourse only returns the first chunk of posts in a topic payload, so most scrapers silently truncate long threads. This one fetches the remaining posts and returns the complete conversation.

Three modes

• Latest topics — the forum's recent activity
• A category — pass a category slug like feature or support/33
• Search results — pass any Discourse search query

Example input

{
  "forumUrl": "https://meta.discourse.org",
  "mode": "latest",
  "maxTopics": 500,
  "includePosts": true,
  "maxPostsPerTopic": 100
}

Example output

{
  "topicId": 1,
  "title": "New to Discourse? Start here!",
  "url": "https://meta.discourse.org/t/new-to-discourse-start-here/1",
  "replyCount": 5,
  "views": 118812,
  "posts": [
    {
      "postNumber": 1,
      "author": "Discourse",
      "createdAt": "2023-05-15T10:00:00.000Z",
      "text": "We're so glad you're here! This is our official community...",
      "likes": 42,
      "isOriginalPost": true
    }
  ],
  "fullText": "Discourse: We're so glad you're here!...",
  "scrapedAt": "2026-09-09T19:19:33.000Z"
}

Cost control

maxTopics and maxPostsPerTopic cap every run. You are charged per topic returned, so a run's cost is knowable before you start it.

Notes

• Uses only public, unauthenticated Discourse JSON endpoints. No login required, no private content accessed.
• Rate-limited and retried politely, with backoff on HTTP 429.
• Works on self-hosted Discourse instances as well as hosted ones.