Discourse Community Scraper

What does Discourse Community Scraper do?

Discourse Community Scraper extracts topics and posts from any Discourse-powered community forum — no per-site configuration needed. It works out of the box on community.n8n.io, community.smartthings.com, forums.developer.nvidia.com, and thousands of other Discourse instances (Discourse powers most modern developer/product community forums). Instead of parsing HTML, it talks directly to Discourse's built-in JSON API (/latest.json, /c/{category}.json, /search.json, /t/{id}.json, ...), which every Discourse site exposes — making the scraper fast, resilient to theme/layout changes, and generic across sites.

Run it on the Apify platform for scheduling, API access, webhooks, and proxy rotation without managing any infrastructure yourself.

Why use Discourse Community Scraper?

• Community research & sentiment analysis — pull recurring questions, complaints, and feature requests from a product's support forum.
• Competitive intelligence — monitor what users say about a competitor's product in its official community.
• Support/knowledge-base mining — build a searchable dataset of solved threads and accepted answers.
• Content/dataset building — collect real-world Q&A pairs or discussion threads for training or analysis.

How to use Discourse Community Scraper

1. Set Forum URL to the base URL of any Discourse forum (e.g. https://community.n8n.io).
2. Pick a Mode:
   • categories — list every category/section on the forum, so you can find the right categorySlug without guessing (run this first if you don't already know it).
   • latest / top — the site-wide topic list.
   • category — one category (set Category slug/path to a category item's category_slug_for_scraping from a categories run, e.g. questions/12).
   • search — Discourse's search endpoint (set Search query; supports Discourse search filters like order:latest #category).
   • topicUrls — scrape only the specific topic URLs you provide.
3. Toggle Include full posts to also fetch every reply in each topic (off = topic metadata only, much faster).
4. Set Max topics / Max posts per topic to bound the run, and hit Start.

Input

See the Input tab for the full schema. Key fields:

Field	Description
forumUrl	Base URL of the Discourse forum
mode	categories, latest, top, category, search, or topicUrls
categorySlug	Category slug/path (mode=category)
searchQuery	Search query (mode=search)
topicUrls	List of direct topic URLs (mode=topicUrls)
includePosts	Fetch full post stream per topic
maxTopics / maxPostsPerTopic	Limits (0 = unlimited)
requestDelaySecs / maxConcurrency	Politeness/rate-limit controls
proxyConfiguration	Apify Proxy settings

Output

Up to three item shapes land in the same dataset, distinguished by type:

{
  "type": "category",
  "forum": "https://community.n8n.io",
  "category_id": 13,
  "title": "Jobs",
  "slug": "jobs",
  "url": "https://community.n8n.io/c/jobs/13",
  "parent_category_id": null,
  "topic_count": 615,
  "category_slug_for_scraping": "jobs/13"
}

{
  "type": "topic",
  "forum": "https://community.n8n.io",
  "topic_id": 302663,
  "title": "AI Assistant on self-hosted n8n: early setup instructions",
  "slug": "ai-assistant-on-self-hosted-n8n-early-setup-instructions",
  "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663",
  "posts_count": 4,
  "views": 512,
  "like_count": 3,
  "created_at": "2026-01-05T10:00:00.000Z"
}

{
  "type": "post",
  "topic_id": 302663,
  "post_id": 565721,
  "post_number": 1,
  "username": "Ophir_Prusak",
  "cooked_html": "<p>Here's how to set it up:</p><ol><li>Install the <strong>self-hosted</strong> package</li><li>Restart n8n</li></ol>",
  "content_markdown": "Here's how to set it up:\n\n1. Install the **self-hosted** package\n2. Restart n8n",
  "content_text": "Here's how to set it up: Install the self-hosted package Restart n8n",
  "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663/1"
}

cooked_html is Discourse's raw rendered post body. content_markdown and content_text are cleaned conversions of it (via markdownify and BeautifulSoup) — use these for LLM input, search indexing, or anywhere raw HTML is unwanted.

Download the dataset as JSON, CSV, Excel, or HTML from the Storage tab.

Data table

Field	Item type	Description
title / topic_title	topic / post	Topic title
url	both	Direct link to the topic or post
username	post	Author's forum username
cooked_html	post	Rendered HTML of the post body
content_markdown	post	Post body as clean Markdown
content_text	post	Post body as clean plain text
posts_count	topic	Number of replies in the topic
like_count	both	Like count
created_at	both	ISO timestamp

Example: market research from a community's Jobs category

You can run this actor end-to-end through the Apify MCP server to answer research questions without writing scraping code. Example: "What kind of freelance work do n8n users want to buy in 2026?"

1. Discover the category — call the actor with mode: "categories" on https://community.n8n.io. This lists every category with a ready-to-use category_slug_for_scraping, e.g. "jobs/13" for the Jobs category.
2. Scrape it with posts — call the actor again with mode: "category", categorySlug: "jobs/13", includePosts: true, maxTopics: 80. This returns each job-post topic plus its replies, with content_text already cleaned of HTML.
3. Analyze — pull content_text across all post items and count keyword mentions (e.g. "AI agent", "CRM", "voice AI", "RAG", "self-hosted") to see what buyers are actually asking for, and compare "hiring"/"looking for" vs. "for hire"/"available" topic titles to gauge demand vs. supply.

A real run against community.n8n.io/c/jobs (61 topics, 204 rows) found: AI agent building was the top request (72 mentions), usually combined with CRM integration (92), document/RAG/OCR processing (60/38/22), and a growing voice AI niche (Vapi + Twilio + n8n stacks, 31+23 mentions). Buy-side posts outnumbered for-hire posts roughly 2:1, and self-hosted n8n (21 mentions) was a common hard requirement. Rates ranged $15–50/hr or $250–1,500 flat, with a shift toward long-term retainers over one-off builds.

This same discover → scrape → analyze flow works for any Discourse forum's category (support backlogs, feature requests, showcase/built-with sections, etc.), not just job boards.

Cost estimation

This actor only makes lightweight JSON API requests (no browser rendering), so it's cheap to run — typically well under 1 compute unit for a few hundred topics without full posts. Enabling includePosts roughly doubles request volume per topic (one extra request per ~20 posts).

Tips

• Leave includePosts off for a quick topic-index crawl, then re-run with topicUrls on just the topics you care about.
• Lower maxConcurrency / raise requestDelaySecs on smaller or self-hosted forums to stay polite.
• Enable Apify Proxy for large crawls to avoid rate limiting.

FAQ & disclaimers

Only scrape publicly accessible forum content, and respect each forum's Terms of Service and robots.txt. This actor is not affiliated with Discourse, n8n, SmartThings, or NVIDIA. Found a Discourse site this doesn't work on? Open an issue in the Issues tab.