# Discourse Community Scraper (`rl1987/discourse-community-scraper`) Actor Generalised scraper for any Discourse-based community forum (topics, posts, categories, search) via Discourse's JSON API. - **URL**: https://apify.com/rl1987/discourse-community-scraper.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 dataset items This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### What does Discourse Community Scraper do? **Discourse Community Scraper** extracts topics and posts from any **Discourse-powered community forum** — no per-site configuration needed. It works out of the box on [community.n8n.io](https://community.n8n.io), [community.smartthings.com](https://community.smartthings.com), [forums.developer.nvidia.com](https://forums.developer.nvidia.com), and thousands of other Discourse instances (Discourse powers most modern developer/product community forums). Instead of parsing HTML, it talks directly to Discourse's built-in JSON API (`/latest.json`, `/c/{category}.json`, `/search.json`, `/t/{id}.json`, ...), which every Discourse site exposes — making the scraper fast, resilient to theme/layout changes, and generic across sites. Run it on the [Apify platform](https://apify.com) for scheduling, API access, webhooks, and proxy rotation without managing any infrastructure yourself. ### Why use Discourse Community Scraper? - **Community research & sentiment analysis** — pull recurring questions, complaints, and feature requests from a product's support forum. - **Competitive intelligence** — monitor what users say about a competitor's product in its official community. - **Support/knowledge-base mining** — build a searchable dataset of solved threads and accepted answers. - **Content/dataset building** — collect real-world Q&A pairs or discussion threads for training or analysis. ### How to use Discourse Community Scraper 1. Set **Forum URL** to the base URL of any Discourse forum (e.g. `https://community.n8n.io`). 2. Pick a **Mode**: - `categories` — list every category/section on the forum, so you can find the right `categorySlug` without guessing (run this first if you don't already know it). - `latest` / `top` — the site-wide topic list. - `category` — one category (set **Category slug/path** to a category item's `category_slug_for_scraping` from a `categories` run, e.g. `questions/12`). - `search` — Discourse's search endpoint (set **Search query**; supports Discourse search filters like `order:latest #category`). - `topicUrls` — scrape only the specific topic URLs you provide. 3. Toggle **Include full posts** to also fetch every reply in each topic (off = topic metadata only, much faster). 4. Set **Max topics** / **Max posts per topic** to bound the run, and hit **Start**. ### Input See the **Input** tab for the full schema. Key fields: | Field | Description | |---|---| | `forumUrl` | Base URL of the Discourse forum | | `mode` | `categories`, `latest`, `top`, `category`, `search`, or `topicUrls` | | `categorySlug` | Category slug/path (mode=`category`) | | `searchQuery` | Search query (mode=`search`) | | `topicUrls` | List of direct topic URLs (mode=`topicUrls`) | | `includePosts` | Fetch full post stream per topic | | `maxTopics` / `maxPostsPerTopic` | Limits (0 = unlimited) | | `requestDelaySecs` / `maxConcurrency` | Politeness/rate-limit controls | | `proxyConfiguration` | Apify Proxy settings | ### Output Up to three item shapes land in the same dataset, distinguished by `type`: ```json { "type": "category", "forum": "https://community.n8n.io", "category_id": 13, "title": "Jobs", "slug": "jobs", "url": "https://community.n8n.io/c/jobs/13", "parent_category_id": null, "topic_count": 615, "category_slug_for_scraping": "jobs/13" } ```` ```json { "type": "topic", "forum": "https://community.n8n.io", "topic_id": 302663, "title": "AI Assistant on self-hosted n8n: early setup instructions", "slug": "ai-assistant-on-self-hosted-n8n-early-setup-instructions", "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663", "posts_count": 4, "views": 512, "like_count": 3, "created_at": "2026-01-05T10:00:00.000Z" } ``` ```json { "type": "post", "topic_id": 302663, "post_id": 565721, "post_number": 1, "username": "Ophir_Prusak", "cooked_html": "

Here's how to set it up:

  1. Install the self-hosted package
  2. Restart n8n
", "content_markdown": "Here's how to set it up:\n\n1. Install the **self-hosted** package\n2. Restart n8n", "content_text": "Here's how to set it up: Install the self-hosted package Restart n8n", "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663/1" } ``` `cooked_html` is Discourse's raw rendered post body. `content_markdown` and `content_text` are cleaned conversions of it (via [markdownify](https://pypi.org/project/markdownify/) and [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)) — use these for LLM input, search indexing, or anywhere raw HTML is unwanted. Download the dataset as JSON, CSV, Excel, or HTML from the **Storage** tab. #### Data table | Field | Item type | Description | |---|---|---| | `title` / `topic_title` | topic / post | Topic title | | `url` | both | Direct link to the topic or post | | `username` | post | Author's forum username | | `cooked_html` | post | Rendered HTML of the post body | | `content_markdown` | post | Post body as clean Markdown | | `content_text` | post | Post body as clean plain text | | `posts_count` | topic | Number of replies in the topic | | `like_count` | both | Like count | | `created_at` | both | ISO timestamp | ### Example: market research from a community's Jobs category You can run this actor end-to-end through the [Apify MCP server](https://mcp.apify.com/) to answer research questions without writing scraping code. Example: *"What kind of freelance work do n8n users want to buy in 2026?"* 1. **Discover the category** — call the actor with `mode: "categories"` on `https://community.n8n.io`. This lists every category with a ready-to-use `category_slug_for_scraping`, e.g. `"jobs/13"` for the `Jobs` category. 2. **Scrape it with posts** — call the actor again with `mode: "category"`, `categorySlug: "jobs/13"`, `includePosts: true`, `maxTopics: 80`. This returns each job-post topic plus its replies, with `content_text` already cleaned of HTML. 3. **Analyze** — pull `content_text` across all `post` items and count keyword mentions (e.g. "AI agent", "CRM", "voice AI", "RAG", "self-hosted") to see what buyers are actually asking for, and compare "hiring"/"looking for" vs. "for hire"/"available" topic titles to gauge demand vs. supply. A real run against `community.n8n.io/c/jobs` (61 topics, 204 rows) found: **AI agent building** was the top request (72 mentions), usually combined with **CRM integration** (92), **document/RAG/OCR processing** (60/38/22), and a growing **voice AI** niche (Vapi + Twilio + n8n stacks, 31+23 mentions). Buy-side posts outnumbered for-hire posts roughly 2:1, and **self-hosted** n8n (21 mentions) was a common hard requirement. Rates ranged $15–50/hr or $250–1,500 flat, with a shift toward long-term retainers over one-off builds. This same discover → scrape → analyze flow works for any Discourse forum's category (support backlogs, feature requests, showcase/built-with sections, etc.), not just job boards. ### Cost estimation This actor only makes lightweight JSON API requests (no browser rendering), so it's cheap to run — typically well under 1 compute unit for a few hundred topics without full posts. Enabling `includePosts` roughly doubles request volume per topic (one extra request per ~20 posts). ### Tips - Leave `includePosts` off for a quick topic-index crawl, then re-run with `topicUrls` on just the topics you care about. - Lower `maxConcurrency` / raise `requestDelaySecs` on smaller or self-hosted forums to stay polite. - Enable Apify Proxy for large crawls to avoid rate limiting. ### FAQ & disclaimers Only scrape publicly accessible forum content, and respect each forum's Terms of Service and `robots.txt`. This actor is not affiliated with Discourse, n8n, SmartThings, or NVIDIA. Found a Discourse site this doesn't work on? Open an issue in the Issues tab. # Actor input Schema ## `forumUrl` (type: `string`): Base URL of any Discourse-powered forum, e.g. https://community.n8n.io ## `mode` (type: `string`): What to crawl. 'categories' lists the forum's categories/sections (use this to discover a categorySlug for mode=category). 'latest'/'top' pull the site-wide topic list, 'category' pulls one category, 'search' pulls search results, 'topicUrls' scrapes only the topics you list. ## `categorySlug` (type: `string`): Required when mode=category. E.g. 'questions' or 'questions/12' (slug or slug/id) as it appears in the category URL. ## `searchQuery` (type: `string`): Required when mode=search. Uses Discourse's /search.json endpoint (supports its query filters, e.g. 'order:latest #category'). ## `topicUrls` (type: `array`): Required when mode=topicUrls. Direct URLs to Discourse topics. ## `includePosts` (type: `boolean`): If true, fetch every topic's full post stream (original post + all replies) in addition to the topic metadata. If false, only topic-level metadata is scraped (much faster, fewer requests). ## `maxTopics` (type: `integer`): Maximum number of topics to scrape. 0 = unlimited. ## `maxPostsPerTopic` (type: `integer`): Maximum number of posts to fetch per topic (only used when includePosts is true). 0 = unlimited. ## `requestDelaySecs` (type: `integer`): Polite delay between outgoing API requests to avoid overloading the forum. ## `maxConcurrency` (type: `integer`): Maximum number of concurrent HTTP requests. ## `proxyConfiguration` (type: `object`): Proxy settings. Recommended if scraping many pages to avoid rate limiting. ## Actor input object example ```json { "forumUrl": "https://community.n8n.io", "mode": "latest", "topicUrls": [], "includePosts": true, "maxTopics": 100, "maxPostsPerTopic": 0, "requestDelaySecs": 1, "maxConcurrency": 3, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `dataset` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "forumUrl": "https://community.n8n.io" }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/discourse-community-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "forumUrl": "https://community.n8n.io" } # Run the Actor and wait for it to finish run = client.actor("rl1987/discourse-community-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "forumUrl": "https://community.n8n.io" }' | apify call rl1987/discourse-community-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rl1987/discourse-community-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Discourse Community Scraper", "description": "Generalised scraper for any Discourse-based community forum (topics, posts, categories, search) via Discourse's JSON API.", "version": "0.1", "x-build-id": "lI36toMRY0NFGnEd3" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/rl1987~discourse-community-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-rl1987-discourse-community-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/rl1987~discourse-community-scraper/runs": { "post": { "operationId": "runs-sync-rl1987-discourse-community-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/rl1987~discourse-community-scraper/run-sync": { "post": { "operationId": "run-sync-rl1987-discourse-community-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "forumUrl", "mode" ], "properties": { "forumUrl": { "title": "Forum URL", "type": "string", "description": "Base URL of any Discourse-powered forum, e.g. https://community.n8n.io" }, "mode": { "title": "Mode", "enum": [ "categories", "latest", "top", "category", "search", "topicUrls" ], "type": "string", "description": "What to crawl. 'categories' lists the forum's categories/sections (use this to discover a categorySlug for mode=category). 'latest'/'top' pull the site-wide topic list, 'category' pulls one category, 'search' pulls search results, 'topicUrls' scrapes only the topics you list.", "default": "latest" }, "categorySlug": { "title": "Category slug/path", "type": "string", "description": "Required when mode=category. E.g. 'questions' or 'questions/12' (slug or slug/id) as it appears in the category URL." }, "searchQuery": { "title": "Search query", "type": "string", "description": "Required when mode=search. Uses Discourse's /search.json endpoint (supports its query filters, e.g. 'order:latest #category')." }, "topicUrls": { "title": "Topic URLs", "type": "array", "description": "Required when mode=topicUrls. Direct URLs to Discourse topics.", "default": [], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "includePosts": { "title": "Include full posts", "type": "boolean", "description": "If true, fetch every topic's full post stream (original post + all replies) in addition to the topic metadata. If false, only topic-level metadata is scraped (much faster, fewer requests).", "default": true }, "maxTopics": { "title": "Max topics", "minimum": 0, "type": "integer", "description": "Maximum number of topics to scrape. 0 = unlimited.", "default": 100 }, "maxPostsPerTopic": { "title": "Max posts per topic", "minimum": 0, "type": "integer", "description": "Maximum number of posts to fetch per topic (only used when includePosts is true). 0 = unlimited.", "default": 0 }, "requestDelaySecs": { "title": "Delay between requests (seconds)", "minimum": 0, "type": "integer", "description": "Polite delay between outgoing API requests to avoid overloading the forum.", "default": 1 }, "maxConcurrency": { "title": "Max concurrency", "minimum": 1, "maximum": 20, "type": "integer", "description": "Maximum number of concurrent HTTP requests.", "default": 3 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Proxy settings. Recommended if scraping many pages to avoid rate limiting.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```