# Discourse.org Forum Scraper (`enezli/discourse-forum-scraper`) Actor Search any public Discourse forum and get clean, de-duplicated topic JSON: title, author, replies, views and canonical topic URLs. One click, no required fields, no LLM. - **URL**: https://apify.com/enezli/discourse-forum-scraper.md - **Developed by:** [Turgay NANTA](https://apify.com/enezli) (community) - **Categories:** Agents, MCP servers, Social media - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Discourse Forum Scraper Search any public Discourse forum and get clean, de-duplicated topic JSON: title, author, replies, views and canonical topic URLs. One click, no required fields, no LLM. ### What it does Tens of thousands of communities — OpenAI, Cloudflare, Home Assistant, most crypto and dev ecosystems — run on Discourse, and every public Discourse forum ships the same open JSON API. This **one actor works on all of them**: point it at a forum URL, optionally add search terms, and get clean topic JSON — titles, reply counts, excerpts and canonical URLs. Use it to research communities, track questions about your product, or feed fresh discussions to an AI agent. **Why this one:** - **One actor for EVERY public Discourse forum (thousands of communities)** - **Search mode + latest-topics mode in one** - **Monitor mode: only NEW topics on scheduled runs** - **No LLM anywhere** — deterministic output, predictable costs, no hallucinated fields - **Clean by default** — canonical URLs (tracking parameters stripped), parsed numbers, merged duplicates ### Quick start (no code) 1. Click **Try for free** / **Start** — every field has a working default, nothing is required. 2. (Optional) change **query** to what you need. 3. Open the **Dataset** tab when the run finishes → export as JSON, CSV or Excel. ### Input | Field | Required | Default | Description | |---|---|---|---| | `query` | no | `https://meta.discourse.org ai agents` | Forum URL followed by optional search terms. URL only → latest topics. Works on ANY public Discourse forum. | | `maxResults` | no | `20` | Maximum clean results (capped at 500) | | `enrich` | no | `false` | Deterministic enrichment per record — see below | | `monitor` | no | `false` | Compare with the previous run, flag NEW records only | Example input: ```json { "query": "https://meta.discourse.org ai agents", "maxResults": 50 } ```` ### Output Real example record (from a live run): ```json { "id": "368942", "title": "Forum Researcher AI Agent guide", "url": "https://meta.discourse.org/t/forum-researcher-ai-agent-guide/368942", "reviews": 13.0, "description": "This guide explains the Forum Researcher agent in Discourse AI, how it works, and how to configure it...", "created_at": "2025-06-05T05:06:40.007Z", "completeness": 0.33 } ``` The final `_summary` row carries run totals (`total_clean`, `deduped`, `enriched`); in monitor mode a `_changes` row lists keys new since the last run. **Field reference** | Field | Meaning | |---|---| | `id` | Topic ID (stable, deduplication key) | | `title` | Topic title | | `url` | Canonical topic URL | | `reviews` | Reply count (engagement metric) | | `description` | Topic excerpt | | `created_at` | Topic creation timestamp | | `views / has_accepted_answer` | Present when the forum exposes them (latest mode) | | `price` / `price_text` | Parsed numeric value + original text, when the source publishes one | | `completeness` | 0–1 filled-fields score (with `enrich`) | ### Use cases - **Community listening** — search every mention of your product across the forums that matter to you. - **Support mining** — collect unanswered questions in your niche and answer them where users are. - **Content research** — the most-replied topics of a niche forum are a ranked list of what people care about. - **Competitive research** — track feature requests and complaints on a competitor's community. - **AI agents** — give your agent live forum context: 'what are people asking about X this week?' ### Enrichment (optional, charged only when it produces something) Set `enrich: true` and every record additionally gets: e-mail addresses extracted from the description (when present), the canonical `domain` of the record's URL, and a `completeness` score (0–1, how many core fields are filled). Deterministic — the same input always yields the same output — and you are **only charged for records that actually got enriched**. Records where enrichment adds nothing are free. ### Monitor mode — change alerts on a schedule Set `monitor: true` and the actor compares the current run with the previous one (per-actor named storage) and flags **only NEW records**. Combine with [Apify Schedules](https://docs.apify.com/platform/schedules) for a daily/hourly watch: the `_changes` summary row lists what appeared since the last run, and the `change-alert` event is charged **per new record only** — an unchanged run costs you almost nothing. ### Use it from your code **Python** ```python from apify_client import ApifyClient client = ApifyClient("") run = client.actor("EnezLi/discourse-forum-scraper").call(run_input={ "query": "https://meta.discourse.org ai agents", "maxResults": 50 }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) ``` **JavaScript** ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: '' }); const { defaultDatasetId } = await client.actor('EnezLi/discourse-forum-scraper').call({ "query": "https://meta.discourse.org ai agents", "maxResults": 50 }); const { items } = await client.dataset(defaultDatasetId).listItems(); console.log(items); ``` **curl** ```bash curl -X POST "https://api.apify.com/v2/acts/EnezLi~discourse-forum-scraper/run-sync-get-dataset-items?token=" \ -H "Content-Type: application/json" \ -d '{ "query": "https://meta.discourse.org ai agents", "maxResults": 50 }' ``` ### Use it with AI agents (MCP) This actor is agent-ready: it appears in Apify's **AGENTS / MCP servers** catalog, so any MCP-capable assistant (Claude, custom agents, LangGraph tools) can discover and call it with a one-line tool call — zero required fields means an agent can run it safely with defaults. Connect your agent to the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp) and ask for live data in natural language. ### Pricing — Pay-Per-Event, start is free | Event | When charged | |---|---| | Actor start | **Free ($0)** — try it with one click | | `result` | Per clean result returned | | `enrichment` | Only per record that actually got enriched | | `change-alert` | Monitor mode: per NEW record since the previous run | No subscription, no minimum. Volume discounts apply automatically through Apify account tiers (up to −44% on GOLD). Typical run cost example: 20 results ≈ a few cents total — you can predict your bill from the numbers above before you run. ### Is this legal? This actor collects **publicly available data only** — the same information any visitor sees in a browser, via public endpoints. It does not bypass logins, collect private personal data, or store credentials. You are responsible for using the output in compliance with the source site's terms and the laws that apply to you (e.g. GDPR when the output contains personal data). ### Support & feedback Found a bug, need another field, or want a variant for a related platform? **Open an issue on the Issues tab** — issues are monitored and answered, and frequently-requested fields get added to the standard output. The actor is maintained as part of a scraper family built on one shared, tested core (bugs fixed once are fixed everywhere). ### Changelog - **0.1** (2026-07) — initial public release: search, dedup, optional enrichment, monitor mode, PPE pricing. ### Limitations (honest ones) Topic-level data only (no full thread text yet). Search relevance is the forum's own search. Private forums are out of scope. ### FAQ **Which forums does this work on?** Any public Discourse instance — if `/latest.json` opens in your browser, it works. Private/login-only forums are not supported. **Can I scrape full posts/replies?** The actor returns topic-level data (title, excerpt, counts). Full thread content is a planned add-on — open an issue if you need it. **How do I monitor a forum daily?** Enable **monitor** and schedule the actor (Apify Schedules). Each run charges only for NEW topics via `change-alert`. **Do I need an API key or account on the source platform?** No. The actor uses public endpoints — you only need your Apify account. **Does it use AI / an LLM?** No. The core is fully deterministic: same input, same output, no hallucinations, no per-token costs. **Can I run it on a schedule?** Yes — use Apify Schedules; combine with monitor mode to pay only for what's new. **What's the maximum number of results?** 500 per run (memory-safe cap). Run multiple queries or schedule runs for more. **How is my bill calculated?** Only from the events in the Pricing table — start is free, and there is no subscription. # Actor input Schema ## `query` (type: `string`): Any public Discourse forum URL followed by optional search terms (e.g. 'https://meta.discourse.org ai agents'). URL only = latest topics. Leave empty to run the default example — no required fields. ## `maxResults` (type: `integer`): Maximum number of clean results to return (capped at 500). ## `enrich` (type: `boolean`): Adds deterministic enrichment per record: extracted emails, canonical domain and a completeness score. Charged per enriched record (see Pricing). ## `monitor` (type: `boolean`): Compares this run with the previous one and flags NEW records only. Ideal for scheduled runs — get alerted when something changes. Charged per change. ## Actor input object example ```json { "query": "https://meta.discourse.org ai agents", "maxResults": 20, "enrich": false, "monitor": false } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "query": "https://meta.discourse.org ai agents" }; // Run the Actor and wait for it to finish const run = await client.actor("enezli/discourse-forum-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "query": "https://meta.discourse.org ai agents" } # Run the Actor and wait for it to finish run = client.actor("enezli/discourse-forum-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "query": "https://meta.discourse.org ai agents" }' | apify call enezli/discourse-forum-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=enezli/discourse-forum-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Discourse.org Forum Scraper", "description": "Search any public Discourse forum and get clean, de-duplicated topic JSON: title, author, replies, views and canonical topic URLs. One click, no required fields, no LLM.", "version": "0.1", "x-build-id": "dwMlBlDawYDWxg8je" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/enezli~discourse-forum-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-enezli-discourse-forum-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/enezli~discourse-forum-scraper/runs": { "post": { "operationId": "runs-sync-enezli-discourse-forum-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/enezli~discourse-forum-scraper/run-sync": { "post": { "operationId": "run-sync-enezli-discourse-forum-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "query": { "title": "Forum URL + search terms", "type": "string", "description": "Any public Discourse forum URL followed by optional search terms (e.g. 'https://meta.discourse.org ai agents'). URL only = latest topics. Leave empty to run the default example — no required fields.", "default": "https://meta.discourse.org ai agents" }, "maxResults": { "title": "Max results", "minimum": 1, "maximum": 500, "type": "integer", "description": "Maximum number of clean results to return (capped at 500).", "default": 20 }, "enrich": { "title": "Enrich results", "type": "boolean", "description": "Adds deterministic enrichment per record: extracted emails, canonical domain and a completeness score. Charged per enriched record (see Pricing).", "default": false }, "monitor": { "title": "Monitor mode (change alerts)", "type": "boolean", "description": "Compares this run with the previous one and flags NEW records only. Ideal for scheduled runs — get alerted when something changes. Charged per change.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```