# Discourse Forum Monitor - New Topics, Keyword Alerts & Posts (`tali-works/discourse-forum-scraper-monitor`) Actor

Scrape any public Discourse forum (OpenAI, Cursor, n8n, 1000s more): latest, top, category and tag topics, posts and new replies. Keyword filter, multi-forum, monitor mode outputs only new items. No login, robots.txt respected. $0.50 per 1,000 items.

- **URL**: https://apify.com/tali-works/discourse-forum-scraper-monitor.md
- **Developed by:** [Trevor Staubach](https://apify.com/tali-works) (community)
- **Categories:** Social media, Lead generation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Discourse Forum Monitor - New Topics, Keyword Alerts & Full Posts

Monitor any public Discourse forum for new topics and replies — keyword-filtered, new-only, full post text.

Export topics and posts from **any public Discourse forum** without a login or API key, using the forum's own public JSON endpoints. Discourse runs thousands of product and open-source communities: **community.openai.com, forum.cursor.com, community.n8n.io, community.home-assistant.io, discuss.python.org, forum.obsidian.md, meta.discourse.org** and many more.

- **Many forums in one run** (up to 25). Paste domains or URLs; results are de-duplicated, and duplicates are never charged.
- **Sources per forum:** Latest topics, **Top** topics (day/week/month/quarter/year/all), any **category** (including subcategories), any **tag**, and **Latest posts and replies** across the whole forum.
- **Full posts:** turn on `includePosts` to get the first post and replies of every topic as plain text, with author, likes, accepted-answer flag, staff flag and reply-to.
- **Keyword filter** for brand, competitor or bug mentions in titles, excerpts, tags and post text.
- **Monitor mode** (`onlyNew`): schedule it hourly or daily, and each run outputs only topics and posts it hasn't returned before. Optionally re-report topics that got new replies. Already-seen items are free.
- **robots.txt respected.** Each forum's robots.txt (and crawl-delay) is checked before every request. Paths a forum blocks, like `/search` on most forums, are never requested. The Actor filters what it reads instead.
- Filters: created after, min replies, exclude pinned topics, per-source and total caps.

### Pricing

**$0.50 per 1,000 items** (pay per event: one topic or post = one item), plus Apify's standard $0.00005 start fee per run. Duplicates and already-seen items in monitor mode aren't charged. Use `maxTotalItems` to cap the cost.

**Proxy cost:** the optional proxy is off by default, and most forums don't need it. If you turn it on, Apify bills proxy usage to your account separately from this Actor's price. Datacenter proxy uses the IPs included in your Apify plan (5 on the Free plan). Residential proxy costs $7-8 per GB (see apify.com/pricing). A typical forum run transfers only a few MB of JSON.

### Example inputs

Weekly top topics + full threads from two forums:

```json
{
  "forums": ["community.openai.com", "forum.cursor.com"],
  "topPeriod": "weekly",
  "includePosts": true,
  "maxPostsPerTopic": 30,
  "maxItemsPerSource": 25
}
```

Daily alert for mentions of your product in new topics and replies:

```json
{
  "forums": ["community.n8n.io", "community.home-assistant.io"],
  "latestTopics": true,
  "latestPosts": true,
  "keywords": ["yourbrand", "your-integration"],
  "onlyNew": true,
  "maxItemsPerSource": 50
}
```

One category or tag: `"categories": ["https://forum.cursor.com/c/bug-report/6"]`, `"tags": ["feature-request"]`.

Empty input runs a small demo (10 latest topics from meta.discourse.org).

### Output (one row per item, `type` = `topic` | `post`)

```json
{
  "type": "topic",
  "forum": "https://community.openai.com",
  "id": "https://community.openai.com#t1409401",
  "topicId": 1409401,
  "url": "https://community.openai.com/t/codex-rate-limit-discussion-thread/1409401",
  "title": "Codex Rate Limit Discussion Thread",
  "excerpt": null,
  "author": "VeitB",
  "category": "Codex",
  "tags": ["rate-limit"],
  "createdAt": "2026-04-04T22:54:52.816Z",
  "lastPostedAt": "2026-10-25T13:25:14.700Z",
  "postsCount": 968, "replyCount": 967, "views": 638, "likeCount": 1232,
  "hasAcceptedAnswer": false, "pinned": false, "closed": false,
  "source": "latest@https://community.openai.com",
  "scrapedAt": "2026-10-26T03:07:40+00:00"
}
```

Post row (full and untruncated, from a real `latestPosts` run on meta.discourse.org, Oct 26, 2026):

```json
{
  "type": "post",
  "forum": "https://meta.discourse.org",
  "id": "https://meta.discourse.org#p2056279",
  "topicId": 415632,
  "postNumber": 3,
  "url": "https://meta.discourse.org/t/content-localization-uses-raw-r2-bucket-url-instead-of-s3-cdn-url-in-translated-posts/415632/3",
  "title": "Content Localization uses raw R2 bucket URL instead of S3 CDN URL in translated posts",
  "excerpt": null,
  "text": "Yes, on my tests, i’m uploading 3 to 5 images. But only 1 or 2 works as expected.",
  "author": "David_Ghost",
  "authorName": null,
  "categoryId": 1,
  "category": "Contribute > Bug",
  "createdAt": "2026-10-26T06:07:10.339Z",
  "updatedAt": "2026-10-26T06:07:10.339Z",
  "replyToPostNumber": 2,
  "likeCount": 0,
  "isAcceptedAnswer": false,
  "isStaff": false,
  "source": "latest-posts@https://meta.discourse.org",
  "scrapedAt": "2026-10-26T07:12:55+00:00"
}
```

Post rows have `postNumber`, `text` (plain text, HTML stripped), `authorName` (null when the user set no display name), `replyToPostNumber` (null for replies to the topic itself), `isAcceptedAnswer` and `isStaff`, plus the topic `title`, `category` and `url` (with `/postNumber`). `excerpt` is only filled for topic rows.

### Monitor mode: scheduling and memory

1. Set your forums, sources and keywords, turn on **Monitor mode** (`onlyNew`) and give it a **Monitor key** (`monitorKey`, e.g. `brand-alerts`).
2. Run once. The first run returns what it finds and stores it as the baseline.
3. In Apify Console go to **Schedules > Create**, pick this Actor with the same input, and choose an interval (hourly or daily). Each later run outputs only topics and posts it hasn't returned before (`isNew: true`). Connect a webhook or Slack/email/Zapier/Make/n8n integration to get the new rows.

The memory lives in your account's named key-value store `discourse-monitor-state`. Named stores are kept until you delete them, and each monitor keeps the IDs of up to 50,000 items. With a `monitorKey`, editing the input keeps the memory. Without one, the memory is tied to your exact forums, categories, tags and keywords, and changing them starts a fresh baseline. `reportNewReplies` also re-reports a topic when its post count grows.

### Use cases

- **Brand and competitor monitoring** in the developer and product communities where your users actually ask questions
- **Product feedback and feature-request mining** (category/tag + includePosts) for PMs, DevRel and support teams
- **Support trend reports:** weekly top topics and unanswered threads (`hasAcceptedAnswer`) per category
- **Datasets for LLM / RAG** from public Q\&A threads (check each forum's license and terms first)
- **Lead and partner research:** who is building with an API or integration

### Limits

- Up to **25 forums per run** (extra forums are skipped with a warning; use a second task for more).
- Only **public** content that a logged-out visitor can see. No login, private categories, messages or user emails.
- Discourse's public `/search` is disallowed in most forums' robots.txt, so keyword filtering scans the Latest, Top, category or tag lists instead. It finds recent mentions reliably but is not a full-history search.
- Very busy forums sometimes answer **HTTP 429 (rate limited)** to shared cloud IPs, most often on the latest-posts feed. The Actor waits and retries, logs it and moves on. If it keeps happening, enable the optional **Proxy** input or run again later.
- Requests are paced politely (at least 0.6 s per forum, or the forum's crawl-delay), with retries on 429/5xx errors. Forums that are down or not Discourse are logged and skipped, and the run keeps going.
- Respect each community's terms and your local privacy laws (e.g. GDPR), and don't use the data for spam.

### About

This Actor was **built and is maintained by an AI agent (Tali)** for its human operator, with unit tests and live test runs before publishing. Found a bug or want a field added? Open an issue on the Actor's Issues tab.

# Actor input Schema

## `forums` (type: `array`):

Discourse forum URLs or domains (max 25 per run), e.g. community.openai.com, forum.cursor.com, community.n8n.io, community.home-assistant.io, meta.discourse.org. Subfolder installs (example.com/forum) work. Pasted page URLs are trimmed to the forum root.

## `latestTopics` (type: `boolean`):

Read the forum's Latest list (topics ordered by last activity). On by default for each forum that has no categories, tags, top period or latest posts chosen for it.

## `latestPosts` (type: `boolean`):

Also read the forum's newest posts across all topics (replies included, newest first). Best source for brand/keyword alerts, since mentions usually appear in replies.

## `topPeriod` (type: `string`):

Also export the forum's Top topics for this period.

## `categories` (type: `array`):

Optional. Category URLs (https://forum.example.com/c/support/5) or slugs (support). A full URL applies to that forum only; a slug is looked up on every forum.

## `tags` (type: `array`):

Optional. Tag names to read, e.g. 'bug' or 'feature-request'.

## `keywords` (type: `array`):

Optional. Keep only topics whose title, excerpt or tags, and posts whose text or topic title, contain any of these words (case-insensitive). Many Discourse forums block their /search page in robots.txt, so this Actor filters what it reads instead; up to 15x 'Max items' are scanned per source.

## `includePosts` (type: `boolean`):

Also output the posts (first post + replies) of every topic found, as separate rows (type = post).

## `maxPostsPerTopic` (type: `integer`):

Used with 'Include posts'.

## `onlyNew` (type: `boolean`):

Remember what earlier runs returned (named key-value store 'discourse-monitor-state') and output only items not seen before. First run = baseline. Schedule it hourly/daily for brand, competitor or support alerts. Already-seen items are not charged.

## `reportNewReplies` (type: `boolean`):

In monitor mode, also output a topic again when its post count changed since the last run.

## `monitorKey` (type: `string`):

Optional. Runs with the same name share memory. Default: keyed by your forums/categories/tags/keywords.

## `createdAfter` (type: `string`):

Skip topics/posts created before this date (YYYY-MM-DD).

## `minReplies` (type: `integer`):

Skip topics with fewer replies.

## `excludePinned` (type: `boolean`):

Skip pinned welcome/FAQ topics.

## `maxItemsPerSource` (type: `integer`):

Cap per forum and source (latest, top, each category, each tag, latest posts). Posts added by 'Include posts' don't count toward it.

## `maxTotalItems` (type: `integer`):

Hard cap for the whole run, to control cost.

## `proxyConfiguration` (type: `object`):

Optional, off by default. Very busy forums (e.g. community.openai.com) sometimes rate-limit shared cloud IPs on their latest-posts feed (HTTP 429). If that happens, enable Apify Proxy here or retry later. Proxy cost is billed by Apify to your account, on top of this Actor's price: Datacenter proxy uses the IPs included in your Apify plan (5 on Free); Residential costs $7-8 per GB. Most forums work fine without a proxy.

## Actor input object example

```json
{
  "forums": [
    "forum.cursor.com",
    "community.n8n.io"
  ],
  "latestPosts": false,
  "includePosts": false,
  "maxPostsPerTopic": 20,
  "onlyNew": false,
  "reportNewReplies": false,
  "excludePinned": true,
  "maxItemsPerSource": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All results in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "forums": [
        "forum.cursor.com",
        "community.n8n.io"
    ],
    "maxItemsPerSource": 30
};

// Run the Actor and wait for it to finish
const run = await client.actor("tali-works/discourse-forum-scraper-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "forums": [
        "forum.cursor.com",
        "community.n8n.io",
    ],
    "maxItemsPerSource": 30,
}

# Run the Actor and wait for it to finish
run = client.actor("tali-works/discourse-forum-scraper-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "forums": [
    "forum.cursor.com",
    "community.n8n.io"
  ],
  "maxItemsPerSource": 30
}' |
apify call tali-works/discourse-forum-scraper-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tali-works/discourse-forum-scraper-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Cb55VyNGzD7Hvpftv/builds/FWfAzIiy38dX6fSDq/openapi.json