Discourse Forum Scraper

Search any public Discourse forum and get clean, de-duplicated topic JSON: title, author, replies, views and canonical topic URLs. One click, no required fields, no LLM.

What it does

Tens of thousands of communities — OpenAI, Cloudflare, Home Assistant, most crypto and dev ecosystems — run on Discourse, and every public Discourse forum ships the same open JSON API. This one actor works on all of them: point it at a forum URL, optionally add search terms, and get clean topic JSON — titles, reply counts, excerpts and canonical URLs. Use it to research communities, track questions about your product, or feed fresh discussions to an AI agent.

Why this one:

• One actor for EVERY public Discourse forum (thousands of communities)
• Search mode + latest-topics mode in one
• Monitor mode: only NEW topics on scheduled runs
• No LLM anywhere — deterministic output, predictable costs, no hallucinated fields
• Clean by default — canonical URLs (tracking parameters stripped), parsed numbers, merged duplicates

Quick start (no code)

1. Click Try for free / Start — every field has a working default, nothing is required.
2. (Optional) change query to what you need.
3. Open the Dataset tab when the run finishes → export as JSON, CSV or Excel.

Input

Field	Required	Default	Description
query	no	https://meta.discourse.org ai agents	Forum URL followed by optional search terms. URL only → latest topics. Works on ANY public Discourse forum.
maxResults	no	20	Maximum clean results (capped at 500)
enrich	no	false	Deterministic enrichment per record — see below
monitor	no	false	Compare with the previous run, flag NEW records only

Example input:

{
  "query": "https://meta.discourse.org ai agents",
  "maxResults": 50
}

Output

Real example record (from a live run):

{
  "id": "368942",
  "title": "Forum Researcher AI Agent guide",
  "url": "https://meta.discourse.org/t/forum-researcher-ai-agent-guide/368942",
  "reviews": 13.0,
  "description": "This guide explains the Forum Researcher agent in Discourse AI, how it works, and how to configure it...",
  "created_at": "2025-06-05T05:06:40.007Z",
  "completeness": 0.33
}

The final _summary row carries run totals (total_clean, deduped, enriched); in monitor mode a _changes row lists keys new since the last run.

Field reference

Field	Meaning
id	Topic ID (stable, deduplication key)
title	Topic title
url	Canonical topic URL
reviews	Reply count (engagement metric)
description	Topic excerpt
created_at	Topic creation timestamp
views / has_accepted_answer	Present when the forum exposes them (latest mode)
price / price_text	Parsed numeric value + original text, when the source publishes one
completeness	0–1 filled-fields score (with enrich)

Use cases

• Community listening — search every mention of your product across the forums that matter to you.
• Support mining — collect unanswered questions in your niche and answer them where users are.
• Content research — the most-replied topics of a niche forum are a ranked list of what people care about.
• Competitive research — track feature requests and complaints on a competitor's community.
• AI agents — give your agent live forum context: 'what are people asking about X this week?'

Enrichment (optional, charged only when it produces something)

Set enrich: true and every record additionally gets: e-mail addresses extracted from the description (when present), the canonical domain of the record's URL, and a completeness score (0–1, how many core fields are filled). Deterministic — the same input always yields the same output — and you are only charged for records that actually got enriched. Records where enrichment adds nothing are free.

Monitor mode — change alerts on a schedule

Set monitor: true and the actor compares the current run with the previous one (per-actor named storage) and flags only NEW records. Combine with Apify Schedules for a daily/hourly watch: the _changes summary row lists what appeared since the last run, and the change-alert event is charged per new record only — an unchanged run costs you almost nothing.

Use it from your code

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("EnezLi/discourse-forum-scraper").call(run_input={  "query": "https://meta.discourse.org ai agents",  "maxResults": 50 })
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });
const { defaultDatasetId } = await client.actor('EnezLi/discourse-forum-scraper').call({  "query": "https://meta.discourse.org ai agents",  "maxResults": 50 });
const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(items);

curl

curl -X POST "https://api.apify.com/v2/acts/EnezLi~discourse-forum-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{  "query": "https://meta.discourse.org ai agents",  "maxResults": 50 }'

Use it with AI agents (MCP)

This actor is agent-ready: it appears in Apify's AGENTS / MCP servers catalog, so any MCP-capable assistant (Claude, custom agents, LangGraph tools) can discover and call it with a one-line tool call — zero required fields means an agent can run it safely with defaults. Connect your agent to the Apify MCP server and ask for live data in natural language.

Pricing — Pay-Per-Event, start is free

Event	When charged
Actor start	Free ($0) — try it with one click
result	Per clean result returned
enrichment	Only per record that actually got enriched
change-alert	Monitor mode: per NEW record since the previous run

No subscription, no minimum. Volume discounts apply automatically through Apify account tiers (up to −44% on GOLD). Typical run cost example: 20 results ≈ a few cents total — you can predict your bill from the numbers above before you run.

Is this legal?

This actor collects publicly available data only — the same information any visitor sees in a browser, via public endpoints. It does not bypass logins, collect private personal data, or store credentials. You are responsible for using the output in compliance with the source site's terms and the laws that apply to you (e.g. GDPR when the output contains personal data).

Support & feedback

Found a bug, need another field, or want a variant for a related platform? Open an issue on the Issues tab — issues are monitored and answered, and frequently-requested fields get added to the standard output. The actor is maintained as part of a scraper family built on one shared, tested core (bugs fixed once are fixed everywhere).

Changelog

• 0.1 (2026-07) — initial public release: search, dedup, optional enrichment, monitor mode, PPE pricing.

Limitations (honest ones)

Topic-level data only (no full thread text yet). Search relevance is the forum's own search. Private forums are out of scope.

FAQ

Which forums does this work on?

Any public Discourse instance — if <forum>/latest.json opens in your browser, it works. Private/login-only forums are not supported.

Can I scrape full posts/replies?

The actor returns topic-level data (title, excerpt, counts). Full thread content is a planned add-on — open an issue if you need it.

How do I monitor a forum daily?

Enable monitor and schedule the actor (Apify Schedules). Each run charges only for NEW topics via change-alert.

Do I need an API key or account on the source platform?

No. The actor uses public endpoints — you only need your Apify account.

Does it use AI / an LLM?

No. The core is fully deterministic: same input, same output, no hallucinations, no per-token costs.

Can I run it on a schedule?

Yes — use Apify Schedules; combine with monitor mode to pay only for what's new.

What's the maximum number of results?

500 per run (memory-safe cap). Run multiple queries or schedule runs for more.

How is my bill calculated?

Only from the events in the Pricing table — start is free, and there is no subscription.