Discourse Forum Scraper - Topics, Replies & Views

Scrape any Discourse community in bulk. Extract topic title, URL, category, tags, author, reply count, views, likes, creation and last-post dates, plus pinned and closed flags to CSV/JSON. No API key, no login.

---

What does the Discourse Scraper do?

This Actor turns Discourse data into a structured dataset. You point it at a whole forum or a single category, it walks the result pages one after another, and it writes one clean row per topic into your dataset — ready to export as JSON, CSV or Excel, or to pull straight from the Apify API.

Every Discourse forum answers with JSON when .json is appended to a listing URL — it is how the forum's own front end loads. The Actor reads those listings directly and resolves category and poster IDs against the same response, so each row names the category and the author rather than leaving numeric IDs behind. Pagination is followed automatically until it runs out of results, hits your page limit or hits your Max items cap, whichever comes first. Every row is de-duplicated across the whole run, so you are never billed twice for the same topic.

There is no API key, no login and no browser involved. That keeps runs fast and cheap, and it means you can schedule the Actor without worrying about credentials expiring.

Who is it for?

• Product and support teams mining their own community for recurring issues.
• Competitive-intelligence analysts watching a rival's user forum.
• Developer-relations teams measuring community activity over time.
• Researchers studying how technical communities discuss a topic.
• Content teams finding real questions worth answering publicly.

Use cases

• Export a forum's topics and rank them by views to find what people care about.
• Track how many replies and likes topics get in a category over time.
• Find unanswered topics by filtering on reply count and accepted answers.
• Build a dataset of real user questions for support or content planning.
• Compare activity across several communities in a single run.

Why use this Discourse Scraper?

• 🔑 Keyless — no account, no API token, no cookies to paste.
• 📦 26 fields per topic — everything the source exposes, already typed.
• 📄 Real pagination — it walks page after page instead of returning the first screen.
• 🎯 Precise caps — Max items stops the run exactly where you want it, so the bill is predictable.
• 📊 Export anywhere — JSON, CSV, Excel or HTML, plus the Apify API and integrations.
• 💸 Pay per result — you pay for rows you actually receive, with no platform fees to calculate.

What data can you extract?

Every run produces one row per topic, with these fields:

Field	Type	Description
forumDomain	string	Domain of the Discourse forum this row came from
topicId	string	Discourse topic ID
title	string	Topic title
slug	string	URL slug of the topic
topicUrl	string	Direct link to the topic
categoryId	string	Category ID
categoryName	string	Category name resolved from the forum's category list
tags	string	Topic tags, separated by a pipe
originalPoster	string	Username of the member who opened the topic
lastPosterUsername	string	Username of the most recent poster
participants	string	Usernames taking part, separated by a pipe
createdAt	string	ISO timestamp the topic was created
lastPostedAt	string	ISO timestamp of the most recent post
bumpedAt	string	ISO timestamp the topic was last bumped
postsCount	number	Total posts in the topic including the first one
replyCount	number	Replies as counted by Discourse
views	number	View count
likeCount	number	Total likes across the topic
excerpt	string	Excerpt of the opening post with HTML stripped
hasAcceptedAnswer	boolean	Whether a reply is marked as the solution
isPinned	boolean	Whether the topic is pinned
isClosed	boolean	Whether the topic is closed to replies
isArchived	boolean	Whether the topic is archived
imageUrl	string	Topic image when the forum shows one
page	number	Listing page the topic appeared on
scrapedAt	string	ISO timestamp of extraction

Output example

{
  "forumDomain": "meta.discourse.org",
  "topicId": "1",
  "title": "New to Discourse? Start here!",
  "slug": "new-to-discourse-start-here",
  "topicUrl": "https://meta.discourse.org/t/new-to-discourse-start-here/1",
  "categoryId": "67",
  "categoryName": "Announcements",
  "tags": null,
  "originalPoster": "Discourse",
  "lastPosterUsername": "Discourse",
  "participants": "Discourse",
  "createdAt": "2013-02-01T04:52:28.073Z",
  "lastPostedAt": "2013-02-01T04:52:28.397Z",
  "bumpedAt": "2025-09-23T19:13:50.543Z",
  "postsCount": 1,
  "replyCount": 5,
  "views": 119223,
  "likeCount": 441,
  "excerpt": ":waving_hand: We\u2019re so glad you\u2019re here! This is our official community, where you can connect, ask questions, and share ideas about Discourse, a powerful platform for\u2026",
  "hasAcceptedAnswer": false,
  "isPinned": true,
  "isClosed": true,
  "isArchived": false,
  "imageUrl": null,
  "page": 1,
  "scrapedAt": "2026-09-19T11:22:03.517Z"
}

How to use

Option A — a whole forum

{
  "listing": "latest",
  "maxItems": 1000,
  "maxPagesPerForum": 30,
  "forumUrls": [
    "https://meta.discourse.org"
  ]
}

1. Open the Actor and fill in the first field.
2. Set the page limit and Max items to bound the run.
3. Click Start, then export from the Output tab.

Option B — a single category

{
  "categorySlug": "bug",
  "listing": "latest",
  "maxItems": 1000,
  "maxPagesPerForum": 30,
  "forumUrls": [
    "https://meta.discourse.org"
  ]
}

Everything in the first field is processed independently, so you can batch several targets into one run and split them apart afterwards.

Input parameters

Parameter	Type	Default	Description
forumUrls	array	–	Home pages of the Discourse forums to scrape, e.
listing	string	latest	Which feed to walk.
categorySlug	string	–	Crawl a single category instead of the whole forum, e.
maxPagesPerForum	integer	8	How many result pages to walk for each forum.
maxItems	integer	500	Stop after this many topics.
maxConcurrency	integer	2	Parallel requests.
proxyConfiguration	object	{"useApifyProxy": true}	Proxy used to fetch pages.

Tips for best results

• Paste the forum's home page, e.g. https://meta.discourse.org — the Actor appends the listing path.
• Listing picks the feed: latest, top or new. Latest is the broadest.
• Set Category slug to crawl one category instead of the whole forum.
• Each listing page returns about 30 topics, so plan page counts accordingly.
• views is the clearest demand signal a public forum publishes.
• hasAcceptedAnswer false with a high replyCount is how you find unresolved discussions.
• Category names are resolved once per forum from its own category list, so the column is readable.
• tags is empty on forums that have tagging switched off — that is a forum setting, not a gap in extraction.
• Private categories never appear, because the Actor reads only what a logged-out visitor sees.
• Keep concurrency low — most Discourse forums are small community servers.

Integrations

Send results straight into the tools you already use: Google Sheets, Slack, Zapier, Make, Airtable or any Webhook. You can also schedule the Actor to run hourly, daily or weekly and have each run append to the same dataset, which is how you build a price or availability history rather than a one-off snapshot.

API usage

Run the Actor and collect results from any language. Replace <YOUR_TOKEN> with your Apify API token.

cURL

curl -X POST "https://api.apify.com/v2/acts/logiover~discourse-forum-scraper/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"listing": "latest", "maxItems": 1000, "maxPagesPerForum": 30, "forumUrls": ["https://meta.discourse.org"]}'

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_TOKEN>' });
const run = await client.actor('logiover/discourse-forum-scraper').call({"listing": "latest", "maxItems": 1000, "maxPagesPerForum": 30, "forumUrls": ["https://meta.discourse.org"]});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('<YOUR_TOKEN>')
run = client.actor('logiover/discourse-forum-scraper').call(run_input={"listing": "latest", "maxItems": 1000, "maxPagesPerForum": 30, "forumUrls": ["https://meta.discourse.org"]})
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

Use with AI agents (MCP)

This Actor is available through the Apify MCP server, so an AI agent can call it as a tool. Point your agent at https://mcp.apify.com and it can run the Discourse Scraper on demand — for example: "Pull the first 500 topics from Discourse and summarise what you find." The agent receives the same structured rows you would get from the UI.

FAQ

Do I need a Discourse account or API key?

No. The Actor reads publicly available data only. There is nothing to authenticate and no credentials to rotate.

How many topics can I get in one run?

As many as the source exposes. Raise the page limit and Max items together; the run stops at whichever limit it reaches first.

Why did I get fewer rows than I asked for?

The source ran out of topics. That is normal for narrow searches — broaden the query or add more targets to one run.

Are results de-duplicated?

Yes. Each topic is emitted once per run, even when it appears on several pages, so you are never billed twice for the same record.

Why are some fields empty?

Discourse does not publish every attribute for every topic. Empty means the source did not supply it, not that extraction failed.

What export formats are supported?

JSON, CSV, Excel, HTML and RSS from the Output tab, plus the Apify API and any integration you connect.

How fast is it?

It is pure HTTP with no browser, so a page of results typically takes a second or two. Raise Max concurrency carefully — the source rate-limits aggressive crawling.

Can I schedule it?

Yes. Use the Apify scheduler to run it on any interval and append each run to the same dataset for time-series analysis.

Does it work behind a proxy?

It uses Apify Proxy automatically. You can switch groups or supply your own proxies in Proxy configuration.

How often does the data change?

Discourse updates continuously. Re-run whenever you need current data; the Actor always reads the live source, never a cache.

Is the output schema stable?

Yes. Field names and types are fixed, so downstream pipelines will not break between runs.

What if the source changes its format?

Open an issue on the Issues tab and it gets fixed. The Actor is actively maintained.

Is it legal?

This Actor reads only publicly available data from Discourse — the same content any visitor or client sees without logging in. It does not bypass authentication, does not collect private data and does not attempt to defeat access controls. You are responsible for how you use the output: respect the source's terms of service, applicable copyright, and data-protection law such as GDPR where personal data is involved. Scraping public data is generally lawful in the EU and the US, but the responsibility for the downstream use of that data sits with you.

Related scrapers

• Reddit Search Scraper — Another community source
• Stack Exchange Scraper — Technical Q&A
• Hacker News Archive — Developer discussion
• Lemmy Scraper — Federated communities