All Forums Thread Scraper

All Forums Thread Scraper is a forum scraper that searches twenty-one sources from one keyword and returns public discussions with complete Markdown content, title, link, author, timestamp, score, and reply count in one normalized shape. One source searches the open web, which is how threads on forums that publish no API are reached at all; the other twenty query their communities' own search surfaces, so rows can carry source-published engagement instead of only a result snippet.

• Twenty-one sources behind one input. Bing for the open web, plus Hacker News, five Stack Exchange sites, two Lemmy instances, five Discourse communities, and seven developer communities in Japanese, Chinese, Korean and Russian.
• Twenty-one documented fields per row. Platform, forum domain, title, canonical and linked-source URLs, list excerpt, complete Markdown content, author, dates, engagement, state, tags, thread ID, and processing provenance.
• One row per discussion, not per source. A thread surfaced by both a search engine and a forum's own API is saved once, keyed on its canonical address.
• Source values only. A search engine knows a thread's title and address but not its score, and those fields stay empty rather than being filled with a zero that would read as a measurement.

The smallest useful run requests one discussion and costs $0.0415 on the FREE tier: $0.04 for Actor Start plus $0.0015 for one Result.

Why Choose All Forums Thread Scraper

One keyword reaches two kinds of source. Forums running XenForo, phpBB, or vBulletin publish no search API, so a web search engine is the only public index of them. Forums that do publish an API return the thread's own numbers. Reading both in one run is what makes a keyword sweep of the forums mean something rather than one platform.

Every row has the same shape and readable content. Switching sources changes where rows come from, never the twenty-one keys they arrive under. The excerpt keeps the source's optional list snippet, while content contains the complete Markdown document fetched for the saved discussion.

Engagement and state where the source publishes them. The official Stack Exchange API documents question fields including score, answer_count, view_count, and is_answered; those values pass through unchanged, including negative scores. Hacker News and Lemmy publish score and reply totals, Discourse publishes reply, solved, and closed state, and Habr, Qiita, Zenn, and Juejin add source-specific view or bookmark totals.

Selectable sources. Leave the platform list empty to read all twenty-one, or narrow it to the ones that fit your subject. A sysadmin question and a Rust question do not need the same forums.

Depth where the source publishes it. Every source is paged one page at a time until your max_results is met or it runs out of matching threads, so a large request reaches past a first page instead of stopping at it — measured at 150 discussions from a single community endpoint. Bing runs dry sooner than the community APIs do: one keyword yields roughly 16 to 60 open-web results before it stops producing new ones. A source that refuses never ends the search; the others still deliver.

Quick Start Guide

Configure

Open the Actor input, enter a keyword, and set max_results. Leave platforms empty to search every source, or select a subset. The input is prefilled with a working rust async runtime example on Hacker News.

Run

Click Start. Selected sources are searched together, and each accepted discussion appears in the Dataset as it is produced.

Collect

Open the Dataset and export JSON, CSV, Excel, or XML, or read it through the API. Rows are deduplicated on the canonical thread address before they are saved.

Input Parameters

All Forums Thread Scraper takes three inputs: what to search for, which sources to read, and how many discussions to keep.

Parameter	Type	Required	Description	Example
keyword	String, 1–200 characters	Yes	The topic, product, error message, or phrase to search for. Every selected source receives the same words.	rust async runtime
platforms	Array of strings, 21 values	No	Which sources to read. Empty means all twenty-one. Each value is also what the row's platform field carries.	["hacker_news"]
max_results	Integer, minimum 1	Yes	How many discussions to save from each selected source. No maximum, and every source gets the same budget, so eight sources at 25 return up to 200 rows.	1

{
  "keyword": "rust async runtime",
  "platforms": ["hacker_news"],
  "max_results": 1
}

The twenty-one platforms values are bing, hacker_news, stack_overflow, super_user, server_fault, ask_ubuntu, unix_linux, lemmy_world, programming_dev, discourse_meta, python_discuss, rust_users, openai_community, obsidian_forum, qiita, zenn, juejin, segmentfault, v2ex, velog, and habr.

Output Data Schema

One Dataset row represents one public discussion thread, with all twenty-one documented fields and its complete Markdown content.

Group	Fields
Source	platform, site
Thread	title, url, source_url, excerpt, content, thread_id
Attribution	author, posted_at, active_at
Engagement	score, reply_count, view_count, bookmark_count
State	is_answered, is_closed, is_nsfw
Classification	tags
Processing	processor, processed_at

platform carries the same value the platforms input accepts, so filtering the Dataset by source uses the identical string you selected it with, and site carries the forum's own domain for reading.

This abbreviated example shortens content — the real row for this thread carries 136,930 Markdown characters. A run follows the same twenty-one-field contract, keeps the complete document, and leaves source fields empty rather than guessing them: this thread is a plain Hacker News story, so tags is empty, while a Show HN or Ask HN thread carries that label.

{
  "platform": "hacker_news",
  "site": "news.ycombinator.com",
  "title": "The State of Async Rust: Runtimes",
  "url": "https://news.ycombinator.com/item?id=37639896",
  "source_url": "https://corrode.dev/blog/async",
  "excerpt": null,
  "content": "# The State of Async Rust: Runtimes\n\nFull discussion content continues in Markdown...",
  "author": "RebootStr",
  "posted_at": "2023-09-25T05:20:13+00:00",
  "active_at": "2026-08-02T17:15:09+00:00",
  "score": 258,
  "reply_count": 198,
  "view_count": null,
  "bookmark_count": null,
  "is_answered": null,
  "is_closed": null,
  "is_nsfw": null,
  "tags": [],
  "thread_id": "37639896",
  "processor": "https://apify.com/username/actor_name",
  "processed_at": "2026-09-01T13:03:11+00:00"
}

Export through the Apify Console, API, webhooks, integrations, or storage clients. Keep string IDs as strings so spreadsheet and JavaScript consumers do not lose precision.

Integration Examples

All Forums Thread Scraper runs by its name-form Actor ID from the API, Apify clients, Make, n8n, or MCP, using the same rust async runtime scenario throughout.

Actor ID

The Actor's API ID is bhyPmTSBK5mkWPUwp; the name form agentx/all-forums-thread-scraper works everywhere the ID does.

bhyPmTSBK5mkWPUwp

HTTP

Start the run, poll it to completion, then read the Dataset.

curl -X POST "https://api.apify.com/v2/acts/agentx~all-forums-thread-scraper/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keyword":"rust async runtime","platforms":["hacker_news"],"max_results":1}'

curl "https://api.apify.com/v2/actor-runs/RUN_ID?token=YOUR_APIFY_TOKEN"

curl "https://api.apify.com/v2/datasets/DATASET_ID/items?token=YOUR_APIFY_TOKEN"

Poll the second call until status is SUCCEEDED, or subscribe to an ACTOR.RUN.SUCCEEDED webhook instead of polling, then read defaultDatasetId from the run and pass it to the third call.

Quick test only. For a small input that finishes well inside 300 seconds you can collapse the three calls into one:

curl -X POST "https://api.apify.com/v2/acts/agentx~all-forums-thread-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keyword":"rust async runtime","platforms":["hacker_news"],"max_results":1}'

The platform cuts that connection at 300 seconds without aborting the run, a timeout on your side never stops a run that is already billing, and each client or gateway retry starts a new billed run. Use the asynchronous flow above for anything larger.

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
started = client.actor("agentx/all-forums-thread-scraper").start(run_input={
    "keyword": "rust async runtime",
    "platforms": ["hacker_news"],
    "max_results": 1,
})
run = client.run(started["id"]).wait_for_finish()
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["platform"], item["title"], item["url"])

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const started = await client.actor('agentx/all-forums-thread-scraper').start({
  keyword: 'rust async runtime',
  platforms: ['hacker_news'],
  max_results: 1,
});
const run = await client.run(started.id).waitForFinish();
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Make.com

Add an Apify Run an Actor module, select Actor ID bhyPmTSBK5mkWPUwp, and map keyword, platforms, and max_results. Read Dataset items from the returned default Dataset ID.

n8n

Use an HTTP Request node with POST https://api.apify.com/v2/acts/bhyPmTSBK5mkWPUwp/runs, Bearer authentication, and the same three-key JSON body. Poll the returned run ID until completion, then read the run's defaultDatasetId and fetch its items.

MCP

Ask the Apify MCP server to run agentx/all-forums-thread-scraper with {"keyword":"rust async runtime","platforms":["hacker_news"],"max_results":1}. The twenty-one-field schema exposes complete Markdown content, titles, links, authors, engagement, and source-published state without guessing field names.

Pricing

Each saved discussion costs $0.0015 on the FREE tier, plus $0.04 of Actor Start per run, so the smallest useful run is $0.0415.

Tier	Result	Actor Start (per GB of run memory)
FREE	$0.00150	$0.01
BRONZE	$0.00135	$0.01
SILVER	$0.00120	$0.01
GOLD	$0.00113	$0.01
PLATINUM	$0.00113	$0.01
DIAMOND	$0.00113	$0.01

The billing unit is one saved discussion. Actor Start is charged once per gigabyte of run memory with a minimum of one event; this Actor runs at 4 GB, so every run pays four events, 4 × $0.01 = $0.04, on every tier. A fifty-discussion run therefore costs $0.04 + 50 × $0.0015 = $0.115 on the FREE tier. Selecting more sources does not raise the per-row price or add Actor Start events; it raises how many distinct discussions one keyword can find, so one run across all twenty-one sources costs less than twenty-one single-source runs. Prices can change — see the pricing page for current values.

Use Cases

• Track how a product is discussed. Search a product or feature name, then group rows by site to see which communities carry the conversation and which stay quiet.
• Find prior art for an error message. Paste the error text and read Stack Exchange, Hacker News, and Discourse together instead of searching each one.
• Build a support knowledge base. Index content as Markdown, keep url and posted_at for provenance, and revisit the same keyword on a schedule.
• Research a technical decision. Compare what a developer forum, a federated community, and the open web say about the same library before choosing it.
• Monitor a niche community. Narrow platforms to the forums that matter for your subject and run the same keyword regularly.

Alternatives

Searching each forum by hand is genuinely fine for one question, and it is the only way to read tone and context properly. It stops scaling the moment the same query has to run across twenty-one sources on a schedule.

A forum's own search page gives you everything that forum knows, including anything this Actor does not model. Use it when a single community is the whole scope.

A general web search API returns a wider slice of the internet with no forum structure over it. This Actor is narrower on purpose: twenty community search surfaces that return the thread's own numbers, plus one open-web pass for the forums that publish nothing to query.

Choose something else if you need comments as separate nested records, private or login-gated communities, real-time streaming, or a guarantee that every forum on the web is covered. This Actor returns each discussion as one Markdown document from twenty-one named sources.

Limits and Troubleshooting

• Fewer rows than max_results → the keyword may simply have fewer public discussions, or a narrow platforms selection may not cover the subject → widen the platform list or broaden the keyword.
• A large max_results is filled mostly by the community sources → every source is paged until your number is met, but Bing's open-web result list runs out after roughly four to eight pages while a community API keeps serving → keep Bing selected for the long tail of forums with no API, and rely on the community endpoints for depth.
• A source contributes nothing → relevance, not failure: sysadmin sites return nothing for a Rust question and a Rust forum returns nothing for an SSH question → check the run log, which reports what each source returned.
• A Bing row that is not a discussion thread → Bing is searched with exactly your keyword and returns whatever the open web ranks for it, so a documentation page or a blog post can appear beside forum threads → filter the Dataset on site, or leave bing out of platforms when only community threads are wanted.
• Empty score or reply_count → search-engine rows carry no engagement numbers, and some forum APIs omit them per thread → treat empty as source absence, never as zero.
• Duplicate-looking rows → two threads on different forums can share a title; deduplication is on the canonical address, so both are kept deliberately.

Report a reproducible problem as an Issue with the run ID and the exact input.

Trust and Reliability

Runs execute on Apify's platform with pay-per-event billing, and every saved row follows the documented twenty-one-field Dataset contract. The complete content document comes from the saved discussion; other fields stay empty when the source does not publish them, and negative scores remain unchanged.

Legal and Compliance

Data scope. The Actor reads public forum search results and public forum search endpoints. It does not sign in, and it does not reach private, member-only, or login-gated content.

Privacy. Rows carry the author name a forum publishes beside a public thread and nothing more. If you process personal data from the output, that processing is yours to justify.

Platform terms. You are responsible for complying with each search engine, each forum, copyright, database, and applicable data-protection rules. Use a reasonable run frequency and do not use results for unlawful profiling, harassment, or rights infringement.

Frequently Asked Questions

How do I search forum posts across multiple forums?

Enter one keyword, leave platforms empty so all twenty-one sources are read, and set max_results. Each source is searched with the same words, and results arrive under one shared twenty-one-field shape with complete Markdown content.

Is there a multi platform forum search api?

Yes — this Actor is one. Call agentx/all-forums-thread-scraper over the Apify API with a keyword and an optional source list, then read the default Dataset. There is no separate key for any of the twenty-one sources.

Which forums does it actually cover?

Bing for the open web, Hacker News, five Stack Exchange sites (Stack Overflow, Super User, Server Fault, Ask Ubuntu, Unix & Linux), two Lemmy instances (lemmy.world, programming.dev), five Discourse communities (Discourse Meta, Python Discuss, Rust Users, OpenAI Community, Obsidian Forum), and seven communities where developers write in their own language: Qiita and Zenn in Japanese, Juejin, SegmentFault and V2EX in Chinese, velog in Korean, and Habr in Russian.

Is there a free forum API?

The APIs behind several of these sources are publicly readable, but each has its own address, response shape, and rate limit. This Actor gives you one input contract and one row shape over all of them instead.

Can I schedule runs to monitor discussions over time?

Yes. Schedule the same input and compare url and posted_at between runs; addresses that were not in the previous Dataset are new discussions.

Why do some rows have no score?

Because a search engine result does not carry one. Hacker News, Stack Exchange, Lemmy, Discourse, Qiita, Zenn, Juejin, velog, and Habr all publish a score that passes through unchanged — on the five Discourse communities it is the topic's like count, which is genuinely 0 on most threads and stays 0 rather than becoming empty. V2EX publishes replies but no score. Bing and SegmentFault rows therefore keep score empty instead of filling it with a search rank or another substitute.

How many discussions can one source return?

As many as max_results asks for, up to what that source publishes. Every source is paged until your number is met or it has no more matching threads — measured at 150 discussions from a single community endpoint where the keyword had that many public threads. Two sources stop earlier because their own result lists do: Bing yields roughly 16 to 60 open-web results for one keyword, and Hacker News search returns at most 1,000 for any keyword however many are requested.

Does it return the replies inside a thread?

It returns the discussion as one Markdown content document, including publicly available replies on the fetched surface. It does not split replies into separate nested Dataset records.

Related AgentX Actors

The AgentX catalog contains 79 Actors; start with the three closest discussion tools, then browse the other 78 by category.

Closest to this Actor:

• Reddit Viral Scraper — search one large community by keyword when Reddit alone is the scope.
• Subreddit Posts Scraper — export a known community's post feed rather than searching across forums.
• Medium Scraper — collect long-form articles when the answer is written up rather than discussed.

Business and Market Intelligence

• All Web Search Scraper — multi-engine web search across ten independent indexes
• Google Keyword Trends — Google Trends keyword interest, compared five at a time
• Google Trends Scraper — Google Trends trending searches
• Google Maps Store Scraper — Google Maps business listings
• LinkedIn Company Lookup — LinkedIn company records
• LinkedIn Profile Lookup — LinkedIn profile records
• Weather Forecast API — forecast and historical weather data

Jobs and Hiring

• All Jobs Scraper — multi-platform job listings
• Bayt Jobs Scraper — Bayt job listings
• Glassdoor Jobs Scraper — Glassdoor job listings
• Indeed Jobs Scraper — Indeed job listings
• JobStreet Company Profile — JobStreet employer records
• Jooble Jobs Scraper — Jooble job listings
• LinkedIn Jobs Scraper — LinkedIn job listings
• Naukri Jobs Scraper — Naukri job listings
• Talent Jobs Scraper — Talent.com job listings
• ZipRecruiter Jobs Scraper — ZipRecruiter job listings

Social Media

• Instagram Creator Scraper — Instagram creator profiles
• Instagram Reels Scraper — Instagram Reels metadata
• Instagram Trending Scraper — Instagram trending content
• Medium Scraper — Medium articles
• Medium Profile Scraper — Medium author profiles
• Reddit Profile Scraper — Reddit user profiles
• Reddit Viral Scraper — Reddit viral posts
• SubReddit Info Scraper — subreddit metadata
• Subreddit Members Scraper — subreddit member lists
• Subreddit Posts Scraper — subreddit post feeds
• Telegram Chat Scraper — Telegram chat messages
• Telegram Info Scraper — Telegram channel metadata
• Telegram Member Adder — Telegram group membership management
• Telegram Member Scraper — Telegram group members
• Telegram Private Group Scraper — private Telegram group data
• TikTok Creator API — TikTok creator profiles
• TikTok User Lookup — TikTok account records
• X Twitter Community API — X community data
• X Twitter Profile Lookup — X profile records
• YouTube Creator Email Scraper — YouTube creator contact data

Video, Transcripts and Downloads

• All Video Scraper — multi-platform video metadata
• Video Transcript — multi-platform transcripts
• Video Captions Downloader — caption files
• Video to Social Post — video repurposed into social copy
• YouTube Transcript — YouTube transcripts
• TikTok Transcript — TikTok transcripts
• X Twitter Transcript — X video transcripts
• Facebook Transcript — Facebook video transcripts
• Bilibili Transcript — Bilibili transcripts
• Dailymotion Transcript — Dailymotion transcripts
• Rutube Transcript — Rutube transcripts
• Loom Transcript — Loom transcripts
• Wistia Transcript — Wistia transcripts
• Instagram Reels Downloader — Reels downloads
• Kick Clip Downloader — Kick clip downloads
• Linkedin Video Downloader — LinkedIn video downloads
• Pinterest Video Downloader — Pinterest video downloads
• Reddit Video Downloader — Reddit video downloads
• Snapchat Video Downloader — Snapchat video downloads
• TED Talk Downloader — TED talk downloads
• TikTok Live Downloader — TikTok live downloads
• Twitch VOD Downloader — Twitch VOD downloads
• Zoom Recording Downloader — Zoom recording downloads

E-Commerce and Retail

• All Shopping Scraper — multi-platform product data
• AliExpress Product Scraper — AliExpress products
• Amazon Storefront Scraper — Amazon Brand Store pages
• Bol Product Scraper — Bol.com products
• eBay Seller Leads API — eBay seller records
• Hepsiburada Product Scraper — Hepsiburada products
• Kakaku Product Scraper — Kakaku.com products
• Rakuten Product Scraper — Rakuten products

Classifieds and Automotive

• All Vehicle Scraper — multi-platform vehicle listings
• AutoTrader Vehicle Search — AutoTrader US listings
• AutoTrader UK Vehicle Search — AutoTrader UK listings

Real Estate

• All Property Scraper — multi-platform property listings
• Homes Property Scraper — Homes.com listings
• MagicBricks Property Scraper — MagicBricks listings
• NoBroker Property Scraper — NoBroker listings
• Property24 Property Scraper — Property24 listings
• Realtor Property Scraper — Realtor.com listings
• Zillow Property Scraper — Zillow listings

Support and Community

Ask about forum sources, Dataset output, or Actor integrations in the AgentX community on Telegram; for a reproducible bug, open an Issue with the run ID and the exact input.

AgentX is an Arcyton brand — arcyton.com.

Last Updated: September 3, 2026