# Discourse Forum Scraper (`automation-lab/discourse-scraper`) Actor
Extract topics, posts, and discussions from any public Discourse forum. Supports latest topics, category filtering, and keyword search. No login required.
- **URL**: https://apify.com/automation-lab/discourse-scraper.md
- **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community)
- **Categories:** AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks
- **User rating**: No ratings yet
## Pricing
Pay per event
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Discourse Forum Scraper
Extract topics, posts, and community discussions from **any public Discourse forum** — including community.openai.com, discuss.huggingface.co, meta.discourse.org, and 5,000+ other Discourse communities worldwide. No API key required.
### 🔍 What does it do?
Discourse Forum Scraper connects to Discourse's public JSON API (no authentication required for public forums) and extracts:
- **Latest topics** — trending and most recent discussions paginated across the entire forum
- **Category topics** — all topics within a specific category or sub-category
- **Search results** — topics matching a keyword or phrase
For each topic, you get structured metadata: title, URL, author, view count, like count, post count, category, tags, excerpt, creation date, and more. Optionally enable **Include Post Content** to fetch the full post body for each topic (including HTML content, poster info, and reaction counts).
The actor uses only Discourse's public JSON API endpoints (`/latest.json`, `/c/{category}.json`, `/search.json`, `/t/{id}.json`) — no browser, no proxies, minimal cost per result.
### 👥 Who is it for?
**AI/ML researchers** who want to monitor community.openai.com or discuss.huggingface.co for emerging topics, model comparisons, and developer pain points — without manually browsing the forum every day.
**Product managers** tracking competitor mentions, feature requests, and user feedback across developer communities. Discourse forums are where power users discuss product issues openly.
**Content marketers and SEO teams** identifying high-traffic discussion topics in their niche, repurposing forum Q&As into blog posts, or monitoring what questions get thousands of views.
**Data scientists and NLP researchers** building training datasets from community discussions, sentiment analysis corpora, or topic clustering studies on domain-specific text.
**Community managers** bulk-exporting discussions for archiving, migration, compliance audits, or import into internal knowledge bases.
**Competitive intelligence analysts** monitoring developer forums for product feedback, bug reports, and integration questions targeting specific categories.
### 💡 Why use it?
Discourse has over 5,000 active public forums across tech, gaming, academia, open-source, and more — but there's no centralized API that lets you query across all of them. This actor gives you a uniform interface to any Discourse forum, producing clean structured JSON you can pipe directly into spreadsheets, databases, or downstream APIs.
Compared to manually browsing Discourse or writing one-off scripts:
- ✅ Works on **any** Discourse forum — just change the URL
- ✅ Handles pagination automatically — get 5,000+ topics with one run
- ✅ Clean JSON output compatible with Google Sheets, Airtable, BigQuery, and more
- ✅ Optional post content extraction for full discussion text
- ✅ Category name resolution — readable `categoryName` field, not just an ID
- ✅ No API keys, no rate limit headaches, no authentication setup
### 📊 What data does it extract?
| Field | Description | Example |
|-------|-------------|---------|
| `topicId` | Unique topic ID | `137192` |
| `title` | Topic title | `"Best practices for GPT-4 system prompts"` |
| `slug` | URL-friendly slug | `"best-practices-for-gpt-4-system-prompts"` |
| `url` | Direct link to the topic | `"https://community.openai.com/t/..."` |
| `categoryId` | Category numeric ID | `7` |
| `categoryName` | Resolved category name | `"API"` |
| `postsCount` | Number of replies | `24` |
| `replyCount` | Direct reply count | `18` |
| `views` | Total view count | `12503` |
| `likeCount` | Total likes received | `89` |
| `createdAt` | ISO timestamp created | `"2023-04-03T10:23:49.213Z"` |
| `lastPostedAt` | ISO timestamp last reply | `"2024-01-15T08:44:11.000Z"` |
| `tags` | List of topic tags | `["gpt-4", "system-prompt"]` |
| `excerpt` | Short text excerpt | `"Has anyone found a good pattern for..."` |
| `authorUsername` | Original poster username | `"johndev42"` |
| `pinned` | Is topic pinned | `false` |
| `closed` | Is topic closed | `false` |
| `posts` | Post array (optional) | See below |
**When `includePostContent: true`**, each topic includes a `posts` array:
| Field | Description |
|-------|-------------|
| `postId` | Unique post ID |
| `postNumber` | Position in thread |
| `username` | Poster's username |
| `displayName` | Poster's display name |
| `createdAt` | Post creation timestamp |
| `content` | Full HTML post content |
| `likeCount` | Number of likes on this post |
| `replyCount` | Direct replies to this post |
| `reads` | How many users read this post |
| `isAcceptedAnswer` | Is this the accepted answer? |
### 💰 How much does it cost to scrape Discourse forum topics?
This actor uses **pay-per-event (PPE) pricing** — you pay only for the data you extract, not for idle time.
| Plan | Price per topic |
|------|----------------|
| FREE | $0.00115 |
| BRONZE | $0.001 |
| SILVER | $0.00078 |
| GOLD | $0.0006 |
| PLATINUM | $0.0004 |
| DIAMOND | $0.00028 |
Plus a flat **$0.005 start fee** per run.
**Cost examples:**
- 100 topics: ~$0.11 (FREE tier) / $0.105 (BRONZE)
- 500 topics: ~$0.585 (FREE) / $0.505 (BRONZE)
- 2,000 topics: ~$2.31 (FREE) / $2.005 (BRONZE)
- 5,000 topics with posts: ~$5.78 (FREE) / $5.005 (BRONZE)
Topics-only runs are cheap. Enabling `includePostContent` doubles API calls (one extra request per topic) but the PPE charge per result stays the same.
**Free plan estimate:** Apify Free plan ($0) includes $5/month in usage, which covers ~4,000 topics per month at BRONZE rates.
### 🚀 How to use it
#### Step 1: Choose your target forum
Find the base URL of any public Discourse forum. Examples:
- `https://community.openai.com` — OpenAI Developer Community
- `https://discuss.huggingface.co` — HuggingFace forums
- `https://meta.discourse.org` — Discourse Meta
- `https://forum.cursor.sh` — Cursor AI community
- `https://community.cloudflare.com` — Cloudflare developers
#### Step 2: Select a scrape mode
- **Latest Topics** — get the freshest discussions sorted by most recently active
- **Category Topics** — get topics from a specific category (requires the category slug, found in the URL)
- **Search Topics** — search for topics matching a keyword
#### Step 3: Set limits
Start small with `maxTopics: 20` to preview the data. Scale up to 5,000 for bulk exports.
#### Step 4: Enable post content (optional)
Set `includePostContent: true` and `maxPostsPerTopic: 10` to fetch the actual post bodies. Best for NLP datasets, content archiving, and answer extraction.
#### Step 5: Run and export
Click **Start** and wait for results. Export to JSON, CSV, or Excel. Connect to Google Sheets or Airtable directly from the Apify platform.
### ⚙️ Input parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `forumUrl` | string | `https://community.openai.com` | Base URL of the Discourse forum |
| `scrapeMode` | enum | `latestTopics` | What to scrape: `latestTopics`, `categoryTopics`, `searchTopics` |
| `categorySlug` | string | — | Category slug for `categoryTopics` mode (e.g., `api`) |
| `searchQuery` | string | — | Search keywords for `searchTopics` mode |
| `maxTopics` | integer | `50` | Maximum number of topics to extract |
| `includePostContent` | boolean | `false` | Fetch full post content for each topic |
| `maxPostsPerTopic` | integer | `10` | Max posts per topic (when `includePostContent` is true) |
| `maxRequestRetries` | integer | `3` | Retry attempts for failed HTTP requests |
### 📤 Output example
```json
{
"topicId": 137192,
"title": "Cant find and use gpt-4 model (I have the gpt-4 Invitation)",
"slug": "cant-find-and-use-gpt-4-model-i-have-the-gpt-4-invitation",
"url": "https://community.openai.com/t/cant-find-and-use-gpt-4-model/137192",
"categoryId": 7,
"categoryName": "API",
"postsCount": 15,
"replyCount": 12,
"views": 8432,
"likeCount": 23,
"createdAt": "2023-04-03T10:23:49.213Z",
"lastPostedAt": "2023-05-12T14:31:00.000Z",
"tags": ["gpt-4", "access"],
"excerpt": "Hi everyone, I recently received an invitation for GPT-4 access...",
"authorUsername": "johndev42",
"pinned": false,
"closed": false
}
````
### 💡 Tips and tricks
**Finding the category slug:** Navigate to a category page and check the URL. For `https://community.openai.com/c/api/7`, the slug is `api`. For subcategories like `/c/api/plugins/23`, use `api/plugins`.
**Filtering noisy topics:** Use `searchTopics` mode with specific technical keywords to get only relevant discussions. For example, `"function calling"` returns topics specifically about that feature.
**Scraping multiple forums:** Run the actor multiple times with different `forumUrl` values. Use the Apify API to start runs programmatically and merge results.
**Pagination behavior:** The actor automatically paginates. Set `maxTopics: 5000` to get the maximum — Discourse's `/latest.json` returns ~30 topics per page, so 5,000 topics requires ~167 API calls.
**Rate limiting:** If you get retried errors or timeouts, reduce throughput by lowering `maxTopics` per run or increasing `maxRequestRetries`. Most Discourse forums have no strict rate limits for anonymous JSON requests.
**searchTopics mode — limited fields:** Discourse's `/search.json` endpoint returns slim topic objects that do not include `views`, `authorUsername`, or `excerpt`. These fields will be empty (`""` or `0`) for search results. This is a Discourse API limitation, not a scraper bug. If you need these fields, use `latestTopics` or `categoryTopics` mode and filter by keyword on your end, or enable `includePostContent: true` (which fetches full topic data per result).
**Private forums:** This actor only works with **public** Discourse forums. Forums requiring login to browse will return empty results or redirect to a login page.
### 🔗 Integrations
#### Workflow: Daily topic monitoring via Google Sheets
Schedule this actor daily, point output to a Google Sheets integration (Apify has a built-in Sheets connector), and track emerging topics over time. Set `scrapeMode: latestTopics` and `maxTopics: 50` for a lightweight daily snapshot.
#### Workflow: Competitive intelligence pipeline
Run `searchTopics` mode against competitor brand names across multiple forums. Feed output into an LLM summarizer to extract sentiment and pain points. Export a weekly digest.
#### Workflow: NLP training dataset creation
Enable `includePostContent: true` and `maxPostsPerTopic: 20` for a specific forum category. Output gives you structured conversation threads with HTML content, which you can clean for fine-tuning or RAG ingestion.
#### Workflow: Community migration
Export all topics and posts from an old Discourse installation before migration. The structured JSON output makes it easy to transform and re-import into a new forum or knowledge base platform.
#### Workflow: SEO content research
Run `searchTopics` mode on niche Discourse forums with target keywords. Topics with high `views` and `likeCount` but few answers are prime candidates for content marketing — write the definitive answer and link back.
### 🔌 API usage
#### Node.js (Apify client)
```javascript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('automation-lab/discourse-scraper').call({
forumUrl: 'https://community.openai.com',
scrapeMode: 'searchTopics',
searchQuery: 'function calling',
maxTopics: 100,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Scraped ${items.length} topics`);
```
#### Python
```python
from apify_client import ApifyClient
client = ApifyClient(token='YOUR_APIFY_TOKEN')
run = client.actor('automation-lab/discourse-scraper').call(run_input={
'forumUrl': 'https://discuss.huggingface.co',
'scrapeMode': 'categoryTopics',
'categorySlug': 'research',
'maxTopics': 200,
'includePostContent': True,
'maxPostsPerTopic': 5,
})
items = list(client.dataset(run['defaultDatasetId']).iterate_items())
print(f'Scraped {len(items)} topics with full post content')
```
#### cURL
```bash
curl -X POST "https://api.apify.com/v2/acts/automation-lab~discourse-scraper/runs?token=YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"forumUrl": "https://community.openai.com",
"scrapeMode": "latestTopics",
"maxTopics": 50
}'
```
### 🤖 MCP (AI assistant integration)
Use this scraper directly inside **Claude Code**, **Claude Desktop**, **Cursor**, or any MCP-compatible AI assistant.
#### Claude Code (terminal)
```bash
claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/discourse-scraper"
```
#### Claude Desktop / Cursor / VS Code
Add to your MCP config file:
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com?tools=automation-lab/discourse-scraper",
"headers": {
"Authorization": "Bearer YOUR_APIFY_TOKEN"
}
}
}
}
```
#### Example AI prompts
Once connected, try these prompts in your AI assistant:
- *"Scrape the latest 50 topics from community.openai.com and show me the ones with over 1,000 views."*
- *"Search discuss.huggingface.co for topics about 'fine-tuning Llama' and give me a summary of the top discussions."*
- *"Get the most recent topics from the 'api' category on community.openai.com and find any that mention authentication errors."*
- *"Export all topics from a Discourse forum to a CSV for analysis."*
### ⚖️ Legality — Is it Legal to Scrape Discourse Forums?
This actor only accesses **public** Discourse forums — the same content visible to any anonymous visitor. It uses Discourse's documented public JSON API, which is explicitly designed for programmatic access. No login, no authentication bypass, and no private data is accessed.
The data extracted is publicly visible on the web. Always review the specific forum's Terms of Service before scraping at large scale. For commercial data re-use or republication, consult the forum's terms.
This actor does **not** scrape private messages, user email addresses, IP addresses, or any non-public data.
### ❓ FAQ
**Q: Which Discourse forums does this work with?**
A: Any public Discourse forum — the software powers thousands of communities including Rust, Ruby, Docker, Wikipedia, and major AI/ML communities. If you can browse topics without logging in, this actor can scrape it.
**Q: Does it require an API key?**
A: No. Public Discourse forums expose a JSON API to anonymous visitors. Just provide the forum URL and run.
**Q: Why is `categoryName` showing a number instead of a name?**
A: This can happen if the forum restricts the `/categories.json` endpoint. The actor falls back to the category ID when the name can't be resolved. Try with a different forum or check if categories require login to view.
**Q: Can I scrape a forum that requires login to view topics?**
A: No — this actor doesn't support authentication. It only works with publicly accessible Discourse forums.
**Q: The run finished but I got fewer topics than expected. Why?**
A: The forum may have fewer public topics than your `maxTopics` limit, or the category you specified has fewer topics. For `searchTopics` mode, the search results are naturally limited by relevance — try a broader search query.
**Q: Can I get topics from subcategories?**
A: Yes. For nested categories, use the full path in `categorySlug`. For example, if the URL is `/c/parent/child/12`, use `parent/child` as the slug.
### 🔗 Related scrapers
Explore other automation-lab scrapers for AI/developer community data:
- [AI Jobs Scraper](https://apify.com/automation-lab/aijobs-scraper) — AI/ML job listings from aijobs.net
- [AIcrowd Scraper](https://apify.com/automation-lab/aicrowd-scraper) — AI competition listings from AIcrowd
- [DevPost Scraper](https://apify.com/automation-lab/devpost-scraper) — hackathon and project listings
- [AI Tools Directory Scraper](https://apify.com/automation-lab/ai-tools-directory-scraper) — AI tool listings
# Actor input Schema
## `forumUrl` (type: `string`):
Base URL of the Discourse forum to scrape (e.g., https://community.openai.com).
## `scrapeMode` (type: `string`):
What to scrape from the forum. Latest Topics fetches the most recent topics. Category Topics fetches topics from a specific category. Search Topics searches by keyword.
## `categorySlug` (type: `string`):
Category slug for Category Topics mode. Found in the category URL, e.g., api from .../c/api/.
## `searchQuery` (type: `string`):
Keywords to search for (only used in Search Topics mode).
## `maxTopics` (type: `integer`):
Maximum number of topics to scrape. Each topic costs $0.001.
## `includePostContent` (type: `boolean`):
Fetch full post content for each topic. Each topic requires one additional API request.
## `maxPostsPerTopic` (type: `integer`):
Maximum posts per topic (only when Include Post Content is enabled).
## `maxRequestRetries` (type: `integer`):
Number of retry attempts for failed HTTP requests.
## Actor input object example
```json
{
"forumUrl": "https://community.openai.com",
"scrapeMode": "latestTopics",
"maxTopics": 20,
"includePostContent": false,
"maxPostsPerTopic": 5,
"maxRequestRetries": 3
}
```
# Actor output Schema
## `overview` (type: `string`):
No description
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"forumUrl": "https://community.openai.com",
"scrapeMode": "latestTopics",
"categorySlug": "",
"searchQuery": "",
"maxTopics": 20,
"includePostContent": false,
"maxPostsPerTopic": 5,
"maxRequestRetries": 3
};
// Run the Actor and wait for it to finish
const run = await client.actor("automation-lab/discourse-scraper").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"forumUrl": "https://community.openai.com",
"scrapeMode": "latestTopics",
"categorySlug": "",
"searchQuery": "",
"maxTopics": 20,
"includePostContent": False,
"maxPostsPerTopic": 5,
"maxRequestRetries": 3,
}
# Run the Actor and wait for it to finish
run = client.actor("automation-lab/discourse-scraper").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"forumUrl": "https://community.openai.com",
"scrapeMode": "latestTopics",
"categorySlug": "",
"searchQuery": "",
"maxTopics": 20,
"includePostContent": false,
"maxPostsPerTopic": 5,
"maxRequestRetries": 3
}' |
apify call automation-lab/discourse-scraper --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=automation-lab/discourse-scraper",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Discourse Forum Scraper",
"description": "Extract topics, posts, and discussions from any public Discourse forum. Supports latest topics, category filtering, and keyword search. No login required.",
"version": "0.1",
"x-build-id": "Fri9RJ48xC72rEx2E"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/automation-lab~discourse-scraper/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-automation-lab-discourse-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/automation-lab~discourse-scraper/runs": {
"post": {
"operationId": "runs-sync-automation-lab-discourse-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/automation-lab~discourse-scraper/run-sync": {
"post": {
"operationId": "run-sync-automation-lab-discourse-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"forumUrl",
"scrapeMode"
],
"properties": {
"forumUrl": {
"title": "🌐 Forum URL",
"pattern": "^https?://",
"type": "string",
"description": "Base URL of the Discourse forum to scrape (e.g., https://community.openai.com)."
},
"scrapeMode": {
"title": "📋 Scrape Mode",
"enum": [
"latestTopics",
"categoryTopics",
"searchTopics"
],
"type": "string",
"description": "What to scrape from the forum. Latest Topics fetches the most recent topics. Category Topics fetches topics from a specific category. Search Topics searches by keyword.",
"default": "latestTopics"
},
"categorySlug": {
"title": "📁 Category Slug",
"type": "string",
"description": "Category slug for Category Topics mode. Found in the category URL, e.g., api from .../c/api/."
},
"searchQuery": {
"title": "🔍 Search Query",
"type": "string",
"description": "Keywords to search for (only used in Search Topics mode)."
},
"maxTopics": {
"title": "📊 Max Topics",
"minimum": 1,
"maximum": 5000,
"type": "integer",
"description": "Maximum number of topics to scrape. Each topic costs $0.001.",
"default": 50
},
"includePostContent": {
"title": "💬 Include Post Content",
"type": "boolean",
"description": "Fetch full post content for each topic. Each topic requires one additional API request.",
"default": false
},
"maxPostsPerTopic": {
"title": "📝 Max Posts Per Topic",
"minimum": 1,
"maximum": 100,
"type": "integer",
"description": "Maximum posts per topic (only when Include Post Content is enabled).",
"default": 10
},
"maxRequestRetries": {
"title": "🔄 Max Request Retries",
"minimum": 1,
"maximum": 10,
"type": "integer",
"description": "Number of retry attempts for failed HTTP requests.",
"default": 3
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```