# Reddit Search Scraper (Relevant Posts Only) (`bustling_spotlight/reddit-search-scraper`) Actor
Reddit scraper that returns only genuinely relevant posts, not viral noise. Enter any keyword and get back only for relevant results. Great for social listening, market research, and lead generation.
- **URL**: https://apify.com/bustling\_spotlight/reddit-search-scraper.md
- **Developed by:** [Bustling Spotlight](https://apify.com/bustling_spotlight) (community)
- **Categories:** Social media, Lead generation, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $30.00 / 1,000 reddit search (per keyword)s
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Reddit Scraper – Relevant Post Search (No Noise)
**Every other Reddit scraper dumps raw, engagement-ranked results on you. This one returns only the posts you actually searched for.**
Reddit's own search is noise: look up *"best project management software"* and you get viral off-topic threads, a celebrity who shares a word with your query, and AITA drama. This **Reddit search scraper** fixes that — for each keyword you get a clean list of **genuinely relevant Reddit posts**, and you **pay only for relevant results**, not for the garbage.
---
### Why this Reddit scraper is different
| | Typical Reddit scrapers | **This actor** |
|---|---|---|
| Results | Everything the query touches (raw, noisy) | **Only genuinely relevant posts** |
| Ranking | Reddit's engagement/viral ranking | **Ranked by real relevance to your query** |
| You pay for | Every row, relevant or not | **Only relevant results** |
| Best for | Bulk dumps | Signal, not volume |
If you need a raw firehose, plenty of cheap scrapers do that. If you need **the right posts**, this is the one.
---
### What you can use it for
- **Social listening & brand monitoring** — track real discussions about a product, tool, or topic.
- **Market research** — see what people genuinely ask and recommend in a niche.
- **Lead generation** — find threads where people are actively looking for a solution.
- **Content & SEO ideas** — mine real questions from the communities that matter.
- **Competitor & product mentions** — surface relevant threads, skip the noise.
Post age comes free with every result, so there's **no slow per-thread scraping** — it's fast and cheap.
---
### Example output
One row per relevant post:
```json
{
"keyword": "best project management software",
"position": 1,
"title": "What project management tool does your team actually use?",
"url": "https://www.reddit.com/r/projectmanagement/comments/…",
"subreddit": "projectmanagement",
"posted_at": "2024-06-01T12:00:00.000Z",
"age_days": 730,
"score": 42,
"num_comments": 31,
"relevance_verified": true,
"coverage_gap": false
}
````
Export to JSON, CSV, Excel, or pull it via the Apify API.
***
### Input
| Field | Default | What it does |
|---|---|---|
| `keywords` | — (required) | Your search queries. |
| `postsPerKeyword` | 25 | Max relevant posts per keyword. |
| `sort` | `relevance` | Reddit ranking within each community: `relevance` / `hot` / `top` / `new` / `comments`. |
| `time` | `all` | Time window for `relevance` / `top` / `comments` (ignored for `hot` / `new`). |
| `domainHint` | — | Optional. Name your domain in a few words for ambiguous keywords; auto-detected per keyword if blank. |
| `proxyCountryCode` | `us` | Proxy country for reaching Reddit. |
Minimal run:
```json
{
"keywords": ["best project management software", "crm for small business"],
"postsPerKeyword": 25
}
```
***
### Pricing — pay per relevant result
No subscription. You pay per keyword searched plus per relevant post returned:
- **$0.03 per search** (per keyword)
- **$0.005 per relevant post** — *$5 per 1,000 relevant posts, priced on relevance, not volume*
You pay $0.03 per keyword plus $0.005 per relevant post. Example: 10 keywords averaging ~15 relevant posts each ≈ $0.30 + $0.75 = **~$1.05**. You're paying for signal — not pages of noise you have to sift yourself.
***
### FAQ
**How is this different from a normal Reddit scraper?**
Normal scrapers return everything a query matches. This one returns only genuinely relevant posts, so you pay for signal, not volume.
**Do I need a Reddit API key or account?**
No. Just enter your keywords and run.
**Can I scrape multiple keywords at once?**
Yes — add as many as you want to `keywords`; each is searched independently.
**Does it scrape comments?**
No. It returns relevant *posts* (with score, comment count, age, and URL). Run a comment scraper on the returned URLs if you need threads.
**Is scraping Reddit legal?**
It only accesses publicly available data. You are responsible for complying with Reddit's terms and applicable law in your use case.
***
Developer & self-host setup (for the actor owner)
#### Deploy
This actor is published under the Apify account **`bustling_spotlight`** → `https://apify.com/bustling_spotlight/reddit-search-scraper`.
1. Connect this GitHub repo to Apify: Console → *Develop new → Import from Git* (auto-rebuilds on every push), or one-off `apify push`. The included `Dockerfile` builds it.
2. Add secrets in *Actor → Settings → Environment variables* (mark **Secret**):
- `SCRAPINGBEE_API_KEY` — reaches Reddit via residential proxies.
- `ANTHROPIC_API_KEY` — the two AI relevance steps.
Keys are owner-only (never exposed to customers). Optional: set `LLM_MODEL` to override the default `claude-haiku-4-5`.
3. Publish with **Pay per event** and set these events (see `.actor/pay_per_event.json`):
| Event name | Price |
|---|---|
| `reddit-search` | $0.03 |
| `relevant-post` | $0.005 |
The code already calls `Actor.charge()` for these names — you only set the prices.
#### Store SEO checklist (set in the Console)
- **Categories:** `SOCIAL_MEDIA`, `LEAD_GENERATION`, `MARKETING`, `AI`.
- **Title / description:** already SEO-tuned in `.actor/actor.json` (keyword "Reddit Scraper" + differentiator).
- Keep the top of this README keyword-rich — Apify Store search and Google index it.
#### Local development
```bash
npm install
cp .env.example .env # add SCRAPINGBEE_API_KEY and ANTHROPIC_API_KEY
apify run # reads storage/key_value_stores/default/INPUT.json
```
#### How margin works
Your only costs are your ScrapingBee + Anthropic keys (~$0.005/keyword, dropping toward ~$0.003 at scale thanks to cached discovery and volume proxy plans; Apify covers platform compute out of its ~20% cut). At the prices above you net roughly **~75% margin**, and the per-search fee guarantees you never lose money on a low-yield keyword.
#### Architecture
```
src/main.js orchestration + pay-per-event charging
src/reddit.js Reddit endpoint URLs + response parsing (multi-query discovery)
src/scrapingbee.js residential-proxy fetch, retries, content gate
src/llm.js AI calls: intent, subreddit validation, title relevance
src/cache.js discovery cache (subreddits + intent, TTL)
src/select.js round-robin spread across subreddits
```
#### Operator dataset (`raw-candidates`) — re-classify without re-scraping
Every run also writes a **separate** `raw-candidates` dataset (not the customer output) with the *full* candidate set per keyword — `{ keyword, intent, entity, search_ok, coverage_gap, subreddits_searched, candidates: [{…, relevant, chosen}] }`. Because the raw titles are stored, you can re-run an improved relevance prompt over an entire past run for cents, with **zero** re-scraping. This is the single highest-leverage tuning lever — the relevance prompt will need iterating once you see real output.
Quality/observability built in:
- **Intent anchoring** — each keyword is expanded to searcher intent, and both the subreddit validator and the title judge reject same-word/different-domain collisions (crypto, piracy/debrid apps, music DAWs, celebrity, AITA, politics, sports). Pass an optional `domainHint` to sharpen it further.
- **Honest outcomes** — a blocked search (`search_ok:false`) is never recorded as a valid empty; "searched, found nothing" and "found few" are flagged `coverage_gap`. Zero relevant is a valid answer, not forced to N.
- **No rubber-stamping** — if relevance filtering is unavailable after a retry, posts are emitted flagged `relevance_verified:false` and **not billed**, instead of faking relevance.
- **Idempotent** — completed keywords are skipped on a resurrected run, so retries don't double-spend.
> Note: cross-keyword post dedup from the internal pipeline is intentionally **not** ported — this actor never thread-fetches (the expensive step), so a shared thread costs nothing extra to keep per keyword, and billing is per (keyword, post).
#### Reliability
- Residential proxies + retries (Reddit blocks datacenter IPs on `.json`), escalating to stealth on repeated blocks.
- Content gate: only real Reddit listings are accepted; a block page is retried, never counted as empty.
- Never a false empty: a blocked search or AI error is a logged per-keyword failure, not "0 relevant."
# Actor input Schema
## `keywords` (type: `array`):
One or more search queries. For each keyword the actor returns only the genuinely relevant Reddit posts — not viral, off-topic noise.
## `postsPerKeyword` (type: `integer`):
Maximum number of relevant posts to return per keyword. You are only charged for posts actually returned.
## `sort` (type: `string`):
How results are ranked. 'Relevance' is best for on-topic search; 'New' for fresh monitoring; 'Top'/'Hot' for popular discussion.
## `time` (type: `string`):
Time window applied when sorting by Relevance, Top, or Comments. Ignored for Hot and New.
## `domainHint` (type: `string`):
Optional. If your keywords are ambiguous, name the domain in a few words (e.g. 'B2B software' or 'content creation, podcasting, video') so results reject same-word matches from unrelated fields. Leave blank to auto-detect from each keyword.
## `proxyCountryCode` (type: `string`):
Two-letter country code for the residential proxy used to reach Reddit (e.g. 'us', 'gb', 'de'). Leave blank for any country.
## `tryCheapProxyFirst` (type: `boolean`):
Probe a cheap datacenter proxy before falling back to residential. ScrapingBee doesn't bill blocked requests, so this is free upside and cuts cost when it works — at the price of slightly higher latency on keywords where Reddit blocks the cheap proxy. Turn off for lowest latency.
## `debugLog` (type: `boolean`):
Log discovered subreddits, candidate counts, and LLM decisions for troubleshooting.
## Actor input object example
```json
{
"keywords": [
"best project management software"
],
"postsPerKeyword": 25,
"sort": "relevance",
"time": "all",
"proxyCountryCode": "us",
"tryCheapProxyFirst": true,
"debugLog": false
}
```
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"keywords": [
"best project management software",
"crm for small business"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("bustling_spotlight/reddit-search-scraper").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = { "keywords": [
"best project management software",
"crm for small business",
] }
# Run the Actor and wait for it to finish
run = client.actor("bustling_spotlight/reddit-search-scraper").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"keywords": [
"best project management software",
"crm for small business"
]
}' |
apify call bustling_spotlight/reddit-search-scraper --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=bustling_spotlight/reddit-search-scraper",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Reddit Search Scraper (Relevant Posts Only)",
"description": "Reddit scraper that returns only genuinely relevant posts, not viral noise. Enter any keyword and get back only for relevant results. Great for social listening, market research, and lead generation.",
"version": "0.0",
"x-build-id": "9mr3iXmoOcimodont"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/bustling_spotlight~reddit-search-scraper/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-bustling_spotlight-reddit-search-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/bustling_spotlight~reddit-search-scraper/runs": {
"post": {
"operationId": "runs-sync-bustling_spotlight-reddit-search-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/bustling_spotlight~reddit-search-scraper/run-sync": {
"post": {
"operationId": "run-sync-bustling_spotlight-reddit-search-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"keywords"
],
"properties": {
"keywords": {
"title": "Keywords / search queries",
"type": "array",
"description": "One or more search queries. For each keyword the actor returns only the genuinely relevant Reddit posts — not viral, off-topic noise.",
"items": {
"type": "string"
}
},
"postsPerKeyword": {
"title": "Relevant posts per keyword",
"minimum": 1,
"maximum": 100,
"type": "integer",
"description": "Maximum number of relevant posts to return per keyword. You are only charged for posts actually returned.",
"default": 25
},
"sort": {
"title": "Sort results by",
"enum": [
"relevance",
"hot",
"top",
"new",
"comments"
],
"type": "string",
"description": "How results are ranked. 'Relevance' is best for on-topic search; 'New' for fresh monitoring; 'Top'/'Hot' for popular discussion.",
"default": "relevance"
},
"time": {
"title": "Time range",
"enum": [
"hour",
"day",
"week",
"month",
"year",
"all"
],
"type": "string",
"description": "Time window applied when sorting by Relevance, Top, or Comments. Ignored for Hot and New.",
"default": "all"
},
"domainHint": {
"title": "Domain hint (optional, advanced)",
"type": "string",
"description": "Optional. If your keywords are ambiguous, name the domain in a few words (e.g. 'B2B software' or 'content creation, podcasting, video') so results reject same-word matches from unrelated fields. Leave blank to auto-detect from each keyword."
},
"proxyCountryCode": {
"title": "Proxy country code",
"type": "string",
"description": "Two-letter country code for the residential proxy used to reach Reddit (e.g. 'us', 'gb', 'de'). Leave blank for any country.",
"default": "us"
},
"tryCheapProxyFirst": {
"title": "Try cheap proxy first (lower cost)",
"type": "boolean",
"description": "Probe a cheap datacenter proxy before falling back to residential. ScrapingBee doesn't bill blocked requests, so this is free upside and cuts cost when it works — at the price of slightly higher latency on keywords where Reddit blocks the cheap proxy. Turn off for lowest latency.",
"default": true
},
"debugLog": {
"title": "Verbose debug logging",
"type": "boolean",
"description": "Log discovered subreddits, candidate counts, and LLM decisions for troubleshooting.",
"default": false
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```