Reddit Url Scraper

Scrape structured data from Reddit post and comment URLs — at small scale or millions of links.

Give the Actor a list of Reddit permalinks (or a Dataset / Excel / DataFrame of URLs).
Get back clean JSON: full text, author, score, media, timestamps, and more.

Built for:

• Bulk URL enrichment (you already have the links)
• Notebook / API pipelines
• Large jobs with auto parallel workers
• Reliable scraping with residential proxies, retries, and adaptive throttling

---

What this Actor does

You provide	Actor returns
Post URL	Post fields + media
Comment URL	That specific comment
Invalid / blocked URL	An error item (run continues)

Supported URL formats

Post

https://www.reddit.com/r/SUBREDDIT/comments/POST_ID/slug/
https://old.reddit.com/r/SUBREDDIT/comments/POST_ID/slug/

Comment

https://www.reddit.com/r/SUBREDDIT/comments/POST_ID/slug/COMMENT_ID/

What it does NOT do

• Does not crawl a whole subreddit from a listing page
• Does not expand the full comment tree under a post (pass comment permalinks if you need comments)
• Does not scrape user profiles or search results

This Actor is a URL enricher: perfect when you already have Reddit links.

---

Quick start (Console)

Small list (paste URLs)

1. Open the Actor → Input
2. Paste URLs into Reddit URLs (small lists) / startUrls
3. Keep Proxy on Apify Proxy → RESIDENTIAL
4. Click Start
5. Open Dataset to download results (JSON / CSV / Excel)

Example input:

{
  "startUrls": [
    { "url": "https://www.reddit.com/r/whatisit/comments/1viuwa5/whats_the_goal_here_range_extender_is_this/" }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  },
  "maxConcurrency": 20
}

Large list (thousands → millions)

1. Put your URLs into an Apify Dataset — each item must look like:

   { "url": "https://www.reddit.com/r/.../comments/..." }
   

2. Copy the Dataset ID
3. Set Actor input:
   • urlsDatasetId = that ID
   • Proxy = RESIDENTIAL
4. Start the Actor

The Actor streams URLs from the dataset through a request queue and scrapes them at high concurrency in a single run. For very large jobs, raise maxConcurrency and give the run more memory in the Actor's resource settings.

---

Input reference (all fields explained)

Required (one of these)

Field	When to use	Details
startUrls	Small / medium jobs (roughly up to ~5–10k URLs)	Array of { "url": "..." }. Easy in Console.
urlsDatasetId	Large jobs (10k → millions)	Apify Dataset ID streamed through a request queue.

You must provide at least one of the two.

URL source options

Field	Default	Explanation
urlField	url	Field name inside each dataset item that holds the Reddit link. Change if your items use e.g. link or permalink.

Speed & scaling (important)

Speed scales with maxConcurrency — how many URLs are fetched at the same time.

Field	Default	Explanation
maxConcurrency	100	Parallel fetches. The Actor runs at this full concurrency from the start (no ramp-up), even for small lists. Higher can trigger more Reddit 403/429 blocks.
minConcurrency	100	Ignored — the Actor always runs at full maxConcurrency. Kept for backward compatibility.
outputDatasetName	—	Optional named dataset to ALSO write results to (alongside the run's default dataset). Set a fixed name so you always know where to read.

Reliability controls (keep these ON for production)

Field	Default	Explanation
proxyConfiguration	Apify Proxy RESIDENTIAL	Critical. Without residential proxies, Reddit often returns 403 and runs fail.
preflightCheck	true	Tests Reddit + proxy once before the crawl.
shuffleUrls	true	Shuffles batches so the same subreddit isn’t hit in a tight loop.
adaptiveThrottle	true	Slows down when block/fail rate spikes; speeds up when healthy.
retryFailedUrls	true	After the main pass, retries failures once at lower concurrency.
maxRequestRetries	8	Retries per URL (with fresh proxy session) before marking error.

Advanced / optional

Field	Default	Explanation
enqueueBatchSize	1000	How many URLs are added to the queue per batch.
maxRequestsPerCrawl	empty	Cap how many URLs to scrape this run (great for cost tests).
requestQueueName	empty	Named queue for resume after crash (single-run mode) or batch id prefix (parallel).
purgeRequestQueue	false	Clear named queue before starting (fresh run, not resume).
progressLogEvery	1000	Log progress every N successes.
datasetOffset / datasetLimit	0 / all	Read only a window of the URLs dataset. Leave default to read the whole dataset.
rawEngine	true	Fast mode for startUrls. Turn OFF for the standard engine (persistent queue + dataset streaming). Standard mode is selected automatically when urlsDatasetId is set.

---

Recommended presets

1) Safe / reliable (default-ish)

{
  "urlsDatasetId": "YOUR_DATASET_ID",
  "maxConcurrency": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  },
  "preflightCheck": true,
  "shuffleUrls": true,
  "adaptiveThrottle": true,
  "retryFailedUrls": true
}

2) Fast large job

{
  "urlsDatasetId": "YOUR_DATASET_ID",
  "maxConcurrency": 150,
  "outputDatasetName": "reddit-scrape-results",
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

3) Cost / quality test (first 1,000 URLs only)

{
  "urlsDatasetId": "YOUR_DATASET_ID",
  "maxRequestsPerCrawl": 1000,
  "maxConcurrency": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

---

Output format

The Actor defines an output schema and dataset schema so the Console Output tab shows:

• A single Results view with all post and comment fields
• Field descriptions for every result property

Every input URL produces one dataset item.

Post item (type: "post")

Field	Meaning
type	"post"
url	Original input URL
subreddit	Subreddit name
author	Username
score	Upvotes / score
full_text	Post title
text	Self-post body (if any)
numComments	Comment count
createdUtc	ISO timestamp
permalink	Reddit path
media	Array of { "type", "url" } (image / video / gallery / thumbnail)

Comment item (type: "comment")

Field	Meaning
type	"comment"
url	Original input URL
subreddit	Subreddit
author	Username
score	Score
full_text	Comment text (main content)
createdUtc	ISO timestamp
permalink	Reddit path

Error item (type: "error")

{
  "type": "error",
  "url": "https://...",
  "error": "Blocked or unavailable"
}

Filter successes in your pipeline with: type == "post" OR type == "comment".

---

Using from a Python notebook / API

Small DataFrame → startUrls

import pandas as pd
from apify_client import ApifyClient

df = pd.read_excel("urls.xlsx")  # column with links
urls = df["url"].dropna().astype(str).str.strip().unique().tolist()

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("cQHw4O99vrVaWVdJe").call(run_input={
    "startUrls": [{"url": u} for u in urls],
    "maxConcurrency": 20,
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
})

items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
results = pd.DataFrame(items)

Large DataFrame → push dataset + start (all in notebook)

You cannot put millions of URLs in one API JSON body. From the notebook, push rows then start:

import pandas as pd
from apify_client import ApifyClient

df = pd.read_excel("urls.xlsx")
urls = df["url"].dropna().astype(str).str.strip().unique().tolist()

client = ApifyClient("YOUR_APIFY_TOKEN")
dataset = client.datasets().get_or_create()
dataset_id = dataset["id"]

batch = []
for u in urls:
    batch.append({"url": u})
    if len(batch) >= 1000:
        client.dataset(dataset_id).push_items(batch)
        batch.clear()
if batch:
    client.dataset(dataset_id).push_items(batch)

run = client.actor("cQHw4O99vrVaWVdJe").start(run_input={
    "urlsDatasetId": dataset_id,
    "maxConcurrency": 100,
    "outputDatasetName": "reddit-scrape-results",
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
})
print("Started:", run["id"])

More API examples: see API_USAGE.md.

---

How to make it reliable

1. Always use RESIDENTIAL proxies in production
2. Start with maxConcurrency: 100; lower it if the success rate drops
3. Keep preflightCheck, shuffleUrls, adaptiveThrottle, retryFailedUrls enabled
4. Watch logs for success rate — if below ~80%, lower concurrency
5. For huge jobs (5M+), prefer batching into a few runs instead of one mega-run

How to make it fast

1. Use urlsDatasetId for very large lists (streamed through the request queue)
2. Raise maxConcurrency (e.g. 150) and give the run more memory in resource settings
3. Remember: higher concurrency costs more proxy + compute and can raise the block rate

---

Pricing / cost tips

• You are charged by Apify usage (compute + residential proxy traffic) and, if published on Store, by the Actor’s pricing model.
• Test with maxRequestsPerCrawl: 1000 first to estimate cost per URL.
• Failed URLs still consume some proxy/compute because of retries.

---

FAQ

Q: Can I paste an Excel file in the Console?
A: Not directly. Convert to a list (startUrls) for small files, or upload URLs into a Dataset / push from a notebook for large files.

Q: Why do I get many 403 errors?
A: Almost always missing RESIDENTIAL proxy, or concurrency too high. Enable residential and lower maxConcurrency.

Q: Will one bad URL stop the run?
A: No. Bad URLs become type: "error" items; the rest continue.

Q: Can it do 1M / 5M / 10M URLs?
A: Yes architecturally (dataset streaming + auto shards). Expect higher cost and longer runtime; for 5M–10M prefer chunked batches.

Q: Where are parallel results stored?
A: In outputDatasetName (recommended). Set a fixed name so every worker writes to the same place.

Q: How do I resume after a crash?
A: For a single run, reuse the same requestQueueName and do not set purgeRequestQueue: true.

---

Limitations

• Reddit may rate-limit or block aggressive traffic even with residential proxies
• Deleted/removed content is returned as Reddit shows it ([deleted], [removed])
• Media extraction is best-effort for images/video/galleries
• Not a replacement for Reddit’s official API for authenticated account actions

---

Support

If a run fails unexpectedly, share:

1. Run ID / Console link
2. Input JSON (hide token)
3. Whether RESIDENTIAL proxy was enabled
4. Approx URL count and maxConcurrency