# Reddit Url Scraper (`krillin/reddit-url-scraper`) Actor

Scrape Reddit posts and comments from a list of URLs. Returns title, author, score, text, media, and more. Built for large batches with parallel runs and residential proxy support.

- **URL**: https://apify.com/krillin/reddit-url-scraper.md
- **Developed by:** [Krillin Kills](https://apify.com/krillin) (community)
- **Categories:** Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Reddit URL Scraper

Scrape **structured data from Reddit post and comment URLs** — at small scale or millions of links.

Give the Actor a list of Reddit permalinks (or a Dataset / Excel / DataFrame of URLs).\
Get back clean JSON: full text, author, score, media, timestamps, and more.

Built for:

- Bulk URL enrichment (you already have the links)
- Notebook / API pipelines
- Large jobs with **auto parallel workers**
- Reliable scraping with residential proxies, retries, and adaptive throttling

***

### What this Actor does

| You provide | Actor returns |
|---|---|
| Post URL | Post fields + media |
| Comment URL | That specific comment |
| Invalid / blocked URL | An `error` item (run continues) |

#### Supported URL formats

**Post**

```text
https://www.reddit.com/r/SUBREDDIT/comments/POST_ID/slug/
https://old.reddit.com/r/SUBREDDIT/comments/POST_ID/slug/
```

**Comment**

```text
https://www.reddit.com/r/SUBREDDIT/comments/POST_ID/slug/COMMENT_ID/
```

#### What it does NOT do

- Does **not** crawl a whole subreddit from a listing page
- Does **not** expand the full comment tree under a post (pass comment permalinks if you need comments)
- Does **not** scrape user profiles or search results

This Actor is a **URL enricher**: perfect when you already have Reddit links.

***

### Quick start (Console)

#### Small list (paste URLs)

1. Open the Actor → **Input**
2. Paste URLs into **Reddit URLs (small lists)** / `startUrls`
3. Keep **Proxy** on **Apify Proxy → RESIDENTIAL**
4. Click **Start**
5. Open **Dataset** to download results (JSON / CSV / Excel)

Example input:

```json
{
  "startUrls": [
    { "url": "https://www.reddit.com/r/whatisit/comments/1viuwa5/whats_the_goal_here_range_extender_is_this/" }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  },
  "maxConcurrency": 20
}
```

#### Large list (thousands → millions)

1. Put your URLs into an **Apify Dataset** — each item must look like:
   ```json
   { "url": "https://www.reddit.com/r/.../comments/..." }
   ```
2. Copy the Dataset ID
3. Set Actor input:
   - `urlsDatasetId` = that ID
   - Proxy = **RESIDENTIAL**
4. Start the Actor

The Actor streams URLs from the dataset through a request queue and scrapes them at
high concurrency in a single run. For very large jobs, raise `maxConcurrency` and give
the run more memory in the Actor's resource settings.

***

### Input reference (all fields explained)

#### Required (one of these)

| Field | When to use | Details |
|---|---|---|
| `startUrls` | Small / medium jobs (roughly up to ~5–10k URLs) | Array of `{ "url": "..." }`. Easy in Console. |
| `urlsDatasetId` | Large jobs (10k → millions) | Apify Dataset ID streamed through a request queue. |

You must provide **at least one** of the two.

#### URL source options

| Field | Default | Explanation |
|---|---|---|
| `urlField` | `url` | Field name inside each dataset item that holds the Reddit link. Change if your items use e.g. `link` or `permalink`. |

#### Speed & scaling (important)

Speed scales with **`maxConcurrency`** — how many URLs are fetched at the same time.

| Field | Default | Explanation |
|---|---|---|
| `maxConcurrency` | `100` | Parallel fetches. The Actor runs at this full concurrency from the start (no ramp-up), even for small lists. Higher can trigger more Reddit 403/429 blocks. |
| `minConcurrency` | `100` | Ignored — the Actor always runs at full `maxConcurrency`. Kept for backward compatibility. |
| `outputDatasetName` | — | Optional named dataset to ALSO write results to (alongside the run's default dataset). Set a fixed name so you always know where to read. |

#### Reliability controls (keep these ON for production)

| Field | Default | Explanation |
|---|---|---|
| `proxyConfiguration` | Apify Proxy RESIDENTIAL | **Critical.** Without residential proxies, Reddit often returns 403 and runs fail. |
| `preflightCheck` | `true` | Tests Reddit + proxy once before the crawl. |
| `shuffleUrls` | `true` | Shuffles batches so the same subreddit isn’t hit in a tight loop. |
| `adaptiveThrottle` | `true` | Slows down when block/fail rate spikes; speeds up when healthy. |
| `retryFailedUrls` | `true` | After the main pass, retries failures once at lower concurrency. |
| `maxRequestRetries` | `8` | Retries per URL (with fresh proxy session) before marking error. |

#### Advanced / optional

| Field | Default | Explanation |
|---|---|---|
| `enqueueBatchSize` | `1000` | How many URLs are added to the queue per batch. |
| `maxRequestsPerCrawl` | empty | Cap how many URLs to scrape this run (great for cost tests). |
| `requestQueueName` | empty | Named queue for **resume** after crash (single-run mode) or batch id prefix (parallel). |
| `purgeRequestQueue` | `false` | Clear named queue before starting (fresh run, not resume). |
| `progressLogEvery` | `1000` | Log progress every N successes. |
| `datasetOffset` / `datasetLimit` | `0` / all | Read only a window of the URLs dataset. Leave default to read the whole dataset. |
| `rawEngine` | `true` | Fast mode for `startUrls`. Turn OFF for the standard engine (persistent queue + dataset streaming). Standard mode is selected automatically when `urlsDatasetId` is set. |

***

### Recommended presets

#### 1) Safe / reliable (default-ish)

```json
{
  "urlsDatasetId": "YOUR_DATASET_ID",
  "maxConcurrency": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  },
  "preflightCheck": true,
  "shuffleUrls": true,
  "adaptiveThrottle": true,
  "retryFailedUrls": true
}
```

#### 2) Fast large job

```json
{
  "urlsDatasetId": "YOUR_DATASET_ID",
  "maxConcurrency": 150,
  "outputDatasetName": "reddit-scrape-results",
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

#### 3) Cost / quality test (first 1,000 URLs only)

```json
{
  "urlsDatasetId": "YOUR_DATASET_ID",
  "maxRequestsPerCrawl": 1000,
  "maxConcurrency": 100,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}
```

***

### Output format

The Actor defines an **output schema** and **dataset schema** so the Console Output tab shows:

- A single **Results** view with all post and comment fields
- Field descriptions for every result property

Every input URL produces **one dataset item**.

#### Post item (`type: "post"`)

| Field | Meaning |
|---|---|
| `type` | `"post"` |
| `url` | Original input URL |
| `subreddit` | Subreddit name |
| `author` | Username |
| `score` | Upvotes / score |
| `full_text` | Post title |
| `text` | Self-post body (if any) |
| `numComments` | Comment count |
| `createdUtc` | ISO timestamp |
| `permalink` | Reddit path |
| `media` | Array of `{ "type", "url" }` (`image` / `video` / `gallery` / `thumbnail`) |

#### Comment item (`type: "comment"`)

| Field | Meaning |
|---|---|
| `type` | `"comment"` |
| `url` | Original input URL |
| `subreddit` | Subreddit |
| `author` | Username |
| `score` | Score |
| `full_text` | **Comment text** (main content) |
| `createdUtc` | ISO timestamp |
| `permalink` | Reddit path |

#### Error item (`type: "error"`)

```json
{
  "type": "error",
  "url": "https://...",
  "error": "Blocked or unavailable"
}
```

Filter successes in your pipeline with: `type == "post" OR type == "comment"`.

***

### Using from a Python notebook / API

#### Small DataFrame → `startUrls`

```python
import pandas as pd
from apify_client import ApifyClient

df = pd.read_excel("urls.xlsx")  # column with links
urls = df["url"].dropna().astype(str).str.strip().unique().tolist()

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("cQHw4O99vrVaWVdJe").call(run_input={
    "startUrls": [{"url": u} for u in urls],
    "maxConcurrency": 20,
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
})

items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
results = pd.DataFrame(items)
```

#### Large DataFrame → push dataset + start (all in notebook)

You cannot put millions of URLs in one API JSON body. From the notebook, push rows then start:

```python
import pandas as pd
from apify_client import ApifyClient

df = pd.read_excel("urls.xlsx")
urls = df["url"].dropna().astype(str).str.strip().unique().tolist()

client = ApifyClient("YOUR_APIFY_TOKEN")
dataset = client.datasets().get_or_create()
dataset_id = dataset["id"]

batch = []
for u in urls:
    batch.append({"url": u})
    if len(batch) >= 1000:
        client.dataset(dataset_id).push_items(batch)
        batch.clear()
if batch:
    client.dataset(dataset_id).push_items(batch)

run = client.actor("cQHw4O99vrVaWVdJe").start(run_input={
    "urlsDatasetId": dataset_id,
    "maxConcurrency": 100,
    "outputDatasetName": "reddit-scrape-results",
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
})
print("Started:", run["id"])
```

More API examples: see [`API_USAGE.md`](API_USAGE.md).

***

### How to make it reliable

1. **Always use RESIDENTIAL proxies** in production
2. Start with `maxConcurrency: 100`; lower it if the success rate drops
3. Keep `preflightCheck`, `shuffleUrls`, `adaptiveThrottle`, `retryFailedUrls` enabled
4. Watch logs for success rate — if below ~80%, lower concurrency
5. For huge jobs (5M+), prefer **batching** into a few runs instead of one mega-run

### How to make it fast

1. Use `urlsDatasetId` for very large lists (streamed through the request queue)
2. Raise `maxConcurrency` (e.g. 150) and give the run more memory in resource settings
3. Remember: higher concurrency costs more proxy + compute and can raise the block rate

***

### Pricing / cost tips

- You are charged by Apify usage (compute + residential proxy traffic) and, if published on Store, by the Actor’s pricing model.
- Test with `maxRequestsPerCrawl: 1000` first to estimate cost per URL.
- Failed URLs still consume some proxy/compute because of retries.

***

### FAQ

**Q: Can I paste an Excel file in the Console?**\
A: Not directly. Convert to a list (`startUrls`) for small files, or upload URLs into a Dataset / push from a notebook for large files.

**Q: Why do I get many 403 errors?**\
A: Almost always missing **RESIDENTIAL** proxy, or concurrency too high. Enable residential and lower `maxConcurrency`.

**Q: Will one bad URL stop the run?**\
A: No. Bad URLs become `type: "error"` items; the rest continue.

**Q: Can it do 1M / 5M / 10M URLs?**\
A: Yes architecturally (dataset streaming + auto shards). Expect higher cost and longer runtime; for 5M–10M prefer chunked batches.

**Q: Where are parallel results stored?**\
A: In `outputDatasetName` (recommended). Set a fixed name so every worker writes to the same place.

**Q: How do I resume after a crash?**\
A: For a single run, reuse the same `requestQueueName` and do **not** set `purgeRequestQueue: true`.

***

### Limitations

- Reddit may rate-limit or block aggressive traffic even with residential proxies
- Deleted/removed content is returned as Reddit shows it (`[deleted]`, `[removed]`)
- Media extraction is best-effort for images/video/galleries
- Not a replacement for Reddit’s official API for authenticated account actions

***

### Support

If a run fails unexpectedly, share:

1. Run ID / Console link
2. Input JSON (hide token)
3. Whether RESIDENTIAL proxy was enabled
4. Approx URL count and `maxConcurrency`

# Actor input Schema

## `startUrls` (type: `array`):

Paste Reddit post or comment permalinks here for small/medium jobs (roughly up to a few thousand URLs). Each item needs a "url" field. For tens of thousands or millions of URLs, use urlsDatasetId instead — large lists do not fit well in this field.

## `urlsDatasetId` (type: `string`):

Apify Dataset ID where each item contains a Reddit URL (default field name: "url"). Use this for very large lists that do not fit in startUrls. Create a dataset in Console or push items from your notebook/API, then paste the dataset ID here.

## `urlField` (type: `string`):

Which field on each dataset item holds the Reddit link. Default is "url". Change this if your items use another key such as "link" or "permalink".

## `outputDatasetName` (type: `string`):

Optional named dataset to ALSO write results to (in addition to the run's default dataset). Set a stable name (e.g. reddit-scrape-results) so you always know where to download. Leave empty to use only the run's default dataset.

## `proxyConfiguration` (type: `object`):

CRITICAL for success. Reddit blocks datacenter IPs. Enable Apify Proxy and select the RESIDENTIAL group for production runs. Without residential proxies you will often see HTTP 403 failures.

## `maxConcurrency` (type: `integer`):

How many URLs are fetched at the same time. The Actor always runs at this full concurrency from the start (no ramp-up), even for small URL lists. Default 100. Too high (e.g. 200) can cause more Reddit 403/429 blocks.

## `maxRequestsPerCrawl` (type: `integer`):

Optional safety cap. Example: set 1000 to estimate cost/success rate before a full million-URL run. Leave empty for no limit.

## `datasetOffset` (type: `integer`):

Start index when reading the URLs dataset. Leave at 0 unless you want to scrape only a window of the dataset.

## `datasetLimit` (type: `integer`):

Max items to read from the URLs dataset for this run. Leave empty to read the whole dataset.

## `progressLogEvery` (type: `integer`):

How often to print progress (success count, fail rate). Use 1000–5000 for large runs so logs stay readable.

## `preflightCheck` (type: `boolean`):

Before scraping, run one quick, time-boxed request to verify proxy/connectivity. Keep ON for production so you fail fast if proxies are misconfigured. Turn OFF for the absolute fastest start (saves ~1 request of startup latency).

## `shuffleUrls` (type: `boolean`):

Randomizes the order so the same subreddit is not hit over and over in a tight loop. Reduces blocks. Keep ON.

## `retryFailedUrls` (type: `boolean`):

After the main pass, retry recoverable failed URLs once at lower concurrency with fresh IPs to recover temporary Reddit blocks. Keep ON unless you want maximum speed with no recovery pass.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.reddit.com/r/python/comments/1b1b1b1/example_post/"
    }
  ],
  "urlField": "url",
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  },
  "maxConcurrency": 100,
  "datasetOffset": 0,
  "progressLogEvery": 1000,
  "preflightCheck": true,
  "shuffleUrls": true,
  "retryFailedUrls": true
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset of post and comment items (one row per input URL).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.reddit.com/r/python/comments/1b1b1b1/example_post/"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("krillin/reddit-url-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.reddit.com/r/python/comments/1b1b1b1/example_post/" }],
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("krillin/reddit-url-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.reddit.com/r/python/comments/1b1b1b1/example_post/"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call krillin/reddit-url-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,krillin/reddit-url-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cQHw4O99vrVaWVdJe/builds/nTRQ1QcqaKvSg0z3i/openapi.json