reddit-posts-scraper-with-author-media-details

Reddit Posts Scraper — Author and Media Details

Reddit Posts Scraper with Author & Media Details turns a subreddit, keyword, or post URL into full post records — every metadata field a plain scraper drops (flair, NSFW/spoiler flags, domain, crosspost count), a real media classification block (video URLs, image previews, gallery detection), and optional per-author account stats — with nested comments included. It's built for content researchers who need the full media and metadata picture per post, not just title and score, social listening teams tracking author account age and karma alongside post content, and moderation or trust-and-safety teams that need NSFW/spoiler flags surfaced directly. No Reddit login is required. Every section below documents an input, an output field, or exactly how media classification and author enrichment work.

What is Reddit Posts Scraper with Author & Media Details?

This Actor scrapes Reddit via old.reddit.com HTML rather than Reddit's .json endpoints, since the module's own comment notes the .json path is 403-walled in-container across every proxy tier tested for this account family. It parses each post's full data-* attribute set — Reddit's own listing-page metadata — into structured rows, plus a media-type classification derived from URL and embedded-content analysis.

Key capabilities, read from the source:

• Full metadata Reddit's own listing page carries but a basic scraper drops. Every row includes authorFullname, subredditType, over18, spoiler, domain, outboundUrl, numCrossposts, linkFlairText, postUrl, and publishedAt (ISO) — all read directly from the post element's data-* attributes, not a separate fetch.
• Real media classification, not a guessed content type. _extract_media() determines mediaType (video/gallery/image/text/link) from actual signals — v.redd.it domain/URL matches, the data-is-video attribute, gallery URL patterns, and direct-image file extensions — with previewResolutions populated from real embedded image URLs, never a placeholder.
• Opt-in author account enrichment with an honest coverage marker. includeAuthorStats fetches each unique author's karma, comment karma, account age, and cake day — trying about.json first, falling back to the old.reddit profile sidebar HTML if that's blocked — and stamps authorSource ("about.json" or "sidebar") so you know which path supplied the data; verified/is_mod are only available via the JSON path and stay null when the sidebar fallback was used.
• Per-author caching within a run. enrich_author() caches results by lowercased username, so 50 posts by the same 10 authors cost 10 author lookups, not 50.
• upvote_ratio deliberately dropped, not faked. The module docstring notes this field is only available via the .json API this Actor doesn't use — rather than approximate it from old.reddit HTML (which doesn't expose it), the field is simply not emitted.
• Uncharged accounting rows for failed sources. A subreddit, keyword, or URL that returns zero posts (blocked or genuinely empty) produces one uncharged error row via a plain Actor.push_data() call with no charged_event_name, rather than either failing the run or silently returning nothing.

What data can I extract with Reddit Posts Scraper with Author & Media Details?

Every field below is read directly from _post_from_thing(), _extract_media(), and enrich_author() in src/main.py — the dataset's default view surfaces all 38 as table columns.

Core post fields

Field	Example Value	Notes
title / author / subreddit / score / num_comments	post basics
created_utc / publishedAt	epoch seconds / ISO string	Same timestamp, two formats
permalink / postUrl	relative path / full URL
body	self-post text	null for link posts
comments	nested array	Recursive replies, capped by maxComments

Metadata fields (dropped by a basic scraper)

Field	Example Value	Notes
authorFullname	t2_abc123	Reddit's internal author ID
subredditType	"public"
over18 / spoiler	false / false	NSFW and spoiler flags
domain / outboundUrl	"i.redd.it" / full URL	Where a link post points
numCrossposts	3
linkFlairText	"Discussion"

Media fields

Field	Example Value	Notes
mediaType	"video", "gallery", "image", "text", or "link"	Determined from real signals, not guessed
isVideo / hasGallery	true / false
videoUrl	v.redd.it URL	Only for genuine Reddit-hosted video
previewResolutions	array of image URLs	Real embedded image URLs

Author enrichment fields (opt-in via includeAuthorStats)

Field	Example Value	Notes
authorKarma / authorCommentKarma	45000 / 12000
authorAccountAgeDays / authorCakeDay	1825 / ISO date
authorVerified / authorIsMod	true / false	Only populated when authorSource is "about.json"
authorSource	"about.json" or "sidebar"	Discloses which fetch path supplied this author's data

Why not build this yourself?

Reddit's .json API endpoints — the obvious first choice for structured data — are blocked entirely for this account family in-container, across every proxy tier this Actor's own testing covered, which is why it fetches old.reddit.com HTML and parses the same data out of the page's data-* attributes instead. That HTML surface carries real, rich metadata (flair, NSFW flags, crosspost counts, video/gallery detection) that a scraper naively grabbing just title/score/permalink would miss entirely, and building reliable media-type classification from URL patterns and embedded-content markers — rather than just "has an image or not" — takes real source-level familiarity with how Reddit structures its post HTML across post types.

How to use data extracted from Reddit Posts Scraper with Author & Media Details?

Content and media research

Filter by mediaType to isolate video, gallery, or image posts specifically within a subreddit or keyword search — useful for studying what content format performs best in a community.

Social listening with author context

Turn on includeAuthorStats to see whether high-engagement posts in your topic tend to come from established, high-karma accounts or newer ones — useful signal for astroturfing or bot-activity research.

Moderation and trust-and-safety research

Use over18/spoiler flags directly rather than inferring them from title text, and cross-reference authorAccountAgeDays for accounts posting flagged content.

AI agents and content pipelines

Because mediaType is a real classification (not inferred downstream), an agent can route video, image, and text posts to different processing pipelines directly from the dataset.

🔼 Input sample

Parameter	Required	Type	Description	Example Value
startUrls	Yes	array	Subreddit name/URL, or free-text search term — one per line, mixed freely.	["news", "r/technology", "artificial intelligence"]
sortOrder	No	string enum	hot, new, top, rising. Default "top".	"top"
timeFilter	No	string enum	hour/day/week/month/year/all — applies to Top/Rising only. Default "week".	"week"
maxPosts	No	integer (1–1000)	Posts to collect per source. Default 10.	50
maxComments	No	integer (0–1000)	Top-level comments (with nested replies) per post. 0 skips comments. Default 5.	10
includeAuthorStats	No	boolean	Fetch author account stats. One extra request per unique author, cached. Default false.	true
proxyConfiguration	No	object	Optional; escalates none → datacenter → residential automatically on a block.	{"useApifyProxy": false}

{
  "startUrls": ["news", "artificial intelligence"],
  "sortOrder": "top",
  "timeFilter": "week",
  "maxPosts": 50,
  "maxComments": 10,
  "includeAuthorStats": true
}

Common pitfall: upvote_ratio is not in this Actor's output at all — it's exclusive to Reddit's .json API, which this Actor doesn't use because that path is blocked in-container for this account family. Don't expect it to appear even with all other fields enabled.

🔽 Output sample

Output is one JSON row per post (plus one uncharged accounting row for a source that returned nothing), pushed to the run's default dataset and charged as one row_result event per successful post.

{
  "post_id": "1abcde",
  "title": "New study on renewable energy adoption rates",
  "author": "example_user",
  "subreddit": "news",
  "score": 4521,
  "num_comments": 312,
  "created_utc": 1721984400,
  "publishedAt": "2026-07-26T09:00:00Z",
  "permalink": "/r/news/comments/1abcde/new_study_on_renewable_energy/",
  "postUrl": "https://www.reddit.com/r/news/comments/1abcde/new_study_on_renewable_energy/",
  "over18": false,
  "spoiler": false,
  "linkFlairText": "Environment",
  "domain": "i.redd.it",
  "mediaType": "image",
  "isVideo": false,
  "previewResolutions": ["https://i.redd.it/abc123.jpg"],
  "authorKarma": 45000,
  "authorSource": "about.json",
  "success": true,
  "error_message": null
}

How do you filter and target specific posts and media?

Media type as a targeting lens. Since mediaType reflects real content classification, you can filter your exported dataset to "video" for short-form video research or "text" for discussion/self-post analysis, without needing to re-derive content type from raw URLs.

Author stats are opt-in for a reason. includeAuthorStats adds one request per unique author — for a run scoped to a small number of active authors, this is cheap; for a wide multi-subreddit sweep with hundreds of distinct authors, weigh the added run time against whether you actually need account-age/karma context.

Comment depth vs. run speed. maxComments: 0 skips the per-post page fetch entirely for subreddit-sourced listings (comments require a second fetch), making a metadata-and-media-only run substantially faster than one that also pulls comment threads.

Three real examples:

{ "startUrls": ["r/videos"], "sortOrder": "top", "timeFilter": "week", "maxComments": 0 }

Fast video-subreddit media sweep, no comment fetching.

{ "startUrls": ["artificial intelligence"], "maxPosts": 100, "includeAuthorStats": true }

Keyword search with full author-account context for astroturfing research.

{ "startUrls": ["news", "worldnews"], "maxComments": 20, "sortOrder": "hot" }

Deep comment-thread pull across two trending-news subreddits.

▶️ Want to try other scrapers?

Scraper	What it extracts
Reddit Trends Scraper with Author Contact Leads	Trending posts with contact-lead signal
Pinterest Comment Scraper (Flattened Bulk Export)	Bulk comment export across many pins
TikTok Comments Scraper — Full Reply Threads	Nested TikTok comment threads
Instagram Comment Engagement Scraper	Per-post comment threads and commenter identity

How to extract Reddit post data programmatically

This Actor runs as a standard Apify Actor call — one API call in, structured JSON dataset out, using your Apify API token.

Python example

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("<YOUR_USERNAME>/reddit-posts-scraper-with-author-media-details").call(run_input={
    "startUrls": ["news"],
    "sortOrder": "top",
    "timeFilter": "week",
    "maxPosts": 50,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("success"):
        print(item["title"], item["mediaType"], item["score"])

Export to spreadsheets or CRM

Every dataset supports one-click CSV/Excel export from the Apify Console. Filter on success to exclude accounting rows before exporting post data for analysis.

Is it legal to scrape Reddit?

Scraping publicly visible Reddit posts and comments is generally lawful, since this data is published for anyone to view without logging in — the underlying legal question was tested directly in hiQ Labs v. LinkedIn (9th Cir.), which held that scraping public, non-password-protected data does not violate the U.S. Computer Fraud and Abuse Act. Reddit usernames are pseudonymous but can still constitute personal data under GDPR/CCPA if tied to an identifiable individual, so treat authorKarma/account-age data accordingly if you store or reuse it at scale, and consult legal counsel for commercial applications.

❓ FAQ

Why does this scraper use old.reddit.com instead of the .json API?

This Actor's own testing found Reddit's .json endpoints 403-walled in-container across every proxy tier for this account family — old.reddit.com HTML remains reachable and carries the same underlying data-* metadata, so the Actor parses that instead.

Why is upvote_ratio missing from the output?

That field is exclusive to Reddit's .json API, which this Actor doesn't use. Rather than approximate or fabricate it from HTML (which doesn't expose it), it's simply omitted.

What's the difference between authorSource "about.json" and "sidebar"?

"about.json" means the richer JSON profile endpoint was reachable, giving you authorVerified/authorIsMod in addition to karma and account age. "sidebar" means that endpoint was blocked and the Actor fell back to parsing the old.reddit profile page's sidebar HTML instead — karma and account age are still real, but authorVerified/authorIsMod stay null since the sidebar doesn't expose them.

Does author enrichment slow down a run a lot?

Results are cached per unique author within a run, so it scales with distinct authors, not total posts — a run with many posts from few repeat authors costs relatively little extra time; a run spanning many distinct authors costs one extra request per author.

What happens if a subreddit or keyword returns nothing?

A single uncharged accounting row is emitted with success: false and an error_message explaining why (empty, blocked, or unavailable) — you're not charged for it, and the run continues to the next source rather than failing entirely.

How does this compare to other Reddit scrapers?

As observed on the Apify Store on 2026-07-26, harshmaur/reddit-scraper covers broad Reddit scraping but its documentation doesn't detail media-type classification logic or an author-enrichment coverage marker like authorSource — that transparency around data provenance is specific to this Actor.

Does this work with AI agent frameworks?

Yes — call it as a standard HTTP endpoint via the Apify API from any agent framework capable of making an API call; there's no MCP-specific integration for this Actor.

Conclusion

Reddit Posts Scraper with Author & Media Details turns a subreddit, keyword, or URL into full post records — real media classification, the metadata a basic scraper drops, and optional author-account context with an honest coverage marker. It fits content research, social listening, and moderation workflows that need more than title and score. Start a run from the Apify Console or the Apify API with your target sources to get your first enriched post export.