Substack Scraper — Posts, Comments, Publications & Authors

Scrape any Substack newsletter without logging in. Point the Actor at publications and get every post as a clean row — title, authors, date, likes, comments, restacks, tags, paywall flag and the full post as HTML and plain text. Add the comment threads, discover publications through Substack's category leaderboards, or look up author profiles.

What does Substack Scraper do?

It reads Substack's public JSON endpoints (the same ones the website uses), so it works for publications on name.substack.com and on custom domains alike. Four modes:

Mode	Input	One row per
posts	publication URLs or subdomains	post (plus comments when enabled)
postUrls	specific post links	post (plus comments when enabled)
publications	category slugs (technology, culture, …)	publication on the leaderboard
authors	Substack handles	author profile

Why use this Actor?

• Full content, not just metadata. bodyHtml and bodyText (subscribe widgets removed) for every free post; paid posts come back as the public preview with isPaywalled: true.
• Comment threads, flattened. Every comment is a row with parentCommentId and depth, so threads survive CSV export.
• Scheduling-friendly. postedAfter stops pagination at the date you give, so a daily run over 50 newsletters costs cents.
• Discovery built in. Category leaderboards give you ranked publications with author, tagline, tier and subscriber signals — the input for your next posts run.
• One constant row shape per record type. Absent values are null, never missing. Managed proxies, honest per-publication shortfall reporting, no $0.09 start fee.

How much does it cost?

$2.50 per 1,000 rows (posts, comments, publications or authors). No per-run fee; proxies and compute included.

Run	Rows	Cost
100 posts with full content, no comments	100	$0.25
100 posts with content and ~10 comments each	~1,100	~$2.75
Top 50 technology publications	50	$0.13
Daily run, 20 newsletters, postedAfter yesterday (~15 new posts)	15	$0.04

Keep maxCommentsPerPost low (default 25) — every comment is a billed row. Paid (subscriber-only) posts return no comment rows: Substack serves their comment threads only to subscribers, so includeComments adds rows for free posts only.

Input

Field	Type	Description
mode	string	posts (default), postUrls, publications or authors.
publicationUrls	array	Publications for posts mode: https://lenny.substack.com, a custom domain, a bare subdomain (lenny) or any post URL. Up to 200.
postUrls	array	Post links for postUrls mode: https://<publication>/p/<slug> or /i/<id>. Up to 1,000.
categories	array	Category slugs or names for publications mode (e.g. technology, business, finance).
authorHandles	array	Handles for authors mode: lenny, @lenny or https://substack.com/@lenny.
maxPostsPerPublication	integer	Posts per publication. Default 50, max 5,000.
sort	string	new (default) or top (most liked first).
searchTerm	string	Substack's own archive search (title and body).
postedAfter / postedBefore	string	YYYY-MM-DD window (UTC). With sort: new, postedAfter stops pagination early.
audienceFilter	string	all (default), free or paid.
includeContent	boolean	Fetch each post for bodyHtml/bodyText (one extra request per post). Default true.
includeComments	boolean	Fetch comment threads (one extra request and up to maxCommentsPerPost billed rows per post). Default false.
maxCommentsPerPost	integer	Default 25, max 2,000.
maxPublicationsPerCategory	integer	Default 50, max 1,000.

Example input:

{
    "publicationUrls": ["https://www.astralcodexten.com", "lenny"],
    "maxPostsPerPublication": 20,
    "postedAfter": "2026-09-01",
    "includeContent": true
}

Output

A post row (recordType: "post"):

{
    "recordType": "post",
    "postId": 219204484,
    "url": "https://www.astralcodexten.com/p/top-forecasters-on-the-plague",
    "slug": "top-forecasters-on-the-plague",
    "publicationName": "Astral Codex Ten",
    "publicationHost": "www.astralcodexten.com",
    "publicationSubdomain": "astralcodexten",
    "publicationUrl": "https://www.astralcodexten.com",
    "publicationTagline": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
    "publicationFreeSubscriberCount": null,
    "title": "Top Forecasters On The Plague",
    "subtitle": "...",
    "description": "...",
    "authorNames": ["Scott Alexander"],
    "authorHandles": ["astralcodexten"],
    "postDate": "2026-10-07T21:13:16.863Z",
    "type": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "wordCount": 880,
    "likes": 243,
    "commentCount": 85,
    "restacks": 10,
    "coverImageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/26f61ee3_1200x655.jpeg",
    "tags": [],
    "sectionName": null,
    "podcastUrl": null,
    "podcastDurationSeconds": null,
    "bodyHtml": "<p>A technician at a Siberian lab has <a href=\"https://apnews.com/...\">died of yersinia infection aka plague</a>. ...</p>",
    "bodyText": "A technician at a Siberian lab has died of yersinia infection aka plague. ...",
    "contentStatus": "full",
    "source": "https://www.astralcodexten.com",
    "scrapedAt": "2026-10-08T14:30:00.000Z"
}

A comment row (recordType: "comment"):

{
    "recordType": "comment",
    "commentId": 355440418,
    "postId": 219204484,
    "postUrl": "https://www.astralcodexten.com/p/top-forecasters-on-the-plague",
    "publicationHost": "www.astralcodexten.com",
    "parentCommentId": null,
    "depth": 0,
    "authorName": "Sasha Putilin",
    "authorHandle": "psychotechnology",
    "authorUserId": 74588887,
    "authorPhotoUrl": "https://bucketeer.s3.amazonaws.com/public/images/27b067e3.png",
    "authorBestsellerTier": null,
    "body": "Russian here! If there are secondary cases, ...",
    "date": "2026-10-07T21:22:05.209Z",
    "editedAt": null,
    "likes": 15,
    "childrenCount": 2,
    "scrapedAt": "2026-10-08T14:30:00.000Z"
}

Publication rows carry name, host, url, authorName, tagline, logoUrl, rankingDetail (e.g. "Tens of thousands of paid subscribers"), tier, categorySlug, categoryRank, language, createdAt; freeSubscriberCount is usually null because the leaderboard only publishes the ranking text. Author rows carry name, handle, profileUrl, bio, subscriberCount, followerCount, bestsellerTier, primaryPublicationHost, publicationHosts, links.

contentStatus is full (free post), preview (paid post — public preview only), not_fetched (includeContent off) or failed (the post page could not be fetched; the archive summary is kept).

Tips and tricks

• Mixing posts and comments: both land in the same dataset; filter on recordType, or use the "Comments" view in the Console.
• Daily monitoring: sort: "new" + postedAfter = yesterday. With sort: "top" the archive is not in date order, so the whole archive (up to the page limit) is scanned and filtered.
• Paid posts are flagged, never bypassed: bodyHtml holds what Substack shows logged-out readers, and their comment threads are not public either (0 comment rows even when commentCount is higher).
• Publication metadata (publicationName, tagline, subscriber count) comes from the publication homepage; a few publications hide subscriber counts, in which case the field is null.
• Search uses each publication's own archive search; there is no cross-Substack keyword search without a session, and this Actor deliberately stays session-free.
• Failures (unknown slug, deleted post, unreachable publication, unknown category with the list of valid slugs) are stored under the ERRORS key of the run's key-value store, and the status message reports every shortfall.

Integrations

Export the dataset as JSON, CSV or Excel from the Apify Console, or pull it through the API:

https://api.apify.com/v2/datasets/<datasetId>/items?format=csv

Connect the Actor to Google Sheets, Zapier, Make, Slack, Notion or a webhook with Apify integrations, or call it from your own code with the Apify API client (JavaScript/Python) and schedule it to run daily.