Substack Posts Scraper — Newsletter Archive, Stats & Full Text

Scrape the complete post archive of any Substack publication — including custom-domain publications — with engagement stats and optional full post text. One clean JSON item per post, no login, no API key, no proxy required.

Why use this Actor?

• Whole archives, not just the latest page — Substack's API paginates from newest to oldest with no hard cap; this Actor walks it end-to-end up to your chosen limit.
• Custom domains supported — pass astralcodexten, name.substack.com or https://any-custom-domain.com; all three resolve correctly.
• Engagement data included — likes, comments, word count and paywall status on every post, ready for analysis.
• Full-text mode — set includeBody: true and get clean article text for AI/RAG pipelines (public posts only).
• Paywall-safe — paywalled posts return their public metadata and preview; protected text is never scraped.

How it works

1. Resolve each publication to its Substack API endpoint — bare names (astralcodexten), subdomains (name.substack.com) and custom domains (https://www.noahpinion.blog) all map to the correct archive API.
2. Walk the archive API's pagination from newest to oldest, fetching post metadata (title, subtitle, audience, paywall flag, engagement stats).
3. With includeBody: true, fetch the full text of each public post from its canonical URL — paywalled posts return metadata + preview only.
4. Push one clean JSON item per post to the dataset, then stop at maxPostsPerPublication.

No login, no cookies, no proxy — the public Substack API is used throughout.

What you get per post

Field	Description
title, subtitle, description	Post heading, subheading and preview text
slug, url	Post slug and canonical URL
publishedAt	Publish date (ISO 8601)
type	newsletter, podcast or thread
audience, isPaywalled	Audience setting and paywall flag
wordCount, likes, comments	Engagement metrics
coverImage	Cover image URL
bodyText	Full post text (only when includeBody: true, public posts)

Input

{
  "publications": ["astralcodexten", "https://www.noahpinion.blog"],
  "maxPostsPerPublication": 100,
  "includeBody": false
}

Field	Type	Default	Description
publications	array of strings	—	Bare names (astralcodexten), subdomains (name.substack.com) or full URLs — custom domains work too. Required.
maxPostsPerPublication	integer	50	Maximum posts per publication, newest first (1–5000).
includeBody	boolean	false	Fetch full text of each public post (slower — one extra request per post).

Output (one item per post)

{
  "publication": "https://astralcodexten.substack.com",
  "postId": 158204926,
  "title": "Your Book Review: ...",
  "subtitle": "Or: why the best books are the ones you argue with",
  "slug": "your-book-review",
  "url": "https://www.astralcodexten.com/p/your-book-review",
  "publishedAt": "2026-06-20T12:01:33.000Z",
  "type": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "wordCount": 8214,
  "likes": 312,
  "comments": 447,
  "coverImage": "https://substackcdn.com/image/fetch/...",
  "bodyText": "The full article text when includeBody is enabled..."
}

Use cases

• Newsletter operators — study what performs in your niche before writing your next issue
• Content & trend research — track topics, formats and engagement across publications over time
• AI / RAG pipelines — feed clean long-form text with metadata into embeddings and fine-tuning
• Journalists & analysts — archive and monitor influential newsletters in your beat
• Competitive intelligence — see which headlines and subjects get the most hearts and comments

Run it on a schedule or from your code

Track your favourite publications automatically — create a Schedule in Apify Console (daily, weekly) and every run appends fresh posts to its own dataset. Or call the Actor straight from your pipeline:

curl -X POST "https://api.apify.com/v2/acts/darknezz~substack-posts-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"publications": ["astralcodexten"], "maxPostsPerPublication": 10}'

The same endpoint returns the posts as JSON in one call — handy for scripts and cron jobs. From Python with the official SDK:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("darknezz/substack-posts-scraper").call(
    run_input={"publications": ["astralcodexten", "https://www.noahpinion.blog"],
               "maxPostsPerPublication": 20, "includeBody": True})
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(post["title"], post["likes"], post["wordCount"])

Pricing

Pay per event: you are charged a small fee per post scraped, plus Apify's standard compute costs. No subscription, no minimums — scan one archive or a hundred.

FAQ

Does it get paywalled content? No. Paywalled posts return their public metadata and preview only; bodyText is fetched exclusively for public posts.

How far back does it go? The whole archive — the API paginates from newest to oldest with no hard cap. Set maxPostsPerPublication to bound the run.

Can I scrape a publication on a custom domain? Yes. Pass the full URL and the Actor resolves it to the publication's Substack API endpoint automatically.

Does it need a login or proxy? No — the public Substack API is used; no cookies, no residential proxies.

What if a publication is inactive or deleted? The run reports it as skipped rather than failing the whole batch, so one dead link never wastes the rest of the run.

Why is wordCount 0 for some posts? A few publications strip word counts from their API response. Likes and comments are still populated for every post.

Can I get just new posts since my last run? Yes — schedule the Actor daily, sort by publishedAt in your pipeline, and keep only posts newer than your watermark. The scraper always returns newest-first, so the diff is cheap.

Limitations

• Public posts only for full text — bodyText is fetched exclusively for public posts; paywalled posts return metadata and preview, never protected content.
• wordCount can be 0 — a few publications strip word counts from their API response; likes/comments are still populated.
• Deleted/inactive publications are skipped and reported, not fatal — the rest of the batch still completes.
• Fresh posts can lag by minutes — Substack's API reflects a new post almost immediately, but there is no webhook; a scheduled run is the reliable way to track changes.
• No login-gated content — anything behind Substack's authenticated views (private publications, subscriber-only archives) is out of scope by design.