Scrape Substack newsletter posts into structured JSON

Turn any Substack publication — *.substack.com subdomain or a custom domain like lennysnewsletter.com — into clean, flat, LLM-ready post records. Newsletter researchers, content analysts, and AI teams use this Actor to export full post archives with article text, engagement metrics, tags, and paywall status. One dataset item represents one published post, with fields such as title, authors, publishedAt, bodyText, likes, restacks, commentCount, tags, and paywallStatus.

Best fit for this Actor

• Start with one or more publication URLs (the homepage of a Substack newsletter) when you want a full, paginated post archive.
• Use a single post URL (/p/some-post) when you want to enrich or capture one specific article.
• Give the Actor both *.substack.com subdomains and Substack custom domains; it resolves either to the same JSON source.

This Actor works as a focused standalone workflow: it turns publication URLs into post datasets. For keyword discovery across all of Substack, or for Substack Notes and reader profiles, choose a dedicated tool for that job.

Newsletter competitive-intelligence workflow

A content analyst tracking the AI-and-tech newsletter space starts with a list of publication homepages they already follow (platformer.substack.com, www.lennysnewsletter.com, doomberg.substack.com). They run the Actor with maxPosts: 50 and includeBodyText: true for each. The Actor paginates each archive and returns one record per post with title, publishedAt, wordCount, likes, restacks, commentCount, tags, and paywallStatus. The analyst compares engagement-by-topic across publications in a spreadsheet, spots which posts perform, and identifies sponsorship or cross-promotion candidates using the authors and commentCount fields. A publishedAfter date filter keeps the dataset to the last quarter for a fresh comparison.

Quick start input

{
    "startUrls": [
        { "url": "https://platformer.substack.com" },
        { "url": "https://www.lennysnewsletter.com" }
    ],
    "maxPosts": 20,
    "includeBodyHtml": true,
    "includeBodyText": true
}

Each entry in startUrls becomes one paginated publication scrape, capped at maxPosts per publication. The Actor prints the maximum event cost at the start of every run.

Input reference

Field	Type	What it controls
startUrls	array (required)	Publication homepages or single /p/ post URLs. Accepts objects { "url": "..." }. Both *.substack.com and custom domains resolve to the same JSON source.
maxPosts	integer	Hard cap on posts scraped and billed per publication. Default 50; set lower for cheap tests.
includeBodyHtml	boolean	When true (default), fetches each post's detail page for the complete article HTML. Public posts return full HTML.
includeBodyText	boolean	When true (default), includes a plain-text body derived from the HTML. Disable to reduce payload size.
publishedAfter	string	ISO date (e.g. 2026-01-01); keep posts published on or after this date.
publishedBefore	string	ISO date; keep posts published on or before this date.
audience	enum	all (default), free, or paid posts only. Use free to receive only posts where full body text is available.

What data you receive

One dataset item is one published Substack post. The Actor normalizes Substack's archive and post-detail JSON into a flat record:

{
    "title": "Why Platformer is leaving Substack",
    "url": "https://platformer.substack.com/p/why-platformer-is-leaving-substack",
    "publicationUrl": "https://platformer.substack.com",
    "authors": ["Casey Newton"],
    "publishedAt": "2024-01-12T01:00:49.665Z",
    "paywallStatus": "free",
    "hasBody": true,
    "likes": 727,
    "restacks": 210,
    "commentCount": 55,
    "wordCount": 4030,
    "tags": ["Substack"],
    "scrapedAt": "2026-08-05T20:38:00.000Z"
}

Field	Meaning
title, subtitle, description	Post headline, dek, and SEO description
url, canonicalUrl, slug	Canonical post URL and Substack slug
publicationUrl	Base URL of the publication this post belongs to
authors	Byline author names
publishedAt, updatedAt	Publish and update timestamps (ISO 8601)
paywallStatus	free (full body included) or paid (metadata and teaser)
hasBody	Whether the record includes article body content
bodyHtml, bodyText	Full article HTML and plain-text body for public posts
likes	❤ reaction count
restacks	Restack (Substack reshare) count
commentCount	Total comment count
tags	Post tags
wordCount	Reported word count
coverImageUrl	Cover/lead image URL
podcastUrl, podcastDurationSeconds	Audio attachment for podcast posts
scrapedAt	Run capture timestamp for freshness

Paid posts return paywallStatus: "paid", hasBody: false, and metadata plus the public teaser rather than the paywalled body. The audience field preserves Substack's original value (everyone / only_paid). Choose audience: "free" to receive only posts where the complete body text is available.

Use through the API

Run the Actor synchronously and receive the dataset items in one response:

curl -X POST "https://api.apify.com/v2/acts/khadinakbar~substack-posts-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url":"https://platformer.substack.com"}],
    "maxPosts": 10
  }'

For large archives, start an async run and poll the dataset:

curl -X POST "https://api.apify.com/v2/acts/khadinakbar~substack-posts-scraper/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://platformer.substack.com"}],"maxPosts":100}'

Capture the id and defaultDatasetId from the response, wait for the run to finish, then read the dataset:

curl -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://api.apify.com/v2/datasets/$DATASET_ID/items?clean=true"

Use with AI agents through Apify MCP

Scrape the 20 most recent posts from platformer.substack.com. Return title, authors, publishedAt, bodyText, likes, restacks, commentCount, and tags. Stop at 20 posts and report the terminal outcome.

The Actor accepts publication or post URLs and returns one flat record per post with engagement and provenance fields. Agents should inspect the run's outcome (COMPLETE, PARTIAL, VALID_EMPTY, UPSTREAM_FAILED, or CONFIG_ERROR) in the OUTPUT / RUN_SUMMARY key-value store records, then read the dataset. Cost is predictable: one post-scraped event per returned post plus the platform start fee. Set up the connection through the Apify MCP configurator for your client.

Pricing

This Actor uses Pay per event plus Apify platform usage. You pay the event charges plus the underlying Apify platform compute and proxy usage for the run. When budgeting, open the live Pricing tab for the current event details, and use Apify's run cost controls to keep the workflow aligned with your budget.

• Actor start — a small per-run fee scaled by allocated RAM.
• Post scraped — the primary billable event, charged once per post returned.

The Actor prints the maximum event cost at the start of every run, so you can forecast spend before any charge fires.

Best results

• Start with publication homepage URLs for full-archive exports; the Actor paginates automatically.
• Use a single /p/ post URL for targeted one-article capture or enrichment.
• Set audience: "free" to receive only posts where full body text is available.
• Disable includeBodyHtml and includeBodyText when you only need metadata and engagement — this roughly halves the runtime by skipping detail-page fetches.
• Use publishedAfter / publishedBefore to bound large archives to a relevant window.

Builder's note

While building this Actor I learned that Substack exposes two clean, unauthenticated JSON surfaces: a paginated archive endpoint that returns the post list with engagement, and a window._preloads JSON blob embedded in each post page that carries the full body_html. This means the Actor reads the same JSON the Substack frontend consumes, with no browser, login, or anti-bot workaround needed. I designed the custom-domain resolution after finding that domains like lennysnewsletter.com serve the identical JSON API as *.substack.com subdomains, so the Actor resolves both to the same source. Paid posts are handled honestly — the paywallStatus field signals when only the public teaser is available rather than the full body, so downstream analysis can filter confidently.

Responsible use

This Actor reads publicly accessible data from Substack publication pages. It respects paywalls, uses no login, and collects only public content. You are responsible for complying with Substack's Terms of Service, each publication's terms, and applicable copyright laws. Use scraped content for lawful purposes such as research, analysis, and personal use, and seek permission before republishing copyrighted material.