Substack Posts Scraper avatar

Substack Posts Scraper

Pricing

from $5.00 / 1,000 post scrapeds

Go to Apify Store
Substack Posts Scraper

Substack Posts Scraper

Scrape Substack newsletter posts from any publication on a subdomain or custom domain. Returns full post text, engagement, tags and paywall status.

Pricing

from $5.00 / 1,000 post scrapeds

Rating

0.0

(0)

Developer

Khadin Akbar

Khadin Akbar

Maintained by Community

Actor stats

0

Bookmarked

1

Total users

0

Monthly active users

5 days ago

Last modified

Features

Limited permissions

Share

What is Substack Posts Scraper?

Turn any Substack publication — *.substack.com subdomain or a custom domain like lennysnewsletter.com — into clean, flat, LLM-ready post records. Newsletter researchers, content analysts, and AI teams use this Actor to export full post archives with article text, engagement metrics, tags, and paywall status. One dataset item represents one published post, with fields such as title, authors, publishedAt, bodyText, likes, restacks, commentCount, tags, and paywallStatus.

Best fit for this Actor

  • Start with one or more publication URLs (the homepage of a Substack newsletter) when you want a full, paginated post archive.

  • Use a single post URL (/p/some-post) when you want to enrich or capture one specific article.

  • Give the Actor both *.substack.com subdomains and Substack custom domains; it resolves either to the same JSON source.

  • Leave the URLs empty and give a topic (searchTopic, for example ai agents) to find posts across all of Substack.

  • Switch on extras when you need them: reader comments, people who liked a post, the author's Notes, plus a free publication profile row (author bio, rounded subscriber count, public social links).

Everything comes from public Substack pages, with no login. Private data such as hidden subscriber lists or email addresses is never collected; the only emails returned are ones the author wrote in their own public bio.

Newsletter competitive-intelligence workflow

A content analyst tracking the AI-and-tech newsletter space starts with a list of publication homepages they already follow (platformer.substack.com, www.lennysnewsletter.com, doomberg.substack.com). They run the Actor with maxPosts: 50 and includeBodyText: true for each. The Actor paginates each archive and returns one record per post with title, publishedAt, wordCount, likes, restacks, commentCount, tags, and paywallStatus. The analyst compares engagement-by-topic across publications in a spreadsheet, spots which posts perform, and identifies sponsorship or cross-promotion candidates using the authors and commentCount fields. A publishedAfter date filter keeps the dataset to the last quarter for a fresh comparison.

Quick start input

{
"startUrls": [
{ "url": "https://platformer.substack.com" }
],
"maxPosts": 10,
"includeBodyHtml": true,
"includeBodyText": true
}

Each entry in startUrls becomes one publication scrape, newest posts first, capped at maxPosts per publication. This example returns 10 posts and costs about $0.015 on the Free plan (less on paid plans), plus one free publication profile row. The Actor prints the maximum post charge at the start of every run.

Input reference

FieldTypeWhat it controls
startUrlsarrayPublication homepages or single /p/ post URLs. Accepts objects { "url": "..." }. Optional when searchTopic is set. Both *.substack.com and custom domains resolve to the same JSON source.
maxPostsintegerHard cap on posts scraped and billed per publication. Default 50; set lower for cheap tests.
includeBodyHtmlbooleanWhen true (default), fetches each post's detail page for the complete article HTML. Public posts return full HTML.
includeBodyTextbooleanWhen true (default), includes a plain-text body derived from the HTML. Disable to reduce payload size.
publishedAfterstringISO date (e.g. a source-provided date); keep posts published on or after this date.
publishedBeforestringISO date; keep posts published on or before this date.
audienceenumall (default), free, or paid posts only. Use free to receive only posts where full body text is available.
sortenumnew (default, newest first) or top (most popular first).
searchQuerystringOptional keyword search inside each publication, so you get only matching posts.
searchTopicstringFinds posts about a topic across all of Substack (about 20 per page), billed per post. Combine with maxPosts, publishedAfter and audience.
includePublicationProfilebooleanDefault on. One free row per publication (recordType: "publication"): author bio, subscriberCountLabel / subscriberCountApprox (Substack rounds these, for example "175K+"), followerCount, public social links.
includeReactors, maxReactorsPerPostboolean, integerAdds one row per person who liked a post (recordType: "reactor"). Substack shows at most 100 per post. Default off.
includeNotes, maxNotesPerPublicationboolean, integerAdds the lead author's public Notes (recordType: "note": text, date, likes, restacks, replies, link). Default off.
includeBodyMarkdownbooleanAdds a bodyMarkdown field with the article as clean Markdown for AI and search pipelines. Default off; no extra charge.
includeComments, maxCommentsPerPostboolean, integerReturns reader comments as extra rows (recordType: "comment", replies linked by parentCommentId). Billed per comment. Substack hides the comments of paywalled posts from the public, so only comments on free posts come back.

What data you receive

One dataset item is one published Substack post. The Actor normalizes Substack's archive and post-detail JSON into a flat record:

{
"title": "Why Platformer is leaving Substack",
"url": "https://platformer.substack.com/p/why-platformer-is-leaving-substack",
"publicationUrl": "https://platformer.substack.com",
"authors": ["Casey Newton"],
"publishedAt": "2024-01-12T01:00:49.665Z",
"paywallStatus": "free",
"hasBody": true,
"likes": 727,
"restacks": 210,
"commentCount": 55,
"wordCount": 4030,
"tags": ["Substack"],
"scrapedAt": "2026-08-05T20:38:00.000Z"
}
FieldMeaning
title, subtitle, descriptionPost headline, dek, and SEO description
url, canonicalUrl, slugCanonical post URL and Substack slug
publicationUrlBase URL of the publication this post belongs to
authorsByline author names
publishedAt, updatedAtPublish and update timestamps (ISO 8601)
paywallStatusfree (full body included) or paid (metadata and teaser)
hasBodyWhether the record includes article body content
bodyHtml, bodyTextFull article HTML and plain-text body for public posts
likes❤ reaction count
restacksRestack (Substack reshare) count
commentCountTotal comment count
tagsPost tags
wordCountReported word count
coverImageUrlCover/lead image URL
podcastUrl, podcastDurationSecondsAudio attachment for podcast posts
bodyMarkdownArticle as Markdown (only when includeBodyMarkdown is on)
recordTypepost for posts; comment for comment rows
scrapedAtRun capture timestamp for freshness

Paid posts return paywallStatus: "paid", hasBody: false, and metadata plus the public teaser rather than the paywalled body. The audience field preserves Substack's original value (everyone / only_paid). Choose audience: "free" to receive only posts where the complete body text is available.

Use through the API

Run the Actor synchronously and receive the dataset items in one response:

curl -X POST "https://api.apify.com/v2/acts/khadinakbar~substack-posts-scraper/run-sync-get-dataset-items" \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"startUrls": [{"url":"https://platformer.substack.com"}],
"maxPosts": 10
}'

For large archives, start an async run and poll the dataset:

curl -X POST "https://api.apify.com/v2/acts/khadinakbar~substack-posts-scraper/runs" \
-H "Authorization: Bearer $APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{"startUrls":[{"url":"https://platformer.substack.com"}],"maxPosts":100}'

Capture the id and defaultDatasetId from the response, wait for the run to finish, then read the dataset:

curl -H "Authorization: Bearer $APIFY_TOKEN" \
"https://api.apify.com/v2/datasets/$DATASET_ID/items?clean=true"

Use with AI agents through Apify MCP

Scrape the 20 most recent posts from platformer.substack.com. Return title, authors, publishedAt, bodyText, likes, restacks, commentCount, and tags. Stop at 20 posts and report the terminal outcome.

The Actor accepts publication or post URLs and returns one flat record per post with engagement and provenance fields. Agents should inspect the run's outcome (COMPLETE, PARTIAL, VALID_EMPTY, UPSTREAM_FAILED, or INVALID_INPUT - a bad or missing input ends as a normal finished run with that outcome and no charge) in the OUTPUT / RUN_SUMMARY key-value store records, then read the dataset. Cost is predictable: one post-scraped event per returned post plus the platform start fee. Set up the connection through the Apify MCP configurator for your client.

Pricing

This Actor uses pay per event. The platform usage of each run is paid by the Actor owner, not added to your bill. Higher Apify plans pay less per item (Free, Bronze, Silver, Gold; Platinum and Diamond pay the Gold price):

EventFreeBronzeSilverGold
Post scraped (full text included)$0.0015$0.0012$0.0009$0.0007
Comment scraped$0.0008$0.0007$0.0005$0.0004
Post liker scraped$0.0004$0.0003$0.00025$0.0002
Note scraped$0.0008$0.0007$0.0005$0.0004

Plus Actor start, $0.00005 per run. The publication profile row is free.

Job (Free plan)What you pay
10 postsabout $0.015
One publication archive, 100 postsabout $0.15
1,000 postsabout $1.50
100 posts with 20 comments eachabout $0.31

Posts that cannot be fetched with their full text, filtered-out posts and failed runs are not charged. The Actor prints the maximum post charge at the start of every run, and Apify's maximum total charge setting stops the run before you pay more than you set. The live Pricing tab always shows the current prices.

Best results

  • Start with publication homepage URLs for full-archive exports; the Actor pages through the archive automatically.
  • Use a single /p/ post URL for targeted one-article capture or enrichment.
  • Set audience: "free" to receive only posts where full body text is available.
  • Turn off includeBodyHtml, includeBodyText and includeBodyMarkdown when you only need metadata and engagement; the Actor then skips the per-post text requests and finishes much faster.
  • Memory is chosen automatically from maxPosts (256 MB up to 150 posts, 512 MB up to 800, 1 GB above). The Actor fetches six posts at a time and slows itself down if Substack asks it to.
  • Use publishedAfter / publishedBefore to bound large archives to a relevant window.

Builder's note

While building this Actor I learned that Substack exposes two clean, unauthenticated JSON surfaces: a cursor-paged archive endpoint that returns the post list with engagement, and a per-post JSON route (with the post page as backup) that carries the full body_html. This means the Actor reads the same JSON the Substack frontend consumes, with no browser, login, or anti-bot workaround needed. I designed the custom-domain resolution after finding that domains like lennysnewsletter.com serve the identical JSON API as *.substack.com subdomains, so the Actor resolves both to the same source. Paid posts are handled honestly — the paywallStatus field signals when only the public teaser is available rather than the full body, so downstream analysis can filter confidently.

Continue the workflow

Responsible use

This Actor reads publicly accessible data from Substack publication pages. It respects paywalls, uses no login, and collects only public content. You are responsible for complying with Substack's Terms of Service, each publication's terms, and applicable copyright laws. Use scraped content for lawful purposes such as research, analysis, and personal use, and seek permission before republishing copyrighted material.