Substack Scraper: Posts, Notes and Profiles

Substack Scraper: Posts, Notes and Profiles

This Substack scraper covers every public Substack surface in one actor: full posts with complete article HTML, nested comment threads, Substack Notes search, author profiles with subscriber signals, publication profiles, and category leaderboards. No login, no cookies, no proxy configuration, no code.

Quick start

1. Put one or more publications in publications, for example astralcodexten.com, slowboring.com or a bare name. Custom domains, *.substack.com subdomains and full URLs all work; redirect and www variants are retried automatically instead of failing with a 404.
2. Turn on what you need: full article content, comment threads, the publication profile. Or skip publications entirely and use authorHandles, noteSearchQueries or leaderboardCategories.
3. Set maxPostsPerPublication or maxItemsTotal so the cost is bounded before you press Start, then run. Export JSON, CSV or Excel, or read the dataset over the Apify API.

Everything this Actor covers

What you want	How	What you get
Posts from a publication	publications: ["astralcodexten.com"]	Title, subtitle, date, authors, likes, comment count, restacks, cover image, podcast audio URL and duration, paywall flag
Full article content	toggle Include full article content	Complete bodyHtml plus clean bodyText plus word count, ready for research, RAG, and AI datasets
Single posts	postUrls: ["https://…/p/slug"]	The same rich record for exactly the posts you point at
Comment threads	toggle Include comment threads	The full nested tree, flattened with parentId and depth, author, likes
Search inside a publication	Search within publications	Server-side archive search (sort: new/top, date window)
Author profiles	authorHandles: ["@handle"]	Follower count, subscriber signals ("Thousands of subscribers"), bestseller tier, bio, social links, primary publication
Authors' Notes	toggle Include authors' recent Notes	Their recent short-form Notes with engagement
Notes keyword search	noteSearchQueries: ["ai agents"]	Platform-wide Notes matching your query: text, author, likes, restacks, images
Category leaderboards	leaderboardCategories: ["technology"]	Ranked publications per category with subscriber signals, paid tier, author: the market map of Substack

Output example (post with content)

{
  "type": "post",
  "publicationHost": "www.astralcodexten.com",
  "id": "212058143",
  "slug": "your-review-tomorrow-and-tomorrow",
  "title": "Your Book Review: Tomorrow, and Tomorrow, and Tomorrow",
  "url": "https://www.astralcodexten.com/p/your-review-tomorrow-and-tomorrow",
  "postDate": "2026-08-22T13:01:33.000Z",
  "postType": "newsletter",
  "audience": "everyone",
  "paywalled": false,
  "authors": [{ "id": 12009663, "name": "Scott Alexander", "handle": "astralcodexten" }],
  "reactionCount": 214,
  "commentCount": 361,
  "podcast": null,
  "bodyHtml": "<p>…full article HTML…</p>",
  "bodyText": "…clean plain text…",
  "wordCount": 5843,
  "contentRetrieved": true
}

Every record type (post, comment, note, author-profile, publication-profile, leaderboard-entry, status) is a flat, documented shape.

Honest paywall handling, read this

• Paywalled posts are delivered as metadata with paywalled: true and an empty body.
• They are never charged at the full-content price. You pay the Post price only.
• This Actor does not bypass paywalls, ever. If you need paid-post bodies, subscribe to the publication.

How much does it cost to scrape Substack?

You pay per delivered record, so a run costs exactly what it delivers. Zero results means zero record charges, with a status record explaining why. Apify Store tiers take 10% off (Silver) or 20% off (Gold) every result event; the Actor start event is the same for everyone.

Event	Free and Bronze	Silver (10% off)	Gold (20% off)
Actor Start	$0.005	$0.005	$0.005
Post	$0.002	$0.0018	$0.0016
Article body (only when the body is actually delivered)	$0.002	$0.0018	$0.0016
Comment	$0.0005	$0.00045	$0.0004
Note	$0.001	$0.0009	$0.0008
Author profile	$0.005	$0.0045	$0.004
Publication profile	$0.005	$0.0045	$0.004
Leaderboard entry	$0.001	$0.0009	$0.0008

Worked examples at the Free and Bronze prices: 25 latest posts with full content = $0.105. A 100-entry technology leaderboard = $0.105. 50 Notes on a keyword = $0.055. An author profile plus their 50 recent Notes = $0.06.

Input

Field	Type	Notes
publications	array	Publications to pull posts from: bare name, *.substack.com, custom domain or full URL
includeContent	boolean	Fetch each post's full body (HTML, plain text, word count)
includeComments	boolean	Fetch the nested comment tree, flattened with parentId and depth
includePublicationProfile	boolean	One publication-profile record per publication
sort	string	Archive order: new (latest first) or top (most popular first)
searchInPublication	string	Server-side keyword search inside each publication's archive
publishedAfter	string	Only posts published on or after this date (YYYY-MM-DD)
maxPostsPerPublication	integer	Cap on charged post records per publication, honored exactly
postUrls	array	Direct post URLs to fetch individually
maxCommentsPerPost	integer	Cap on charged comment records per post
authorHandles	array	Author handles: @handle, handle, or the full profile URL
includeAuthorNotes + maxNotesPerAuthor	boolean, integer	Each author's recent public Notes, capped
noteSearchQueries + maxNotesPerQuery	array, integer	Platform-wide Notes keyword search, capped
leaderboardCategories + maxPublicationsPerCategory	array, integer	Category leaderboards, capped
maxItemsTotal	integer	Overall cap on charged records for the run

What this Actor does NOT do

• ❌ No paywall bypass. Paid post bodies stay with paying subscribers.
• ❌ No global post search. Substack's site-wide post search requires a logged-in session, so it is not offered. Per-publication archive search is offered (server-side), and Notes search is platform-wide.
• ❌ No people search by keyword. That endpoint is login-gated; profile lookup by handle is offered.
• ❌ No private data. Emails, subscriber lists and revenue are not public and are not returned. Subscriber signals are the same rounded figures Substack shows publicly ("Thousands of subscribers", bestseller tiers).
• ❌ No exact subscriber counts where Substack publishes only rounded or hidden values. You get null or the public text, never a guess.

Reliability notes

• Pure HTTP against Substack's public JSON endpoints, fast and cheap (the whole default demo run finishes in about 2 seconds).
• Polite pacing plus automatic retry with backoff on rate limits.
• Every cap (maxPostsPerPublication, maxItemsTotal and the rest) is honored exactly, so cost is bounded before you press Start.
• If anything fails (unknown publication, bad handle, unknown category) you get a plain-English status record and no charge for it.

Fair use and compliance

This Actor reads the same public, unauthenticated endpoints that power Substack's own web pages, at a polite request rate, and collects only published content and public profile data. You are responsible for complying with applicable laws, Substack's terms, and content licenses for your use case. Respect authors: cite what you quote.

FAQ

Can I see how many subscribers a Substack has?

You get the same public subscriber signals Substack itself shows, such as "Thousands of subscribers", bestseller tiers, follower counts and a publication's payments state. Substack does not publish exact subscriber numbers, so this Actor returns null or the public text rather than an invented figure.

Does this extract paid or paywalled post content?

No. Paywalled posts come back as metadata with paywalled: true and an empty body, and they are never charged at the Article body price. This Actor does not bypass paywalls; if you need paid bodies, subscribe to the publication.

Can I track a Substack newsletter's new posts automatically?

Yes. Schedule the Actor on Apify with sort: new, a publishedAfter date and a small maxPostsPerPublication, so each run only reaches for the newest posts. Because you pay per delivered record, a run that finds nothing new costs only the Actor start event.

How do I scrape a Substack newsletter archive?

Put the publication in publications, choose sort: new or sort: top, and raise maxPostsPerPublication to how deep you want to go. Add includeContent for the full article HTML, text and word count, publishedAfter to bound the window, and searchInPublication to run a server-side keyword search inside the archive.

Does Substack have a public API?

There is no documented, supported public API you can sign up for. This Actor reads the same public JSON endpoints that power Substack's own web pages, with no login, no cookies and no browser, which is why the default demo run finishes in about 2 seconds.

How do I find the top newsletters in a Substack category?

Use leaderboardCategories, for example ["technology"], and cap it with maxPublicationsPerCategory. Each entry is a ranked publication with its public subscriber signals, paid tier and author, which together read as a market map of that category.

Why does this Actor run with limited permissions?

Least privilege. The Actor only reads and writes its own storages: the run's dataset and key-value store. It needs no access to anything else on your account, and it never authenticates to Substack.

More data tools by Automation Craft

• Telegram Channel Scraper: Posts, Views, Dates for public channel posts with a true date filter.
• G2 Reviews Scraper: Ratings, Pros and Cons for software reviews without blocked runs.
• Meta Ad Library Scraper - All Placements, Filters for ads across every Meta placement.
• Instagram Ads Library Scraper - Creatives, Video for Instagram creatives and video ads.
• Google Trends Scraper - Compare and Trending Now for the search demand behind a topic.
• LinkedIn Jobs Scraper - No Login, Real Dates for hiring signals, no login required.

Missing an angle you need? Open an issue on the Actor's Issues tab. The roadmap is complaint-driven.