Substack Scraper — posts, comments, newsletters, authors & leaderboards

Substack scraper and Substack API alternative. Export posts with full text and Markdown (ready for AI / RAG), public comments, newsletter profiles with subscriber counts and subscription prices, author profiles with social links, Substack-wide search and category leaderboards (Bestsellers and Rising) — all in one actor, no login, no cookies, no browser. Pay only per result, no start fee.

• ✅ Any newsletter: astralcodexten, https://lenny.substack.com, custom domains like https://www.lennysnewsletter.com
• ✅ Full post content as clean text, Markdown and/or HTML, plus likes, comments count, restacks, word count, tags, authors
• ✅ Public comments and replies with author, date, likes and thread depth
• ✅ Newsletter data: subscriber count ("Over 1,200,000 subscribers" → 1200000), paid plans and prices, bestseller tier, podcast, sections, contributors
• ✅ Leaderboards for 30+ categories (Technology, Business, Finance, Culture, Politics, Science, Crypto...): Bestsellers (paid) and Rising (trending)
• ✅ Search all of Substack: posts, or people & newsletters (creator discovery, lead lists)
• ✅ Monitor mode: scheduled runs return only new posts — you never pay twice for the same post
• ✅ Filters: date range ("7 days", 2026-01-01), free vs paywalled, keyword search inside a newsletter, newest or most popular first
• ✅ From $0.30 per 1,000 comments and $0.70 per 1,000 posts. No start fee. Error rows are free.

What can you do with this Substack scraper?

Use case	How
Newsletter market research	Export the Bestsellers and Rising leaderboards of any category with subscriber counts, prices and authors. Spot fast-growing newsletters before everybody else.
Competitor & content analysis	Pull the full archive of competing newsletters: topics, posting frequency, word count, likes, comments and restacks per post. Find what performs best.
Creator discovery & influencer marketing	Search "people & newsletters" for any topic and get writers with subscriber/follower counts, bios and links (X/Twitter, LinkedIn, website, YouTube).
Sponsorship & partnership sales	Build lead lists of newsletters by category and size, with pricing and contributor names.
AI / LLM / RAG pipelines	Get clean Markdown or plain text of posts to feed embeddings, summarisation, or fine-tuning datasets.
Audience & sentiment research	Export public comments with likes and reply threads to measure what readers care about.
Monitoring & alerts	Schedule hourly/daily runs in monitor mode and send new posts to Slack, Discord, Google Sheets or a webhook.
Academic & media research	Archive what public newsletters publish, with timestamps and authors.

Quick start (under 1 minute)

1. Click Try for free and paste one or more newsletter URLs (https://www.astralcodexten.com, lennysnewsletter.com).
2. Choose Posts, Search or Leaderboards, set the max posts, and decide whether you want full text/Markdown and comments.
3. Click Start. The pre-filled input (10 posts with full text) finishes in about 10 seconds. Download JSON, CSV or Excel, or schedule it as a monitor.

Input

Field	What it does
publications	Newsletters to scrape (subdomain, Substack URL or custom domain). Post and profile URLs pasted here are detected automatically.
postUrls	Individual posts (/p/slug, open.substack.com/pub/..., substack.com/home/post/p-123).
authors	Profiles (lenny, @lenny, https://substack.com/@lenny).
searchQueries + searchType	Search Substack for posts, people (writers & newsletters) or both; maxSearchResults per query.
leaderboardCategories + leaderboardRanking	Category names/slugs (technology, finance, us-politics...); ranking paid (Bestsellers), trending (Rising), both, or all (category directory); maxLeaderboardEntries.
maxPostsPerPublication	Posts per newsletter (default 10; 0 = only the newsletter profile).
postSort	new (newest first) or top (most popular first).
archiveSearch	Keyword search inside the given newsletters.
dateFrom / dateTo	2026-01-01, ISO date-time, or relative (7 days, 6 months, 1 year).
audience	all, free or paid (paywalled) posts.
includeContent / contentFormats	Full body as text, markdown and/or html (default text + Markdown). Off = cheaper metadata-only rows.
includeComments / maxCommentsPerPost / commentSort	Public comments with replies (best_first, most_recent_first, oldest_first).
includePublicationInfo	One profile row per newsletter (default on).
onlyNewPosts / monitorStateKey	Monitor mode: only posts newer than the last run.
proxyConfiguration	Apify Proxy (datacenter is usually enough).

Example: full archive of a newsletter, as Markdown, with comments

{
  "publications": ["https://www.lennysnewsletter.com"],
  "maxPostsPerPublication": 1000,
  "contentFormats": ["markdown"],
  "includeComments": true,
  "maxCommentsPerPost": 200
}

Example: Substack leaderboards (Bestsellers + Rising) for three categories

{
  "leaderboardCategories": ["technology", "finance", "business"],
  "leaderboardRanking": "both",
  "maxLeaderboardEntries": 100,
  "maxPostsPerPublication": 0
}

Example: find newsletters and writers about a topic

{
  "searchQueries": ["climate tech", "product management"],
  "searchType": "both",
  "maxSearchResults": 50,
  "includeContent": false
}

Example: daily monitor of new posts (no double charges)

{
  "publications": ["astralcodexten", "https://www.noahpinion.blog", "lenny"],
  "onlyNewPosts": true,
  "monitorStateKey": "my-daily-newsletters",
  "includePublicationInfo": false,
  "maxPostsPerPublication": 50
}

Schedule it in Apify Console → Schedules and connect an integration (Slack, Google Sheets, webhook...). Runs with nothing new cost $0. Keep includePublicationInfo on if you also want a daily subscriber-count snapshot (one publication row per newsletter per run).

Output

Every row has a type: post, comment, publication, leaderboard_entry, author or error (errors are free). The dataset has ready-made views: Posts, Publications & leaderboards, Comments, Authors. Real sample (shortened; full sample in sample-output.json):

{
  "type": "post",
  "post_id": 221215977,
  "url": "https://www.astralcodexten.com/p/why-global-trepanation",
  "title": "Why Global Trepanation?",
  "post_date": "2026-10-21T21:53:11.440Z",
  "audience": "everyone",
  "is_paywalled": false,
  "publication_name": "Astral Codex Ten",
  "publication_url": "https://www.astralcodexten.com",
  "authors": [{"name": "Scott Alexander", "handle": "astralcodexten", "profile_url": "https://substack.com/@astralcodexten"}],
  "wordcount": 1307,
  "reaction_count": 479,
  "restacks": 22,
  "comment_count": 223,
  "tags": [],
  "cover_image": "https://substack-post-media.s3.amazonaws.com/public/images/...jpeg",
  "content_is_preview": false,
  "body_text": "A friend asked me to investigate the following mystery: why do 5-10% of skulls...",
  "body_markdown": "A friend asked me to investigate the following mystery: why do [5-10% of skulls](https://pmc.ncbi.nlm.nih.gov/...)...",
  "source": "publication"
}

{
  "type": "leaderboard_entry",
  "rank": 1,
  "ranking": "bestseller",
  "category_name": "Technology",
  "name": "SemiAnalysis",
  "url": "https://newsletter.semianalysis.com",
  "author_name": "Dylan Patel",
  "subscriber_count": 326000,
  "subscriber_count_text": "Over 326,000 subscribers",
  "paid_subscriber_range": "Tens of thousands of paid subscribers",
  "monthly_price": 50.0,
  "yearly_price": 500.0,
  "currency": "USD"
}

{
  "type": "author",
  "name": "Nathan Lambert",
  "handle": "natolambert",
  "subscriber_count": 87000,
  "follower_count": 130209,
  "primary_publication_name": "Interconnects AI",
  "primary_publication_url": "https://www.interconnects.ai",
  "twitter_handle": "natolambert",
  "links": [{"type": "linkedin", "label": "LinkedIn", "url": "https://www.linkedin.com/in/natolambert/"}],
  "leaderboard": {"ranking": "paid", "rank": 40, "label": "Technology"}
}

{
  "type": "comment",
  "comment_id": 366969428,
  "post_url": "https://www.astralcodexten.com/p/why-global-trepanation",
  "parent_comment_id": null,
  "depth": 0,
  "body": "Yet another example of people trying to be holeier-than-thou",
  "date": "2026-10-21T21:59:19.244Z",
  "author_name": "The Shallow Diver",
  "author_handle": "shallowdiver",
  "reaction_count": 108,
  "reply_count": 6
}

Post fields: post_id, url, slug, title, subtitle, description, post_date, updated_at, post_type (newsletter, podcast, thread, video), audience, is_paywalled, publication_id, publication_name, publication_url, authors, wordcount, reaction_count, reactions, restacks, comment_count, child_comment_count, tags, section_name, language, cover_image, podcast_url, podcast_duration, preview_text, content_is_preview, body_text, body_markdown, body_html, source, search_query, scraped_at.

Publication / leaderboard fields: publication_id, name, url, subdomain, custom_domain, description, about_text, language, logo_url, created_at, first_post_date, author_* (id, name, handle, profile URL, bio, photo), twitter_handle, subscriber_count, subscriber_count_text, subscriber_range, paid_subscriber_range, paid_subscribers_order_of_magnitude, bestseller_tier, paid_subscriptions_enabled, monthly_price, yearly_price, founding_price, currency, has_podcast, community_chat_enabled, sections, contributors; leaderboard rows add rank, ranking, ranking_title, category_id, category_name, category_slug.

Pricing

Pay per event — you only pay for rows you get:

Event	Price	Per 1,000
Post (metadata only)	$0.0007	$0.70
Post with full content	$0.0014	$1.40
Comment	$0.0003	$0.30
Publication profile	$0.0015	$1.50
Leaderboard entry	$0.0015	$1.50
Author profile	$0.0015	$1.50
Run start	free	—
Error rows	free	—

Examples: the 100 Bestsellers of a category ≈ $0.15. 1,000 posts with full text ≈ $1.40. A daily monitor of 5 newsletters costs cents per month, and runs with nothing new cost $0. Set Max cost per run in the run options to cap any run: the scraper stops cleanly when the limit is reached.

FAQ

Do I need a Substack account, cookies or an API key? No. The actor reads the public JSON endpoints that the Substack website itself uses, plus public newsletter pages.

Can it get paywalled (paid-only) posts? Only the free preview that Substack shows to every visitor (marked content_is_preview: true, is_paywalled: true). It never logs in or bypasses the paywall. Comments on paid-only posts are not public and are skipped.

How accurate are subscriber counts? They are the public figures Substack displays ("Over 1,200,000 subscribers", "87K+"), converted to numbers. Some newsletters hide them (null).

Which leaderboard categories exist? All main Substack categories: culture, technology, business, us-politics, finance, food, sports, art, world-politics, health-politics, news, fashionandbeauty, music, faith, climate, science, literature, fiction, health, design, travel, parenting, philosophy, comics, international, crypto, history, humor, education, film-and-tv, home-garden, games. Substack does not publish rankings for subcategories (you get a free error row suggesting the parent category).

How far back can it go? The whole public archive. Raise maxPostsPerPublication, or use dateFrom / dateTo.

Does it work with custom domains? Yes (www.lennysnewsletter.com, www.noahpinion.blog, ...). Substack redirects are followed automatically.

What about Substack Notes and chat? Not included in this version (Notes require a separate feed); posts, comments, profiles, search and leaderboards are covered.

Is it legal? It only reads information that writers chose to publish openly on the web, without logging in or bypassing any protection. You are responsible for how you use the data (e.g. GDPR when processing personal data such as commenter names).

Is this the official Substack API? No. Substack has no official public API; this is an independent tool that works as a simple, no-code Substack API alternative.

Limits

• Paywalled posts return only the free preview Substack shows to every visitor (content_is_preview: true). The Actor never logs in.
• Comments on paid-only posts are not public and are skipped.
• Subscriber counts are the rounded public figures Substack shows ("Over 1,200,000"); some newsletters hide them.
• Substack Notes and chat are not included. Leaderboards exist only for the main categories, not subcategories.

Use it from code (API)

Every Apify Actor is also an API. Replace <YOUR_API_TOKEN> with the token from your Apify account (Settings, API & Integrations).

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });
const run = await client.actor('phantom-x64/substack-scraper').call({
    "publications": [
      "https://www.astralcodexten.com"
    ],
    "maxPostsPerPublication": 10
  });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('<YOUR_API_TOKEN>')
run = client.actor('phantom-x64/substack-scraper').call(run_input={
      "publications": [
        "https://www.astralcodexten.com"
      ],
      "maxPostsPerPublication": 10
    })
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

You can also start it from Make, Zapier, n8n or Google Sheets with the Apify integrations, schedule it, or call it as a tool from an AI agent through the Apify MCP server.

Support and feedback

Found a bug, need a new field or another source? Open an issue in the Issues tab of this Actor page. Reports with a run link are fixed fastest. If it saves you time, a short review on the Store page helps other people find it.