Substack Scraper — Posts, Comments & Newsletter Intelligence

The newsletter economy runs on Substack — tens of millions of active subscriptions, thousands of paid writers — and none of it has a public API or bulk export. This actor turns any Substack publication into structured data: posts with full article text, likes and comment counts, paywall status, reader comments, and publication metadata. Search by keyword across the whole platform or target specific newsletters. No login required. Export to JSON, CSV, or Excel.

Why Use This Scraper?

Substack has grown into one of the most important platforms for independent journalism, thought leadership, and niche expertise. With over 35 million active subscriptions and 17,000+ paid writers, it's a goldmine for researchers, marketers, and analysts — but Substack offers no bulk export or public API.

This actor solves that. It programmatically extracts posts, comments, and publication info from any Substack newsletter, giving you clean, structured data ready for analysis.

Key Features

• Three scrape modes: Posts, comments, and publication info
• Search across Substack: Find posts by keyword across the entire platform
• Publication-specific scraping: Target one or more newsletters by subdomain
• Full article text: Optionally include the complete body text of each post
• Flexible sorting: Sort by newest or top-performing posts
• Scale control: Scrape from 1 to 500 items per run
• No authentication needed: Works without any Substack account
• Multiple export formats: JSON, CSV, Excel, XML, HTML

Use Cases

Sponsor & Advertiser Research

Evaluating a newsletter for sponsorship? Scrape its last 50 posts and you have publishing cadence, engagement per post (likes, comments), paywall ratio, and topic mix — the due-diligence data a media kit won't show you. At $0.005/result that's a $0.25 audit per newsletter.

1. Content Research & Competitive Analysis

Track what topics are trending across newsletters in your industry. Monitor competitors' publishing frequency, engagement, and content strategy.

2. Media Monitoring & PR Intelligence

Set up regular scrapes to track mentions of your brand, product, or industry across Substack newsletters. Stay ahead of narratives before they hit mainstream media.

3. Academic & Market Research

Collect large datasets of expert opinion pieces, industry analysis, and commentary for qualitative research. Study how narratives form and spread through independent media.

4. Newsletter Discovery & Curation

Search for newsletters covering specific topics, then scrape their publication info to evaluate subscriber counts, posting cadence, and content quality.

5. Sentiment & Trend Analysis

Extract posts about specific topics or companies, then run NLP or sentiment analysis on the text. Detect shifts in expert opinion over time.

6. Lead Generation for B2B

Find Substack authors writing about your domain and extract their publication details. These are high-value contacts who are actively engaged in your space.

7. Content Repurposing & Summarization

Pull posts from newsletters you subscribe to and feed them into LLMs for summarization, translation, or content repurposing workflows.

Input Parameters

Parameter	Type	Required	Default	Description
publications	Array of strings	No	—	Substack subdomains to scrape (e.g., platformer for platformer.substack.com)
searchQuery	String	No	—	Search keyword to find posts across all of Substack
scrapeType	String	No	posts	What to scrape: posts, comments, or info
maxItems	Integer	No	50	Maximum items to return (1–500)
sortBy	String	No	new	Sort order: new (newest first) or top (most popular)
includeBodyText	Boolean	No	false	Include the full body text of each post

Tip: Use publications to target specific newsletters, or searchQuery to search across the entire platform. You can combine both.

Sample Output

Posts Output

{
  "title": "The AI Trust Crisis",
  "subtitle": "Why users are losing faith in AI-generated content",
  "slug": "the-ai-trust-crisis",
  "publishedAt": "2026-03-01T10:30:00.000Z",
  "canonicalUrl": "https://platformer.substack.com/p/the-ai-trust-crisis",
  "author": "Casey Newton",
  "publicationName": "Platformer",
  "publicationSubdomain": "platformer",
  "likes": 847,
  "comments": 132,
  "wordCount": 2450,
  "isPaywalled": false,
  "previewText": "The past month has brought a reckoning for AI companies...",
  "coverImage": "https://substackcdn.com/image/fetch/...",
  "tags": ["AI", "trust", "technology"]
}

Comments Output

{
  "body": "This is exactly what I've been seeing in my industry...",
  "author": "John Reader",
  "date": "2026-03-01T14:22:00.000Z",
  "likes": 23,
  "postTitle": "The AI Trust Crisis",
  "publicationSubdomain": "platformer"
}

Publication Info Output

{
  "name": "Platformer",
  "subdomain": "platformer",
  "description": "Tech and democracy coverage",
  "authorName": "Casey Newton",
  "heroImage": "https://substackcdn.com/image/fetch/...",
  "logoUrl": "https://substackcdn.com/image/fetch/...",
  "themeColor": "#FF6719",
  "subscriberCount": 250000,
  "postCount": 1200
}

Integration Examples

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "publications": ["platformer", "thebrowser"],
    "scrapeType": "posts",
    "maxItems": 50,
    "sortBy": "new",
    "includeBodyText": True,
}

run = client.actor("cryptosignals/substack-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{item['title']} — {item.get('likes', 0)} likes")

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const input = {
    publications: ["platformer", "thebrowser"],
    scrapeType: "posts",
    maxItems: 50,
    sortBy: "new",
    includeBodyText: true,
};

const run = await client.actor("cryptosignals/substack-scraper").call(input);

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item => {
    console.log(`${item.title} — ${item.likes || 0} likes`);
});

Using the Apify API Directly

curl -X POST "https://api.apify.com/v2/acts/cryptosignals~substack-scraper/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "publications": ["platformer"],
    "scrapeType": "posts",
    "maxItems": 20
  }'

💰 Pricing

Pay-per-result: $0.005 per item (post, comment, or publication record). You pay for data delivered, not compute time.

• 50 posts from one publication = $0.25
• Full 500-post archive scrape = $2.50
• Weekly monitoring of 10 newsletters (20 posts each) = ~$4/month

Tips for Best Results

1. Start small: Set maxItems to 5–10 for your first run to verify the output format meets your needs.
2. Use publication subdomains: For platformer.substack.com, enter just platformer in the publications list.
3. Enable body text selectively: Full article text significantly increases output size. Only enable it when you need the content for analysis.
4. Combine with Apify integrations: Send results directly to Google Sheets, Slack, Zapier, Make, or webhooks for automated workflows.
5. Schedule regular runs: Set up recurring scrapes to build longitudinal datasets or monitor newsletters over time.

Frequently Asked Questions

Can I scrape paywalled/subscriber-only posts?

The scraper extracts publicly available data. For paywalled posts, you'll get the title, preview text, metadata, and publication info, but not the full subscriber-only content.

How do I find a publication's subdomain?

Look at the newsletter URL. For https://platformer.substack.com, the subdomain is platformer. For custom domains, check the Substack about page.

Can I scrape custom domain Substack newsletters?

Yes. Use the publication's original Substack subdomain (before they switched to a custom domain). You can usually find it referenced on their about page or through a web search.

How often is the data updated?

Every run fetches live data directly from Substack. You always get the latest posts, comments, and metrics.

Is there a rate limit?

The scraper handles rate limiting automatically with built-in delays and retries. You don't need to configure anything.

Can I search for posts about a specific topic?

Yes! Use the searchQuery parameter to search across all of Substack, or combine it with publications to search within specific newsletters.

What export formats are available?

Apify supports JSON, CSV, Excel (XLSX), XML, HTML, and RSS. You can download in any format from the dataset tab after a run completes.

How do I integrate this with my existing workflow?

Use Apify's built-in integrations (Zapier, Make, Google Sheets, webhooks) or call the API directly from any programming language. See the code examples above.

Can I run this on a schedule?

Yes. Apify supports cron-like scheduling. Set up daily, weekly, or custom schedules from the actor's Schedules tab. Each run stores results in a new dataset.

What happens if a publication doesn't exist?

The scraper will log a warning for invalid subdomains and continue processing the remaining publications. Your run won't fail because of one bad input.

⭐ Leave a Review

If this actor saved you time, please take 30 seconds to leave a review: 👉 https://apify.com/cryptosignals/substack-scraper/reviews

Reviews are the primary way this actor reaches new users and gets maintained and updated.