# Substack Post Scraper (`romy/substack-post-scraper`) Actor

Substack Post Scraper extracts full post content and comments from any Substack newsletter. Give it a newsletter subdomain and it returns complete article HTML, metadata, and reader comments — no login required for free posts

- **URL**: https://apify.com/romy/substack-post-scraper.md
- **Developed by:** [Romy](https://apify.com/romy) (community)
- **Categories:** News, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $30.00 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### What does Substack Post Scraper do?

**Substack Post Scraper** extracts full post content and comments from any [Substack](https://substack.com) newsletter. Give it a newsletter subdomain and it returns complete article HTML, metadata, and reader comments — no login required for free posts. Runs on the Apify platform with scheduling, proxy rotation, and API access built in.

### Why use Substack Post Scraper?

- **Content intelligence**: Read full article text to analyze topics, tone, and writing style
- **NLP & AI pipelines**: Feed article HTML into summarizers, classifiers, or embedding models
- **Competitive research**: Monitor what top newsletters are publishing and how readers respond
- **Comment analysis**: Scrape reader feedback and sentiment for market research
- **Content archiving**: Build a searchable archive of newsletter content over time

### How to use Substack Post Scraper

1. Open the actor on Apify and click **Try for free**
2. Enter the **Newsletter Subdomain** (e.g. `howtogrowanewsletter` from `howtogrowanewsletter.substack.com`)
3. Set **Max Posts** and choose **Audience Filter** (free posts only, or all posts)
4. Toggle **Include Comments** on or off
5. Click **Start** — results appear in the **Output** tab within seconds

### Input

```json
{
  "subdomain": "howtogrowanewsletter",
  "maxPosts": 20,
  "audienceFilter": "everyone",
  "includeComments": true
}
```

Also accepts full URLs:

```json
{
  "subdomain": "https://howtogrowanewsletter.substack.com",
  "maxPosts": 50
}
```

### Output

Each item in the dataset is one post with full content and comments:

```json
{
  "id": 208676609,
  "title": "Substack Just Launched AI Detection. Here's What It Means For You.",
  "subtitle": "A new feature lets readers scan any post to see how much was written by a human.",
  "slug": "substack-just-launched-ai-detection",
  "canonical_url": "https://howtogrowanewsletter.substack.com/p/substack-just-launched-ai-detection",
  "post_date": "2026-07-27T13:08:45.764Z",
  "audience": "everyone",
  "wordcount": 542,
  "reaction_count": 2,
  "reactions": {"❤": 2},
  "comment_count": 2,
  "restacks": 1,
  "cover_image": "https://substack-post-media.s3.amazonaws.com/...",
  "body_html": "<p>This is the first issue of a new free series...</p>...",
  "publishedBylines": [
    {
      "name": "How to Grow a Newsletter",
      "handle": "howtogrowanewsletter",
      "bio": "I run four Substack newsletters pulling in nearly $250K/year"
    }
  ],
  "postTags": [],
  "comments": [
    {
      "id": 302090035,
      "name": "Phaetrix",
      "body": "AI detection may give readers additional information...",
      "date": "2026-07-27T16:44:04.418Z",
      "reactions": {"❤": 2},
      "reaction_count": 2,
      "children": []
    }
  ]
}
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

### Data table

| Field | Description |
|-------|-------------|
| `title` | Post title |
| `subtitle` | Post subtitle / description |
| `canonical_url` | Full URL to the post |
| `post_date` | Publication date (ISO 8601) |
| `audience` | `everyone` = free, `only_paid` = paywalled |
| `wordcount` | Approximate word count |
| `reaction_count` | Total reactions (likes/hearts) |
| `restacks` | Number of restacks (reposts) |
| `body_html` | Full article HTML (free posts only) |
| `cover_image` | Cover image URL |
| `publishedBylines` | Author info: name, handle, bio, photo |
| `postTags` | Post tags/topics |
| `comments[].body` | Comment text |
| `comments[].name` | Commenter name |
| `comments[].reactions` | Comment reactions |
| `comments[].children` | Nested replies |

### Audience filter

| Value | Behavior |
|-------|----------|
| `everyone` | Free posts only — `body_html` contains the full article |
| `all` | All posts including paid — paywalled posts have empty `body_html` |

### Pricing

- **$0.05** per run start
- **$0.03** per post scraped

Scraping 100 free posts costs approximately **$3.05**. The free Apify tier gives you enough credits to test before subscribing.

### Tips

- Set `audienceFilter: "everyone"` to only get posts with full content — saves credits by skipping paywalled posts
- Set `includeComments: false` if you only need article text — faster and cheaper
- Newsletters with custom domains (e.g. `citrini` → `citriniresearch.com`) are supported automatically
- Combine with **Substack Newsletter Scraper** to first discover newsletters, then scrape their posts

### FAQ & Disclaimers

**Is scraping Substack legal?** This actor only accesses publicly available content through Substack's official mobile API — the same data visible to anyone without logging in. Paywalled content is never accessed.

**Why is `body_html` empty for some posts?** Posts with `audience: "only_paid"` are behind a paywall — Substack does not serve the article body without authentication.

**Supports custom domains?** Yes — newsletters that have migrated to a custom domain are followed automatically.

**Support:** Open an issue in the Issues tab on this actor's page. Custom solutions available on request.

# Actor input Schema

## `subdomain` (type: `string`):

Substack subdomain (e.g. 'howtogrowanewsletter' from howtogrowanewsletter.substack.com). Also accepts full URL.

## `maxPosts` (type: `integer`):

Maximum number of posts to scrape.

## `audienceFilter` (type: `string`):

Filter posts by audience. 'everyone' = free posts only (includes full body\_html). 'all' = include paid posts (body will be empty for paywalled content).

## `includeComments` (type: `boolean`):

Fetch comments for each post.

## `proxyConfiguration` (type: `object`):

Proxy settings.

## Actor input object example

```json
{
  "subdomain": "howtogrowanewsletter",
  "maxPosts": 20,
  "audienceFilter": "everyone",
  "includeComments": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "subdomain": "howtogrowanewsletter"
};

// Run the Actor and wait for it to finish
const run = await client.actor("romy/substack-post-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "subdomain": "howtogrowanewsletter" }

# Run the Actor and wait for it to finish
run = client.actor("romy/substack-post-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "subdomain": "howtogrowanewsletter"
}' |
apify call romy/substack-post-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,romy/substack-post-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/AeXWsAWplfsi1LHug/builds/ZXDUQNLdTVWutFZnh/openapi.json