# Substack Newsletter Scraper (`straightforward_hydra/substack-newsletter-scraper`) Actor

Scrape any Substack publication: posts with reactions, comment counts, wordcount and paid/free status, plus full post text, comment threads and newsletter recommendation graphs. No API key, no login.

- **URL**: https://apify.com/straightforward\_hydra/substack-newsletter-scraper.md
- **Developed by:** [Dev D](https://apify.com/straightforward_hydra) (community)
- **Categories:** Social media, News, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Newsletter Scraper

Scrape any Substack publication — posts, full text, comment threads and
newsletter recommendation graphs. **No API key, no login, no phone number, no
proxy required.**

Point it at a handle (`astralcodexten`), a subdomain
(`bigtechnology.substack.com`), a custom domain
(`newsletter.pragmaticengineer.com`) or any post URL. All four work.

### Modes

#### `posts` — one row per post

Title, subtitle, publish date, author, section, tags, wordcount, likes, comment
count, restacks, paid/free status, cover image, podcast duration and canonical
URL.

Turn on **Include full post text** to also get `body_html` and clean
`content_text`. Free posts return in full; paid posts return the free preview
Substack serves to logged-out visitors.

#### `comments` — one row per comment

The whole reply tree, flattened, with `depth` and `parent_comment_id` so you can
rebuild the thread. Includes author name, handle, date, body and reaction count,
stamped with the post it belongs to.

#### `recommendations` — discover newsletters in a niche

The publications each seed recommends, with name, description, author, subdomain
and URL. Feed those subdomains back in to walk the graph outward — a practical
way to build a list of newsletters in one topic for sponsorship or competitor
research.

### Useful options

| Option | What it does |
| --- | --- |
| **Search within publication** | Substack's own search, e.g. only posts mentioning `openai`. |
| **Sort** | Newest first, or the publication's most popular posts. |
| **Only posts after** | ISO date. Stops at the first older post, so a daily run collects only what is new. |
| **Max posts per publication** | Archive rows per publication, 50 per request. |
| **Max comments per post** | Caps comments kept per post, replies included. |

### Example input

```json
{
  "publications": ["astralcodexten", "bigtechnology"],
  "mode": "posts",
  "maxPostsPerPublication": 100,
  "sort": "new",
  "includeContent": false
}
```

### Example output (`posts`)

```json
{
  "publication": "astralcodexten.substack.com",
  "publication_id": 89120,
  "post_id": 209742524,
  "slug": "open-thread-446",
  "title": "Open Thread 446",
  "post_date": "2026-08-10T16:17:33.848Z",
  "type": "newsletter",
  "audience": "everyone",
  "is_paid": false,
  "author": "Scott Alexander",
  "wordcount": 516,
  "reaction_count": 74,
  "comment_count": 729,
  "url": "https://www.astralcodexten.com/p/open-thread-446"
}
```

### Notes

- Only publicly visible content is read. Nothing paywalled is bypassed and no
  account is used.
- Some newsletters keep their domain after leaving Substack. Those are reported
  with a clear message and the run continues with the rest.
- Runs stop cleanly before the platform timeout and keep everything collected so
  far, rather than failing.

### Pricing

Pay per result:

| Event | What it is |
| --- | --- |
| `post` | One post from the archive. |
| `post-content` | One post including its full text (one extra request). |
| `comment` | One comment. |
| `recommendation` | One recommended publication. |

### Support

Found a field that stopped populating, or a publication that won't load? Open an
issue on the Actor's page and include the input you used.

# Actor input Schema

## `publications` (type: `array`):

Substack publications. Accepts any format: astralcodexten, @astralcodexten, astralcodexten.substack.com, a custom domain like newsletter.pragmaticengineer.com, or a full post URL.

## `mode` (type: `string`):

posts = one row per post. comments = one row per comment on those posts. recommendations = the publications each seed recommends, for discovering newsletters in a niche.

## `maxPostsPerPublication` (type: `integer`):

How many posts to read from each publication's archive, 50 per request. In comments mode this is how many posts get their comments collected.

## `sort` (type: `string`):

new = newest posts first. top = the publication's most popular posts first.

## `search` (type: `string`):

Optional keyword. Only posts matching this term are returned, using Substack's own search.

## `minPostDate` (type: `string`):

Optional ISO date, e.g. 2026-01-01. With sort = new and no search term, the run stops at the first older post, so daily runs only collect what is new. With sort = top or a search term the results are not in date order, so older posts are filtered out instead.

## `includeContent` (type: `boolean`):

Fetch each post's body as HTML and plain text. Costs one extra request per post, so large runs take longer. Paid posts return only the free preview Substack shows logged-out visitors.

## `maxCommentsPerPost` (type: `integer`):

Comments mode only. Caps how many comments are kept per post, replies included.

## `proxyConfiguration` (type: `object`):

Optional. Substack is reachable without a proxy; enable this for very large runs.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten",
    "bigtechnology"
  ],
  "mode": "posts",
  "maxPostsPerPublication": 50,
  "sort": "new",
  "includeContent": false,
  "maxCommentsPerPost": 200,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten",
        "bigtechnology"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("straightforward_hydra/substack-newsletter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "astralcodexten",
        "bigtechnology",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("straightforward_hydra/substack-newsletter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten",
    "bigtechnology"
  ]
}' |
apify call straightforward_hydra/substack-newsletter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,straightforward_hydra/substack-newsletter-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/drKn7gP02DntpvIqa/builds/hdKVwb7id9fZ6E8gg/openapi.json