# Substack Scraper — Posts, Comments & Search (`axlymxp/substack-scraper`) Actor

Scrape Substack posts by keyword or from any newsletter's archive: full text (HTML + plain text), author, date, likes, comment count, paywall flag, plus optional full comment threads. Incremental runs by date. No login or browser. Pay only for the rows you get.

- **URL**: https://apify.com/axlymxp/substack-scraper.md
- **Developed by:** [axly](https://apify.com/axlymxp) (community)
- **Categories:** Social media, News, Lead generation
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 dataset items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Posts, Comments & Newsletter Search

Turn Substack into a clean dataset. Search posts by keyword across **all of Substack**, pull the full archive of any newsletter (including custom domains), or fetch specific posts. You get the full text, author, date, likes, comment count, paywall status and, if you want them, **every comment and reply** as structured rows. No login, no browser, no proxy setup.

### Who uses this

- **Media, PR & comms teams** track what newsletters say about a brand, a competitor or a topic, every day.
- **Researchers & analysts** build corpora of long-form writing for discourse analysis, trend tracking and sentiment.
- **AI / RAG builders** feed clean plain-text posts into LLM pipelines, vector stores and summarizers.
- **Creators & growth teams** benchmark rival newsletters: what they publish, how often, and which posts get the most likes and comments.
- **Community & audience researchers** mine comment threads to see what readers actually care about.

### What data you get

One row per post (`rowType: "post"`), plus one row per comment when comments are enabled (`rowType: "comment"`).

**Post rows**

| Field | Description |
|---|---|
| `title`, `subtitle`, `url`, `slug`, `postId` | Post identity |
| `postDate` | Publish time (ISO-8601, UTC) |
| `authorName`, `authorHandle`, `authorId`, `coAuthors` | Byline(s) |
| `likeCount`, `commentCount`, `restackCount` | Engagement |
| `audience`, `isPaywalled` | `everyone`, `only_paid` or `founding` |
| `wordCount` | Full length of the post |
| `bodyText`, `bodyHtml` | Full content as clean plain text and as HTML |
| `contentTruncated` | `true` when only the free preview of a paid post was available |
| `tags`, `sectionName`, `coverImage`, `description`, `language` | Metadata |
| `podcastUrl`, `podcastDurationSec` | For podcast episodes |
| `publicationId`, `publicationName`, `publicationUrl` | The newsletter |
| `source`, `scrapedAt` | Which input produced the row, and when |

**Comment rows**

| Field | Description |
|---|---|
| `commentId`, `postId`, `postTitle`, `url` | Which post the comment belongs to |
| `commentBody`, `commentDate`, `editedAt` | The comment |
| `authorName`, `authorHandle`, `authorId` | Commenter |
| `depth`, `parentCommentId`, `replyCount` | Thread structure: rebuild the tree, or keep top-level comments only |
| `likeCount`, `isDeleted` | Engagement / status |

### Use cases

1. **Daily media monitoring.** Search `"your brand"` with *Published within: Past 24 hours* on a schedule, and push new mentions to Slack or a sheet.
2. **Competitor newsletter benchmarking.** Scrape 10 rival newsletters with *Most popular first* and compare topics, length, likes and comment volume.
3. **Incremental archive sync.** Run once with no date filter to backfill a newsletter, then daily with *Published after* set to yesterday to pick up only new posts.
4. **Reader-voice research.** Enable *Include comments* on a newsletter's top posts to collect thousands of reader opinions for analysis.
5. **LLM knowledge base.** Export `bodyText` for a set of newsletters into your vector store for grounded Q\&A.

### Input

| Parameter | Default | Description |
|---|---|---|
| `mode` | `search` | `search` (keywords), `publications` (newsletter archives), or `posts` (URLs) |
| `searchQueries` | — | Keywords, one per line (search mode) |
| `dateRange` | `any` | `day`, `week`, `month`, `year`: only recent posts (search mode) |
| `publications` | — | Subdomain (`astralcodexten`), `x.substack.com` URL or custom domain (`www.noahpinion.blog`) |
| `sortBy` | `new` | `new` or `top` (most popular first), for archives |
| `postUrls` | — | Specific post URLs (`…/p/<slug>`) |
| `publishedAfter` | — | `YYYY-MM-DD`; skip older posts (archives stop as soon as they reach older posts) |
| `includeContent` | `true` | Add `bodyHtml` + `bodyText` |
| `includeComments` | `false` | Add one row per comment and reply |
| `maxCommentsPerPost` | `100` | Comment cap per post (`0` = all) |
| `commentSort` | `best_first` | `best_first` or `most_recent_first` |
| `maxPostsPerSource` | `50` | Post cap per keyword or newsletter (`0` = no cap) |
| `maxItems` | `200` | Stop after this many rows in total (`0` = no limit) |

#### Example input — keyword search, last week

```json
{
    "mode": "search",
    "searchQueries": ["artificial intelligence", "semiconductors"],
    "dateRange": "week",
    "maxPostsPerSource": 100,
    "maxItems": 500
}
```

#### Example input — two newsletters with comments

```json
{
    "mode": "publications",
    "publications": ["astralcodexten", "https://www.noahpinion.blog"],
    "sortBy": "top",
    "includeComments": true,
    "maxCommentsPerPost": 50,
    "maxPostsPerSource": 20,
    "maxItems": 0
}
```

### Example output

```json
[
  {
    "rowType": "post",
    "postId": 216809249,
    "title": "Open Thread 452",
    "url": "https://www.astralcodexten.com/p/open-thread-452",
    "postDate": "2026-09-21T22:05:59.990Z",
    "postType": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "wordCount": 414,
    "likeCount": 61,
    "commentCount": 390,
    "restackCount": 1,
    "authorName": "Scott Alexander",
    "authorHandle": "astralcodexten",
    "tags": [],
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/cab976a2-…_1536x1024.png",
    "language": "en",
    "bodyText": "This is the weekly visible open thread. Post about anything you want, ask random questions, whatever. …",
    "contentTruncated": false,
    "publicationId": 89120,
    "publicationName": "Astral Codex Ten",
    "publicationUrl": "https://www.astralcodexten.com",
    "source": "url",
    "scrapedAt": "2026-09-23T01:38:30Z"
  },
  {
    "rowType": "comment",
    "commentId": 342672468,
    "postId": 216809249,
    "postTitle": "Open Thread 452",
    "parentCommentId": null,
    "depth": 0,
    "commentDate": "2026-09-21T22:15:53.598Z",
    "commentBody": "Since point 4 implies that normally all coauthors of a finalist review get a free subscription, …",
    "authorName": "Reader Name",
    "authorHandle": "reader-handle",
    "likeCount": 14,
    "replyCount": 2,
    "isDeleted": false,
    "publicationName": "Astral Codex Ten"
  }
]
```

The dataset has two ready-made views: **Posts** and **Comments**.

### Scheduling, webhooks & integrations

- **Schedule** the actor (Console → Schedules) daily with `dateRange: "day"` or a rolling `publishedAfter` date for monitoring.
- **Webhooks** can call your endpoint when a run succeeds; fetch the dataset as JSON, CSV, Excel or XML.
- Connect directly to **Google Sheets, Slack, Zapier, Make, Airbyte or S3** through Apify integrations.

### Use it from AI assistants (MCP)

This actor works as a tool through the [Apify MCP server](https://mcp.apify.com), so Claude, ChatGPT, Cursor and other MCP clients can call it directly:

> "Find Substack posts from the past week about GLP-1 drugs and summarize the main arguments."

The agent calls the actor with `mode: "search"`, `dateRange: "week"` and reads back `bodyText`.

### FAQ

**Do I need a Substack account?** No. The actor reads the same public data Substack shows logged-out visitors.

**Can it get paid-only posts?** It returns their metadata and the free preview that Substack shows publicly. Paid-only rows are flagged with `isPaywalled: true` and `contentTruncated: true`. The actor does not bypass paywalls.

**How far back can archives go?** All the way. Archives are paginated until the first post, unless you set `maxPostsPerSource` or `publishedAfter`.

**Does it work with custom domains?** Yes. Pass the custom domain (e.g. `www.noahpinion.blog`) or the `*.substack.com` subdomain; both resolve to the same newsletter.

**How fresh is the data?** Live. Every run queries Substack at run time.

**How reliable is it?** It uses Substack's own JSON API over plain HTTP, with no headless browser to break. Transient errors are retried automatically, and progress is checkpointed so a restarted run resumes without duplicate rows.

**Is scraping Substack legal?** The actor only collects publicly available information. You are responsible for using the data in line with applicable laws (e.g. GDPR for personal data in comments) and Substack's terms.

**Something broke or you need a field?** Open an issue on the actor's Issues tab.

# Actor input Schema

## `mode` (type: `string`):

Search posts by keyword across all of Substack, scrape the archive of specific newsletters, or fetch specific post URLs.

## `searchQueries` (type: `array`):

Keywords to search posts for across all of Substack (mode: Search). One search per line.

## `dateRange` (type: `string`):

Only return posts published within this window (mode: Search).

## `publications` (type: `array`):

Newsletters to scrape (mode: Newsletter archives). Accepts a subdomain (`astralcodexten`), a `*.substack.com` URL or a custom domain (`www.noahpinion.blog`).

## `sortBy` (type: `string`):

Newest posts first, or the newsletter's most popular posts first (mode: Newsletter archives).

## `postUrls` (type: `array`):

Specific post URLs, e.g. https://www.astralcodexten.com/p/open-thread-452 (mode: Specific post URLs).

## `publishedAfter` (type: `string`):

Skip posts published before this date. Use it for incremental daily/weekly runs: archives stop as soon as older posts are reached.

## `includeContent` (type: `boolean`):

Add the post body as HTML and clean plain text (great for AI/RAG pipelines). Paid-only posts include the free preview only.

## `includeComments` (type: `boolean`):

Also output every comment and reply of each post as its own row (rowType = 'comment'), with thread depth and parent id.

## `maxCommentsPerPost` (type: `integer`):

Cap comment rows per post. 0 = all comments.

## `commentSort` (type: `string`):

Which comments come first when a cap is set.

## `maxPostsPerSource` (type: `integer`):

Cap posts per search keyword or per newsletter. 0 = no cap.

## `maxItems` (type: `integer`):

Stop the run after this many rows (posts + comments). 0 = no limit.

## Actor input object example

```json
{
  "mode": "search",
  "searchQueries": [
    "artificial intelligence"
  ],
  "dateRange": "any",
  "publications": [
    "https://www.astralcodexten.com"
  ],
  "sortBy": "new",
  "includeContent": true,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "commentSort": "best_first",
  "maxPostsPerSource": 50,
  "maxItems": 200
}
```

# Actor output Schema

## `dataset` (type: `string`):

Substack posts (and optionally comments), one row each.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "artificial intelligence"
    ],
    "publications": [
        "https://www.astralcodexten.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("axlymxp/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": ["artificial intelligence"],
    "publications": ["https://www.astralcodexten.com"],
}

# Run the Actor and wait for it to finish
run = client.actor("axlymxp/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "artificial intelligence"
  ],
  "publications": [
    "https://www.astralcodexten.com"
  ]
}' |
apify call axlymxp/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,axlymxp/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fz8MZ7VlLm3VYSqzQ/builds/FnCebwFxqKApkfF1p/openapi.json