# Substack Scraper — Posts, Authors & Comments (Bulk) (`hipersoft/substack-scraper`) Actor

Scrape posts from any Substack publication in bulk: title, subtitle, author, URL, publish date, likes, comment count, excerpt, full body text and paid/free status. Optionally pull top comments per post. Clean structured JSON, CSV or Excel.

- **URL**: https://apify.com/hipersoft/substack-scraper.md
- **Developed by:** [hiper soft](https://apify.com/hipersoft) (community)
- **Categories:** Social media, News
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.0005 / post scraped

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Scraper — Posts, Authors & Comments (Bulk)

Scrape posts from **any Substack publication** as clean, structured **JSON, CSV or Excel**: **title, subtitle, author, URL, publish date, likes, comment count, excerpt, full body text** and whether the post is **paid or free**. Point it at one publication or dozens, cap how many posts you want per publication, and optionally pull the **top comments** on each post — all in one run, with no account to manage.

### What does Substack Scraper do?

This Substack scraper turns any publication's archive into a structured dataset. Give it a handle (like `bigtechnology`), a Substack URL, or a publication's custom domain, and it walks the archive newest-first and returns one flat record per post — with the headline metadata, the full article text, and the paid/free flag. Turn on comments to also capture the top community replies under each post. Use it to build newsletter datasets, monitor writers and topics, feed research pipelines, or archive your own publication.

### What data can you scrape from Substack?

| Data point | Description |
| --- | --- |
| `publication` | The publication handle or domain the post belongs to. |
| `postId` | Numeric Substack post id. |
| `slug` | URL slug of the post. |
| `title` | Post title. |
| `subtitle` | Post subtitle / deck. |
| `author` | Author name(s) / byline. |
| `url` | Canonical URL of the post. |
| `publishedAt` | Publish date/time (ISO 8601). |
| `type` | Post type (newsletter, podcast, thread, etc.). |
| `likes` | Number of reactions/likes. |
| `commentCount` | Number of comments on the post. |
| `wordCount` | Approximate word count of the article. |
| `excerpt` | Short preview/teaser text. |
| `bodyText` | Full article body as plain text. |
| `isPaid` | Whether the post is subscriber-only. |
| `coverImage` | Cover image URL, if any. |
| `comments` | Top comments (author, body, date, likes) when enabled. |

### Use cases

- **Newsletter datasets** — collect a publication's back catalogue for analysis or search.
- **Writer & topic monitoring** — track new posts from the publications you care about on a schedule.
- **Research pipelines** — feed structured post text into BI tools, notebooks, models or RAG apps.
- **Competitive intelligence** — see cadence, engagement (likes/comments) and paid vs free mix across publications.
- **Personal archive** — export your own Substack posts and comments to JSON, CSV or Excel.

### How to scrape Substack data

1. Add **Substack Scraper** to your Apify account and open its input.
2. Enter one or more **publications** — a handle (`bigtechnology`), a Substack URL (`https://bigtechnology.substack.com`) or a custom domain (`https://www.astralcodexten.com`).
3. Set **maxPosts** per publication (or `0` for the entire archive).
4. Optionally enable **includeComments** and set **maxComments**.
5. Click **Run**, then export the results as JSON, CSV or Excel — or fetch them through the Apify API.

### Input

```json
{
  "publications": ["bigtechnology", "https://www.astralcodexten.com"],
  "maxPosts": 50,
  "includeComments": false,
  "maxComments": 20
}
```

| Field | Description |
| --- | --- |
| `publications` | Publications to scrape — handles, Substack URLs or custom domains. |
| `maxPosts` | Max posts per publication, newest first (`0` = entire archive). |
| `includeComments` | Also fetch the top comments for each post. |
| `maxComments` | Max top-level comments to keep per post when comments are on. |

### What you get

One flat JSON record per post, ready to export to CSV, Excel, JSON or XML:

```json
{
  "publication": "bigtechnology",
  "postId": 209279848,
  "slug": "when-artificial-intelligence-is-too",
  "title": "When Artificial Intelligence Is Too Valuable To Sell",
  "subtitle": "Just because the AI labs have sold models on a meter until now doesn't mean they'll always do so.",
  "author": "Alex Kantrowitz",
  "url": "https://www.bigtechnology.com/p/when-artificial-intelligence-is-too",
  "publishedAt": "2026-07-31T19:30:17.712Z",
  "type": "newsletter",
  "likes": 83,
  "commentCount": 0,
  "wordCount": 1139,
  "excerpt": "It's time to delete the assumption that the frontier AI labs will always license their best models…",
  "bodyText": "…full article text…",
  "isPaid": true,
  "coverImage": "https://substackcdn.com/image/…",
  "comments": []
}
```

### Related Actors

Building a broader newsletter, news or research dataset? Pair this with:

- [Google News Scraper](https://apify.com/hipersoft/google-news-scraper) — headlines by keyword and topic.
- [Hacker News Scraper](https://apify.com/hipersoft/hacker-news-scraper) — stories, comments and users from Hacker News.
- [Reddit Scraper](https://apify.com/hipersoft/reddit-scraper) — posts, comments and users, no login.
- [Website Content Crawler](https://apify.com/hipersoft/website-content-crawler) — crawl any site into clean text for LLM/RAG.

### FAQ

**Do I need a Substack account or login?**
No. Just add the publications you want and run — no account, login or setup required.

**How many posts can I scrape per run?**
There's no fixed cap. `maxPosts` limits how many posts are collected per publication (`0` = the whole archive), and `maxComments` limits comments per post. Larger runs simply take longer.

**Can I scrape comments?**
Yes. Turn on `includeComments` and the Actor returns the top publicly visible comments — author, body, date and likes — nested under each post.

**Does it work with custom domains?**
Yes. Many publications use their own domain (e.g. `www.astralcodexten.com`). Enter the handle or the custom domain and the Actor resolves it automatically.

**Is scraping Substack legal?**
The Actor collects only publicly available data. You are responsible for using it in line with Substack's terms and applicable laws — avoid collecting personal data you don't have a lawful basis to process.

**What's the output format?**
A structured JSON dataset with one item per post (comments nest inside their post). Export as JSON, CSV or Excel from the Apify Console or API.

**Can I use this with n8n, Make or Zapier?**
Yes. Run it from [n8n](https://n8n.io) via the Apify node, or connect it to [Make](https://apify.com/integrations/make), [Zapier](https://apify.com/integrations/zapier), [Slack](https://docs.apify.com/platform/integrations/slack), [Google Drive](https://docs.apify.com/platform/integrations/drive) and [many more](https://apify.com/integrations), plus the [Apify API](https://docs.apify.com/api/v2), JavaScript/Python clients, MCP and [webhooks](https://docs.apify.com/platform/integrations/webhooks).

### Notes

Original clean-room implementation. Returns only public data; you are responsible for compliance with Substack's terms. Not affiliated with Substack.

# Actor input Schema

## `publications` (type: `array`):

One or more Substack publications to scrape. Accepts a bare handle (e.g. "bigtechnology"), a Substack URL (e.g. "https://bigtechnology.substack.com") or a publication's custom domain (e.g. "https://www.astralcodexten.com"). Custom domains are followed automatically.

## `maxPosts` (type: `integer`):

Maximum number of posts to scrape from each publication, newest first. Use 0 for no limit (scrape the entire archive).

## `includeComments` (type: `boolean`):

Also fetch the top comments for each post (author, body, date and likes). Adds one extra request per post and only returns publicly visible comments.

## `maxComments` (type: `integer`):

When "Include comments" is on, the maximum number of top-level comments to keep per post (sorted best first).

## Actor input object example

```json
{
  "publications": [
    "bigtechnology",
    "https://www.astralcodexten.com"
  ],
  "maxPosts": 50,
  "includeComments": false,
  "maxComments": 20
}
```

# Actor output Schema

## `results` (type: `string`):

The results as dataset items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "bigtechnology"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hipersoft/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": ["bigtechnology"] }

# Run the Actor and wait for it to finish
run = client.actor("hipersoft/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "bigtechnology"
  ]
}' |
apify call hipersoft/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hipersoft/substack-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jWZ0wRZnQv3Qj6PLF/builds/NeUaF1SA7V9nLXvZW/openapi.json