# Substack Scraper – Posts, Comments & Newsletter Stats (`phantom-x64/substack-scraper`) Actor

Extract Substack newsletter posts (full text or Markdown), comments, publication profiles with subscriber counts and prices, authors, Substack-wide search and category leaderboards. No login. Pay per result: posts from $0.70 per 1,000, no start fee.

- **URL**: https://apify.com/phantom-x64/substack-scraper.md
- **Developed by:** [Phantom Data](https://apify.com/phantom-x64) (community)
- **Categories:** Social media, News, Lead generation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — posts, comments, newsletters, authors & leaderboards

**Substack scraper and Substack API alternative.** Export posts with full text and **Markdown** (ready for AI / RAG), public **comments**, **newsletter profiles** with subscriber counts and subscription prices, **author profiles** with social links, **Substack-wide search** and **category leaderboards** (Bestsellers and Rising) — all in one actor, **no login, no cookies, no browser**. **Pay only per result, no start fee.**

- ✅ Any newsletter: `astralcodexten`, `https://lenny.substack.com`, custom domains like `https://www.lennysnewsletter.com`
- ✅ Full post content as clean **text**, **Markdown** and/or **HTML**, plus likes, comments count, restacks, word count, tags, authors
- ✅ Public **comments and replies** with author, date, likes and thread depth
- ✅ **Newsletter data**: subscriber count ("Over 1,200,000 subscribers" → `1200000`), paid plans and prices, bestseller tier, podcast, sections, contributors
- ✅ **Leaderboards** for 30+ categories (Technology, Business, Finance, Culture, Politics, Science, Crypto...): Bestsellers (paid) and Rising (trending)
- ✅ **Search** all of Substack: posts, or people & newsletters (creator discovery, lead lists)
- ✅ **Monitor mode**: scheduled runs return only new posts — you never pay twice for the same post
- ✅ Filters: date range (`"7 days"`, `2026-01-01`), free vs paywalled, keyword search inside a newsletter, newest or most popular first
- ✅ **From $0.30 per 1,000 comments and $0.70 per 1,000 posts. No start fee. Error rows are free.**

### What can you do with this Substack scraper?

| Use case | How |
|---|---|
| **Newsletter market research** | Export the Bestsellers and Rising leaderboards of any category with subscriber counts, prices and authors. Spot fast-growing newsletters before everybody else. |
| **Competitor & content analysis** | Pull the full archive of competing newsletters: topics, posting frequency, word count, likes, comments and restacks per post. Find what performs best. |
| **Creator discovery & influencer marketing** | Search "people & newsletters" for any topic and get writers with subscriber/follower counts, bios and links (X/Twitter, LinkedIn, website, YouTube). |
| **Sponsorship & partnership sales** | Build lead lists of newsletters by category and size, with pricing and contributor names. |
| **AI / LLM / RAG pipelines** | Get clean Markdown or plain text of posts to feed embeddings, summarisation, or fine-tuning datasets. |
| **Audience & sentiment research** | Export public comments with likes and reply threads to measure what readers care about. |
| **Monitoring & alerts** | Schedule hourly/daily runs in monitor mode and send new posts to Slack, Discord, Google Sheets or a webhook. |
| **Academic & media research** | Archive what public newsletters publish, with timestamps and authors. |

### Quick start (under 1 minute)

1. Click **Try for free** and paste one or more newsletter URLs (`https://www.astralcodexten.com`, `lennysnewsletter.com`).
2. Choose *Posts*, *Search* or *Leaderboards*, set the max posts, and decide whether you want full text/Markdown and comments.
3. Click **Start**. The pre-filled input (10 posts with full text) finishes in about 10 seconds. Download JSON, CSV or Excel, or schedule it as a monitor.

### Input

| Field | What it does |
|---|---|
| `publications` | Newsletters to scrape (subdomain, Substack URL or custom domain). Post and profile URLs pasted here are detected automatically. |
| `postUrls` | Individual posts (`/p/slug`, `open.substack.com/pub/...`, `substack.com/home/post/p-123`). |
| `authors` | Profiles (`lenny`, `@lenny`, `https://substack.com/@lenny`). |
| `searchQueries` + `searchType` | Search Substack for `posts`, `people` (writers & newsletters) or `both`; `maxSearchResults` per query. |
| `leaderboardCategories` + `leaderboardRanking` | Category names/slugs (`technology`, `finance`, `us-politics`...); ranking `paid` (Bestsellers), `trending` (Rising), `both`, or `all` (category directory); `maxLeaderboardEntries`. |
| `maxPostsPerPublication` | Posts per newsletter (default 10; `0` = only the newsletter profile). |
| `postSort` | `new` (newest first) or `top` (most popular first). |
| `archiveSearch` | Keyword search inside the given newsletters. |
| `dateFrom` / `dateTo` | `2026-01-01`, ISO date-time, or relative (`7 days`, `6 months`, `1 year`). |
| `audience` | `all`, `free` or `paid` (paywalled) posts. |
| `includeContent` / `contentFormats` | Full body as `text`, `markdown` and/or `html` (default text + Markdown). Off = cheaper metadata-only rows. |
| `includeComments` / `maxCommentsPerPost` / `commentSort` | Public comments with replies (`best_first`, `most_recent_first`, `oldest_first`). |
| `includePublicationInfo` | One profile row per newsletter (default on). |
| `onlyNewPosts` / `monitorStateKey` | Monitor mode: only posts newer than the last run. |
| `proxyConfiguration` | Apify Proxy (datacenter is usually enough). |

#### Example: full archive of a newsletter, as Markdown, with comments

```json
{
  "publications": ["https://www.lennysnewsletter.com"],
  "maxPostsPerPublication": 1000,
  "contentFormats": ["markdown"],
  "includeComments": true,
  "maxCommentsPerPost": 200
}
```

#### Example: Substack leaderboards (Bestsellers + Rising) for three categories

```json
{
  "leaderboardCategories": ["technology", "finance", "business"],
  "leaderboardRanking": "both",
  "maxLeaderboardEntries": 100,
  "maxPostsPerPublication": 0
}
```

#### Example: find newsletters and writers about a topic

```json
{
  "searchQueries": ["climate tech", "product management"],
  "searchType": "both",
  "maxSearchResults": 50,
  "includeContent": false
}
```

#### Example: daily monitor of new posts (no double charges)

```json
{
  "publications": ["astralcodexten", "https://www.noahpinion.blog", "lenny"],
  "onlyNewPosts": true,
  "monitorStateKey": "my-daily-newsletters",
  "includePublicationInfo": false,
  "maxPostsPerPublication": 50
}
```

Schedule it in **Apify Console → Schedules** and connect an integration (Slack, Google Sheets, webhook...). Runs with nothing new cost **$0**. Keep `includePublicationInfo` on if you also want a daily subscriber-count snapshot (one `publication` row per newsletter per run).

### Output

Every row has a `type`: `post`, `comment`, `publication`, `leaderboard_entry`, `author` or `error` (errors are free). The dataset has ready-made views: **Posts**, **Publications & leaderboards**, **Comments**, **Authors**. Real sample (shortened; full sample in `sample-output.json`):

```json
{
  "type": "post",
  "post_id": 221215977,
  "url": "https://www.astralcodexten.com/p/why-global-trepanation",
  "title": "Why Global Trepanation?",
  "post_date": "2026-10-21T21:53:11.440Z",
  "audience": "everyone",
  "is_paywalled": false,
  "publication_name": "Astral Codex Ten",
  "publication_url": "https://www.astralcodexten.com",
  "authors": [{"name": "Scott Alexander", "handle": "astralcodexten", "profile_url": "https://substack.com/@astralcodexten"}],
  "wordcount": 1307,
  "reaction_count": 479,
  "restacks": 22,
  "comment_count": 223,
  "tags": [],
  "cover_image": "https://substack-post-media.s3.amazonaws.com/public/images/...jpeg",
  "content_is_preview": false,
  "body_text": "A friend asked me to investigate the following mystery: why do 5-10% of skulls...",
  "body_markdown": "A friend asked me to investigate the following mystery: why do [5-10% of skulls](https://pmc.ncbi.nlm.nih.gov/...)...",
  "source": "publication"
}
```

```json
{
  "type": "leaderboard_entry",
  "rank": 1,
  "ranking": "bestseller",
  "category_name": "Technology",
  "name": "SemiAnalysis",
  "url": "https://newsletter.semianalysis.com",
  "author_name": "Dylan Patel",
  "subscriber_count": 326000,
  "subscriber_count_text": "Over 326,000 subscribers",
  "paid_subscriber_range": "Tens of thousands of paid subscribers",
  "monthly_price": 50.0,
  "yearly_price": 500.0,
  "currency": "USD"
}
```

```json
{
  "type": "author",
  "name": "Nathan Lambert",
  "handle": "natolambert",
  "subscriber_count": 87000,
  "follower_count": 130209,
  "primary_publication_name": "Interconnects AI",
  "primary_publication_url": "https://www.interconnects.ai",
  "twitter_handle": "natolambert",
  "links": [{"type": "linkedin", "label": "LinkedIn", "url": "https://www.linkedin.com/in/natolambert/"}],
  "leaderboard": {"ranking": "paid", "rank": 40, "label": "Technology"}
}
```

```json
{
  "type": "comment",
  "comment_id": 366969428,
  "post_url": "https://www.astralcodexten.com/p/why-global-trepanation",
  "parent_comment_id": null,
  "depth": 0,
  "body": "Yet another example of people trying to be holeier-than-thou",
  "date": "2026-10-21T21:59:19.244Z",
  "author_name": "The Shallow Diver",
  "author_handle": "shallowdiver",
  "reaction_count": 108,
  "reply_count": 6
}
```

**Post fields:** `post_id`, `url`, `slug`, `title`, `subtitle`, `description`, `post_date`, `updated_at`, `post_type` (newsletter, podcast, thread, video), `audience`, `is_paywalled`, `publication_id`, `publication_name`, `publication_url`, `authors`, `wordcount`, `reaction_count`, `reactions`, `restacks`, `comment_count`, `child_comment_count`, `tags`, `section_name`, `language`, `cover_image`, `podcast_url`, `podcast_duration`, `preview_text`, `content_is_preview`, `body_text`, `body_markdown`, `body_html`, `source`, `search_query`, `scraped_at`.

**Publication / leaderboard fields:** `publication_id`, `name`, `url`, `subdomain`, `custom_domain`, `description`, `about_text`, `language`, `logo_url`, `created_at`, `first_post_date`, `author_*` (id, name, handle, profile URL, bio, photo), `twitter_handle`, `subscriber_count`, `subscriber_count_text`, `subscriber_range`, `paid_subscriber_range`, `paid_subscribers_order_of_magnitude`, `bestseller_tier`, `paid_subscriptions_enabled`, `monthly_price`, `yearly_price`, `founding_price`, `currency`, `has_podcast`, `community_chat_enabled`, `sections`, `contributors`; leaderboard rows add `rank`, `ranking`, `ranking_title`, `category_id`, `category_name`, `category_slug`.

### Pricing

Pay per event — you only pay for rows you get:

| Event | Price | Per 1,000 |
|---|---|---|
| Post (metadata only) | $0.0007 | $0.70 |
| Post with full content | $0.0014 | $1.40 |
| Comment | $0.0003 | $0.30 |
| Publication profile | $0.0015 | $1.50 |
| Leaderboard entry | $0.0015 | $1.50 |
| Author profile | $0.0015 | $1.50 |
| Run start | **free** | — |
| Error rows | free | — |

Examples: the 100 Bestsellers of a category ≈ **$0.15**. 1,000 posts with full text ≈ **$1.40**. A daily monitor of 5 newsletters costs cents per month, and runs with nothing new cost **$0**. Set **Max cost per run** in the run options to cap any run: the scraper stops cleanly when the limit is reached.

### FAQ

**Do I need a Substack account, cookies or an API key?** No. The actor reads the public JSON endpoints that the Substack website itself uses, plus public newsletter pages.

**Can it get paywalled (paid-only) posts?** Only the free preview that Substack shows to every visitor (marked `content_is_preview: true`, `is_paywalled: true`). It never logs in or bypasses the paywall. Comments on paid-only posts are not public and are skipped.

**How accurate are subscriber counts?** They are the public figures Substack displays ("Over 1,200,000 subscribers", "87K+"), converted to numbers. Some newsletters hide them (`null`).

**Which leaderboard categories exist?** All main Substack categories: culture, technology, business, us-politics, finance, food, sports, art, world-politics, health-politics, news, fashionandbeauty, music, faith, climate, science, literature, fiction, health, design, travel, parenting, philosophy, comics, international, crypto, history, humor, education, film-and-tv, home-garden, games. Substack does not publish rankings for subcategories (you get a free error row suggesting the parent category).

**How far back can it go?** The whole public archive. Raise `maxPostsPerPublication`, or use `dateFrom` / `dateTo`.

**Does it work with custom domains?** Yes (`www.lennysnewsletter.com`, `www.noahpinion.blog`, ...). Substack redirects are followed automatically.

**What about Substack Notes and chat?** Not included in this version (Notes require a separate feed); posts, comments, profiles, search and leaderboards are covered.

**Is it legal?** It only reads information that writers chose to publish openly on the web, without logging in or bypassing any protection. You are responsible for how you use the data (e.g. GDPR when processing personal data such as commenter names).

**Is this the official Substack API?** No. Substack has no official public API; this is an independent tool that works as a simple, no-code Substack API alternative.

### Limits

- Paywalled posts return only the free preview Substack shows to every visitor (`content_is_preview: true`). The Actor never logs in.
- Comments on paid-only posts are not public and are skipped.
- Subscriber counts are the rounded public figures Substack shows ("Over 1,200,000"); some newsletters hide them.
- Substack Notes and chat are not included. Leaderboards exist only for the main categories, not subcategories.

### Use it from code (API)

Every Apify Actor is also an API. Replace `<YOUR_API_TOKEN>` with the token from your Apify account (Settings, API & Integrations).

**JavaScript**

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });
const run = await client.actor('phantom-x64/substack-scraper').call({
    "publications": [
      "https://www.astralcodexten.com"
    ],
    "maxPostsPerPublication": 10
  });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient('<YOUR_API_TOKEN>')
run = client.actor('phantom-x64/substack-scraper').call(run_input={
      "publications": [
        "https://www.astralcodexten.com"
      ],
      "maxPostsPerPublication": 10
    })
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)
```

You can also start it from Make, Zapier, n8n or Google Sheets with the Apify integrations, schedule it, or call it as a tool from an AI agent through the Apify MCP server.

### Support and feedback

Found a bug, need a new field or another source? Open an issue in the **Issues** tab of this Actor page. Reports with a run link are fixed fastest. If it saves you time, a short review on the Store page helps other people find it.

# Actor input Schema

## `publications` (type: `array`):

Substack newsletters to scrape: subdomain (astralcodexten), Substack URL (https://lenny.substack.com) or custom domain (https://www.lennysnewsletter.com). You can also paste post URLs (…/p/slug) or profile URLs (substack.com/@handle) here; they are detected automatically.

## `postUrls` (type: `array`):

Individual posts: https://<publication>/p/<slug>, https://open.substack.com/pub/<sub>/p/<slug> or https://substack.com/home/post/p-<id>.

## `authors` (type: `array`):

Substack profiles to read: handle (lenny, @lenny) or URL (https://substack.com/@lenny). Returns subscriber and follower counts, bio, links (X/Twitter, LinkedIn, website...), publications and leaderboard rank.

## `searchQueries` (type: `array`):

Search all of Substack, like the search box on substack.com. Returns matching posts (semantic search) and/or people & newsletters, depending on "Search for".

## `searchType` (type: `string`):

"Posts" returns matching articles from every publication. "People & newsletters" returns writer profiles with their newsletter and subscriber count (great for creator discovery and lead lists). "Both" returns both.

## `maxSearchResults` (type: `integer`):

Maximum posts (and, separately, people) saved per search query.

## `leaderboardCategories` (type: `array`):

Substack leaderboard categories to export, by name, slug or URL: technology, business, finance, culture, us-politics, news, science, health, crypto, sports, food, music, history, fiction... (https://substack.com/leaderboard/technology/paid also works).

## `leaderboardRanking` (type: `string`):

"Bestsellers" = top paid newsletters, "Rising" = fastest-growing (trending) newsletters, "Top" = the category directory listing.

## `maxLeaderboardEntries` (type: `integer`):

How many ranked publications to save per category and ranking (Substack pages them 25 at a time).

## `maxPostsPerPublication` (type: `integer`):

Newest (or top) posts to save per publication after filters. Set 0 to get only the publication profile.

## `postSort` (type: `string`):

"Newest first" walks the archive by date. "Top (most popular) first" uses Substack's own popularity ranking of the archive.

## `archiveSearch` (type: `string`):

Optional keyword: only posts of the given publications that match it (Substack's archive search).

## `dateFrom` (type: `string`):

Only posts published on or after this date. YYYY-MM-DD, an ISO date-time, or relative: "7 days", "2 weeks", "6 months", "1 year". Leave empty for no limit.

## `dateTo` (type: `string`):

Only posts published on or before this date (a plain YYYY-MM-DD includes the whole day). Same formats as the oldest date.

## `audience` (type: `string`):

Keep all posts, only free (public) posts, or only paywalled posts. For paywalled posts only the free preview that Substack shows to everyone is returned.

## `includeContent` (type: `boolean`):

Fetch the post body (one extra request per post). Off = metadata only (title, date, likes, comments count, authors...), which is cheaper.

## `contentFormats` (type: `array`):

Formats of the post body: clean plain text, Markdown (best for AI / RAG / LLM pipelines) and/or the original HTML.

## `includeComments` (type: `boolean`):

Also save the public comments (with replies) of every post. Comments on paid-only posts are not public and are skipped.

## `maxCommentsPerPost` (type: `integer`):

Maximum comments (including replies) saved per post.

## `commentSort` (type: `string`):

Order in which comments are read (matters when the max comments limit cuts the list).

## `includePublicationInfo` (type: `boolean`):

Add one row per publication with name, author, subscriber count, paid plans and prices, bestseller tier, podcast, sections and contributors.

## `onlyNewPosts` (type: `boolean`):

Remembers the newest post date of every publication and, on the next run, returns only newer posts. Ideal for scheduled runs: you never pay twice for the same post.

## `monitorStateKey` (type: `string`):

Name of the saved monitor state. Use a different name for each independent monitor or schedule. Only used in monitor mode.

## `maxConcurrency` (type: `integer`):

How many publications / searches / categories are processed at the same time.

## `requestDelayMs` (type: `integer`):

Minimum pause between two requests of the same input, in milliseconds (a small random jitter is added). Raise it if you see HTTP 429 warnings.

## `maxRetries` (type: `integer`):

How many times a failed request (network error, HTTP 429/5xx or anti-bot page) is retried with a new proxy session and exponential backoff.

## `proxyConfiguration` (type: `object`):

Apify Proxy spreads requests over many IPs and avoids rate limits on big runs. Datacenter proxies are usually enough for Substack.

## Actor input object example

```json
{
  "publications": [
    "https://www.astralcodexten.com"
  ],
  "searchType": "posts",
  "maxSearchResults": 20,
  "leaderboardRanking": "paid",
  "maxLeaderboardEntries": 25,
  "maxPostsPerPublication": 10,
  "postSort": "new",
  "audience": "all",
  "includeContent": true,
  "contentFormats": [
    "text",
    "markdown"
  ],
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "commentSort": "best_first",
  "includePublicationInfo": true,
  "onlyNewPosts": false,
  "monitorStateKey": "default",
  "maxConcurrency": 3,
  "requestDelayMs": 300,
  "maxRetries": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.astralcodexten.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("phantom-x64/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["https://www.astralcodexten.com"],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("phantom-x64/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.astralcodexten.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call phantom-x64/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,phantom-x64/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/2TnqftkRI9Vp3s2BK/builds/rQkxXEk7zJYENHZJX/openapi.json