# Substack Scraper (`exciting_pawpaw/substack-scraper`) Actor

Scrape any Substack newsletter: all posts with likes, comments, authors, paywall status, word count and optional full text as clean Markdown for LLM/RAG. Search posts or newsletters (with subscriber counts) by keyword. No login, fast JSON API.

- **URL**: https://apify.com/exciting_pawpaw/substack-scraper.md
- **Developed by:** [Markus Essl](https://apify.com/exciting_pawpaw) (community)
- **Categories:** Social media, AI, News
- **Stats:** 4 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Posts, Full Text & Newsletter Search

Scrape any Substack newsletter: get every archive post with title, date, authors, likes, comments, paywall status, word count and (optionally) the **full text as clean Markdown**, ready for LLMs and RAG pipelines. You can also **search all of Substack** for posts or newsletters by keyword, with subscriber counts. No login and no browser, so runs are fast and cheap.

### What can you do with it?

- **Feed AI agents and LLM / RAG pipelines** with newsletter content as clean Markdown. You can also call it from AI agents through the [Apify MCP server](https://mcp.apify.com).
- **Monitor newsletters**: track new posts from competitors, analysts or industry writers on a schedule (for example with `postedAfter: "7 days"`).
- **Content research**: find the most-liked posts of any newsletter (`sort: "top"`), or what Substack writers say about a topic.
- **Newsletter discovery and influencer research**: find newsletters in a niche with subscriber counts, categories and author info, for sponsorships, partnerships or market sizing.
- **Analytics**: likes, comments, restacks, word counts and free/paid mix over time.

### What data can you get?

**Posts** (from a newsletter archive, a post URL or a keyword search):

| Field | Description |
|---|---|
| `title`, `subtitle`, `description` | Headline, subtitle and teaser |
| `url`, `slug` | Absolute post URL |
| `publishedAt`, `updatedAt` | ISO-8601 dates (UTC) |
| `authors`, `authorHandles` | Byline names and Substack handles |
| `reactionCount`, `commentCount`, `restackCount` | Likes, comments and restacks |
| `audience`, `isPaywalled` | `everyone` (free), `only_paid`, `founding` … |
| `wordcount` | Post length in words |
| `coverImage` | Cover image URL |
| `tags`, `sectionName`, `language` | Post tags, newsletter section, language |
| `postType`, `podcastUrl`, `podcastDurationSeconds` | newsletter / podcast / thread, and audio info |
| `publicationId`, `publicationName`, `publicationUrl` | The newsletter the post belongs to |
| `contentMarkdown` | Full post body as clean Markdown (when `includeContent` is on) |
| `contentIsPreview` | `true` when the post is paywalled and only the free preview is included |
| `searchQuery`, `sourceInput`, `scrapedAt` | Where the record came from and when |

**Newsletters / publications** (from a keyword search with `searchType: "publications"`):

| Field | Description |
|---|---|
| `name`, `url`, `subdomain`, `customDomain` | Newsletter name and address |
| `description`, `logoUrl`, `createdAt` | Tagline, logo, launch date |
| `subscriberCount`, `subscriberCountNumber`, `subscriberBand` | Public subscriber count as Substack shows it (e.g. `"25K+"`, `25000`, `"Tens of thousands"`) |
| `paidSubscriptionsEnabled` | Whether it offers paid subscriptions |
| `authorName`, `authorHandle`, `authorProfileUrl`, `authorBio`, `authorFollowerCount` | Public writer profile info |
| `bestsellerTier`, `leaderboardCategory`, `leaderboardRank` | Substack bestseller badge and category leaderboard rank, when present |

No email addresses or other private contact data are collected.

### How to use

1. Add newsletter URLs to **Newsletter or post URLs** (`https://name.substack.com`, a custom domain like `https://www.astralcodexten.com`, just `name`, `https://substack.com/@handle` or a single post URL `…/p/slug`), and/or add **Search keywords**.
2. Choose options: **Include full text (Markdown)**, **Only posts published after**, **Only free posts**, **Archive order** (newest or top), and the result limits.
3. Click **Start**, then download the results as JSON, CSV or Excel, or read them through the API.

#### Input example

```json
{
  "startUrls": ["https://www.astralcodexten.com", "noahpinion", "https://substack.com/@climatehuman"],
  "searchQueries": ["retrieval augmented generation"],
  "searchType": "posts",
  "includeContent": true,
  "postedAfter": "1 year",
  "maxItems": 200,
  "maxItemsPerInput": 50
}
```

### Output example

A post found by keyword search, with `includeContent` on (Markdown shortened here):

```json
{
  "type": "post",
  "id": 139244404,
  "title": "A Practitioners Guide to Retrieval Augmented Generation (RAG)",
  "subtitle": "How basic techniques can be used to build powerful applications with LLMs...",
  "url": "https://cameronrwolfe.substack.com/p/a-practitioners-guide-to-retrieval",
  "slug": "a-practitioners-guide-to-retrieval",
  "publishedAt": "2024-02-05T10:24:15Z",
  "postType": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "authors": ["Cameron R. Wolfe, Ph.D."],
  "authorHandles": ["cwolferesearch"],
  "reactionCount": 99,
  "commentCount": 5,
  "restackCount": 10,
  "wordcount": 6574,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/a38a821b-0915-4907-9bc8-d0aae93b38db_1916x1072.png",
  "tags": ["LLM Fundamentals"],
  "language": "en",
  "publicationId": 1092659,
  "publicationName": "Deep (Learning) Focus",
  "publicationUrl": "https://cameronrwolfe.substack.com",
  "contentMarkdown": "(from [1, 5])\n\nThe recent surge of interest in generative AI has led to …",
  "contentIsPreview": false,
  "searchQuery": "retrieval augmented generation",
  "sourceInput": "retrieval augmented generation",
  "scrapedAt": "2026-10-09T13:50:13Z"
}
```

A newsletter found by keyword search (`searchType: "publications"`):

```json
{
  "type": "publication",
  "publicationId": 459118,
  "name": "The Bitcoin Layer",
  "url": "https://thebitcoinlayer.substack.com",
  "subdomain": "thebitcoinlayer",
  "description": "Bitcoin and global macro research firm.",
  "createdAt": "2021-08-25T19:31:30Z",
  "paidSubscriptionsEnabled": true,
  "authorName": "Nik Bhatia",
  "authorHandle": "thebitcoinlayer",
  "authorProfileUrl": "https://substack.com/@thebitcoinlayer",
  "subscriberCount": "25K+",
  "subscriberCountNumber": 25000,
  "subscriberBand": "Tens of thousands",
  "authorFollowerCount": 34627,
  "bestsellerTier": 100,
  "searchQuery": "bitcoin",
  "scrapedAt": "2026-10-09T13:50:24Z"
}
```

In the Apify Console, the **Posts** and **Newsletters** views show each record type as its own table.

### Pricing

**$1.00 per 1,000 results**, with no start fee. You pay only for the results you get, and full text costs nothing extra. A result is one post or one newsletter. Set `maxItems` to cap the cost of a run. The run also stops by itself when it reaches the maximum cost you set for the run.

### Tips

- **Whole archive:** set `maxItems` high (e.g. 5,000). Large newsletters with 1,000+ posts take a few minutes, longer with full text on.
- **Most popular posts:** set `sort` to `top`.
- **Only recent posts:** `postedAfter` accepts `2025-01-31` or relative values like `30 days`, `6 months`, `1 year`.
- **Precise post search:** Substack's post search is semantic, so it also returns loosely related posts. Turn on `strictKeywordMatch` to keep only posts that contain every keyword.

### FAQ

**Is it legal to scrape Substack?**
This Actor reads only public data that Substack serves to any visitor without logging in: public post metadata, free post content, and public newsletter and profile info. It does not get around paywalls or log in, and it does not collect emails. You are responsible for how you use the data, including copyright and privacy rules (e.g. GDPR) in your country. Don't republish other writers' content without permission.

**Can I get the full text of paid posts?**
No. For paywalled posts you get all the metadata plus the free preview Substack shows publicly (`contentIsPreview: true`).

**Do I need proxies?**
Usually not. The Actor uses Substack's public JSON endpoints and retries with backoff when rate-limited. If you run very large jobs and see repeated HTTP 429 warnings, enable Apify Proxy in the input.

**Are there limits?**

- Keyword search returns what Substack's own search returns. Newsletter search typically yields up to about 50–100 newsletters per keyword, and post search a few hundred posts per keyword.
- Subscriber counts are the rounded public figures Substack displays (e.g. "25K+"). Some writers hide them, and then the field is empty.
- Substack Notes (short social posts) and comments are not scraped.
- A single bad or non-Substack URL is logged and skipped. It doesn't stop the run.

**Can I use it with AI agents?**
Yes. Call the Actor from any MCP-compatible agent through the [Apify MCP server](https://mcp.apify.com), or from LangChain, LlamaIndex or your own code through the Apify API. Turn on `includeContent` to get LLM-ready Markdown.

# Actor input Schema

## `startUrls` (type: `array`):

Substack newsletters to scrape all archive posts from. Accepts newsletter URLs (https://name.substack.com), custom domains (https://www.astralcodexten.com), bare subdomains (name), profile URLs (https://substack.com/@handle) and single post URLs (…/p/post-slug).

## `searchQueries` (type: `array`):

Keywords to search across all of Substack. What is returned depends on 'Search for' below: matching posts, or matching newsletters.

## `searchType` (type: `string`):

What a keyword search returns: posts (articles) or publications (newsletters with subscriber counts).

## `strictKeywordMatch` (type: `boolean`):

Substack's post search is semantic and also returns loosely related posts. Enable to keep only posts whose title, subtitle, description, preview text or tags contain every word of your keyword.

## `maxItems` (type: `integer`):

Maximum number of results for the whole run. You pay only for results delivered.

## `maxItemsPerInput` (type: `integer`):

Optional cap per newsletter URL or search keyword. 0 = no per-input cap (only the total limit applies).

## `includeContent` (type: `boolean`):

Add the full post body as clean Markdown (field contentMarkdown) - great for LLM / RAG pipelines. Available for free posts; paywalled posts return no body. Needs one extra request per post, so runs are a bit slower.

## `sort` (type: `string`):

Order of newsletter archive posts: newest first, or most popular (top) first.

## `postedAfter` (type: `string`):

Optional. Only return posts published after this date. Absolute (2025-01-31) or relative (e.g. '30 days', '6 months', '1 year').

## `onlyFreePosts` (type: `boolean`):

Skip paywalled (paid-subscriber-only) posts.

## `proxyConfiguration` (type: `object`):

Usually not needed - Substack's public JSON API works without proxies. Enable Apify Proxy if you get rate-limited on very large runs.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.astralcodexten.com"
  ],
  "searchQueries": [],
  "searchType": "posts",
  "strictKeywordMatch": false,
  "maxItems": 20,
  "maxItemsPerInput": 0,
  "includeContent": false,
  "sort": "new",
  "postedAfter": "",
  "onlyFreePosts": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `publications` (type: `string`):

No description

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.astralcodexten.com"
    ],
    "maxItems": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("exciting_pawpaw/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.astralcodexten.com"],
    "maxItems": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("exciting_pawpaw/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.astralcodexten.com"
  ],
  "maxItems": 20
}' |
apify call exciting_pawpaw/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,exciting_pawpaw/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4ZCVR2x1vxghJz3eJ/builds/yPggMqw1qQOEz9n85/openapi.json