# Substack Scraper - Markdown Posts, Search & Comments (`gunpark/substack-scraper`) Actor

Scrape public Substack posts into clean Markdown for LLMs and RAG, with metadata and optional comments. Find publications by search; scrape by name, URL or custom domain. Filter by date or collect only new posts across scheduled runs. Paywalled posts: public previews only.

- **URL**: https://apify.com/gunpark/substack-scraper.md
- **Developed by:** [Alejandro Rodriguez](https://apify.com/gunpark) (community)
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.20 / 1,000 post delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper - Markdown Posts, Search & Comments

Scrape public Substack publications through their JSON API and get every post as **clean Markdown**, ready to drop into an LLM, a RAG pipeline or a vector database.

**When to use it:** choose this Actor to find Substack publications or retrieve their public posts as clean Markdown with metadata and optional comment threads for LLM/RAG ingestion or monitoring, including date filtering and incremental collection across scheduled runs.

Full paywalled content is unavailable; only public previews can be retrieved. Not affiliated with Substack.

- **Fast and cheap** - plain HTTP requests. No headless browser, no residential proxies.
- **LLM-ready output** - subscribe buttons, share widgets and embeds markup are stripped; headings, lists, code blocks, images, captions and footnotes are kept as standard Markdown.
- **Search built in** - give it a topic and it finds matching publications for you. Discovery is free.
- **Comments on demand** - optionally attach the comment thread to each post.
- **Incremental runs** - schedule it and receive only posts you have not seen yet.
- **Fair billing** - you pay only for posts actually delivered. Errors, duplicates and skipped paywalled posts are never charged.

### Quick start

Run the Actor with the prefilled input. It fetches the 10 latest posts of one publication in a few seconds and costs about $0.01:

```json
{
    "publications": ["astralcodexten"],
    "maxPosts": 10
}
```

Open the **Output** tab of the run to see the posts, or download them as JSON, CSV or Excel. Then replace the publication with your own and raise `maxPosts`.

### What you can scrape

| Data | Fields |
| --- | --- |
| Post | `id`, `url`, `slug`, `title`, `subtitle`, `type`, `publishedAt`, `updatedAt`, `description`, `coverImage` |
| Content | `markdown`, `markdownLength`, `wordCount`, `isPaywalled`, `contentComplete` |
| People & topics | `authors[]` (`id`, `name`, `handle`), `tags[]` |
| Engagement | `reactionCount`, `commentCount` |
| Comments (optional) | `comments[]` with `id`, `parentId`, `depth`, `author`, `authorHandle`, `date`, `body`, `reactionCount` |
| Publication | `publicationId`, `publicationUrl` |

### Input

| Field | Description |
| --- | --- |
| `publications` | Names (`astralcodexten`), Substack URLs or custom domains. |
| `searchQuery` | Optional topic. Matching publications are scraped as well. |
| `maxSearchPublications` | How many publications from the search to scrape (default 5, max 25). |
| `maxPosts` | Total posts to deliver in the run (default 100). |
| `maxPostsPerPublication` | Optional cap per publication (0 = no cap). |
| `sinceDate` | Only posts published on or after this date. `YYYY-MM-DD` or relative, e.g. `7 days`. |
| `incremental` | Deliver only posts newer than the previous run (see below). |
| `paywalledPosts` | `skip` (default, not charged) or `preview` (public preview, flagged). |
| `includeComments` / `maxCommentsPerPost` | Attach up to N comments and replies per post, best first. |
| `proxyConfiguration` | Off by default. Enable Apify datacenter proxy only if a run reports a block. |

Provide `publications`, `searchQuery`, or both. Posts are delivered newest first.

```json
{
    "publications": ["astralcodexten", "https://www.lennysnewsletter.com"],
    "maxPosts": 200,
    "sinceDate": "30 days"
}
```

### Output

```json
{
    "id": 224550256,
    "publicationId": 89120,
    "publicationUrl": "https://astralcodexten.substack.com",
    "url": "https://www.astralcodexten.com/p/ai-company-employees-should-unionize",
    "title": "AI Company Employees Should Unionize",
    "type": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "contentComplete": true,
    "publishedAt": "2026-11-12T23:09:15.486Z",
    "authors": [{ "id": 12009663, "name": "Scott Alexander", "handle": "astralcodexten" }],
    "tags": [],
    "wordCount": 1273,
    "reactionCount": 306,
    "commentCount": 211,
    "markdown": "# AI Company Employees Should Unionize\n\n**I.**\n\nShould safety-conscious employees ...",
    "markdownLength": 11328,
    "commentsStatus": "not_requested",
    "comments": null,
    "scrapedAt": "2026-11-15T21:50:40.000Z"
}
```

The `markdown` field starts with the title as an H1 (and the subtitle, when there is one) so each document is self-contained when chunked.

The key-value store of every run also contains:

- `RUN_SUMMARY` - why the run stopped (`completed`, `max_posts`, `budget`, `blocked`), per-publication status, and how many posts were skipped and why.
- `DISCOVERED_PUBLICATIONS` - the publications found by `searchQuery`.

### Pricing

Pay per event. You are charged only when a post is stored in your dataset.

| Event | Price | When |
| --- | --- | --- |
| Post delivered | $0.0012 ($1.20 per 1,000 posts) | A valid post with content is stored in the dataset. |
| Post comments | $0.0005 ($0.50 per 1,000 posts) | Only with `includeComments`, once per post that has at least one comment delivered. It is per post, not per comment. |
| Actor start | $0.00005 | Once per run. |

What typical runs cost (plus $0.00005 for the start):

| Run | Cost |
| --- | --- |
| 10 latest posts | $0.012 |
| 500 posts | $0.60 |
| A 10,000-post archive | $12.00 |
| 1,000 posts, all with comments | $1.70 |

Never charged: failed requests, duplicates, posts with no content, paywalled posts you chose to skip, and search discovery. A run that delivers nothing only costs the start event.

Set a maximum cost per run in the run options: the Actor stops cleanly when the limit is reached and never goes over it.

### Incremental mode

With `incremental: true` the Actor remembers, per publication, the newest post it delivered.

- The **first run** works normally and sets the baseline.
- **Later runs** deliver only posts published after that baseline, oldest first, so a run interrupted by `maxPosts` or your cost limit simply continues next time. Nothing is lost and nothing is delivered twice.
- State is kept in a named key-value store, `substack-scraper-state`, in your account. Use a different `stateNamespace` for each independent job.

Incremental mode tracks new posts only. A post edited after it was delivered is not delivered again.

To keep an index or a feed up to date:

1. Set your publications and `incremental: true`, and save that configuration as a task.
2. Create a schedule for the task, for example once a day.
3. Each run's dataset then contains only the posts published since the previous run. Forward them to your pipeline with a webhook or any Apify integration.

### Run it from code

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("gunpark/substack-scraper").call(
    run_input={"publications": ["astralcodexten"], "maxPosts": 20}
)

for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(post["title"], len(post["markdown"]))
```

### Use it from AI agents (MCP)

The Actor works as a tool for any MCP client through the hosted Apify MCP server. Preload it with:

```json
{
    "mcpServers": {
        "apify": {
            "url": "https://mcp.apify.com?tools=gunpark/substack-scraper"
        }
    }
}
```

The agent can then call it with a publication name or a search query and receive Markdown it can read directly.

### Limits and notes

- **Paywalled posts** expose only a public preview without a subscription. With `paywalledPosts: "preview"` they are delivered with `isPaywalled: true` and `contentComplete: false`; some have no public preview at all and are skipped for free.
- **Podcast and video posts** are delivered when they have written content; the media itself is not downloaded.
- **Blocks and rate limits** - requests are retried with backoff. On a hard block the run stops, keeps what it already delivered and reports `blocked` in `RUN_SUMMARY`. If nothing was delivered the run is marked as failed.
- **Personal data** - comments include public author names and handles. Make sure your use of that data complies with the laws that apply to you.
- This Actor reads only publicly available data and is not affiliated with Substack.

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape. Accepts a name (<code>astralcodexten</code>), a Substack URL (<code>https://name.substack.com</code>) or a custom domain (<code>https://www.example.com</code>).

## `searchQuery` (type: `string`):

Optional. Finds publications matching this query and scrapes them too. Discovery is free: you only pay for delivered posts.

## `maxSearchPublications` (type: `integer`):

How many publications found by the search query to scrape.

## `maxPosts` (type: `integer`):

The run stops after delivering this many posts across all publications. Start small to try it out, then raise it.

## `maxPostsPerPublication` (type: `integer`):

Optional cap per publication. 0 means no per-publication cap.

## `sinceDate` (type: `string`):

Skip posts published before this date. Use an absolute date (YYYY-MM-DD) or a relative one such as "7 days".

## `incremental` (type: `boolean`):

Remembers the newest post delivered per publication and, on later runs, delivers only posts published after it. The first run sets the baseline. Ideal for scheduled runs. Does not re-deliver edited posts.

## `stateNamespace` (type: `string`):

Use a different name for each independent incremental job so they do not share state.

## `paywalledPosts` (type: `string`):

Paid posts only expose a public preview. Skip them (not charged) or deliver the preview flagged with <code>isPaywalled: true</code>.

## `includeComments` (type: `boolean`):

Adds the comment thread to each post. Charged as a separate event, once per post that has at least one comment delivered.

## `maxCommentsPerPost` (type: `integer`):

Comments and replies are flattened, best first, up to this number.

## `proxyConfiguration` (type: `object`):

Not needed in normal use. Turn on Apify datacenter proxy only if a run reports being blocked. Residential proxies are never required.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten"
  ],
  "maxSearchPublications": 5,
  "maxPosts": 10,
  "maxPostsPerPublication": 0,
  "incremental": false,
  "stateNamespace": "default",
  "paywalledPosts": "skip",
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

One item per delivered post: clean Markdown content, metadata and optional comments.

## `runSummary` (type: `string`):

Why the run stopped (completed, max_posts, budget or blocked), per-publication status, and how many posts were skipped without charge.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten"
    ],
    "maxPosts": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("gunpark/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["astralcodexten"],
    "maxPosts": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("gunpark/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten"
  ],
  "maxPosts": 10
}' |
apify call gunpark/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gunpark/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/y60NxUWmd21xzmpiY/builds/jsmgyJDao15rVXHYC/openapi.json
