# Substack Scraper: Posts, Full Text, Reactions & Paywall (`themineworks/substack-scraper`) Actor

Scrape any Substack newsletter archive: title, subtitle, full post text, author, publish date, reactions, comment count and whether the post is behind the paywall. No login, no API key, pay per post.

- **URL**: https://apify.com/themineworks/substack-scraper.md
- **Developed by:** [The Mine Works](https://apify.com/themineworks) (community)
- **Categories:** News, Social media, MCP servers
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 📰 Substack Scraper: Posts, Full Text, Reactions & Paywall

> ⚡ Pure HTTP. No login, no API key, no proxy, no browser.

> 💸 You are only charged for a post that actually lands in your dataset. Publications that fail to load are never billed.

### Overview

Substack Scraper turns any Substack newsletter into structured rows. Give it a publication name, a subdomain, or a full URL, and it returns every post with title, subtitle, author, publish date, word count, reaction count, comment count, the canonical URL, and the full post text.

It reads each publication's own archive endpoint, so there is nothing to log into and no key to manage. Custom domains work exactly like `*.substack.com` addresses.

✅ Any publication, including custom domains · ✅ Free and paywalled posts identified · ✅ Engagement metrics per post · ✅ MCP-ready for AI agents

### What you get that a plain RSS reader does not

Substack's public RSS feed carries roughly the last 20 posts and no engagement data at all. This actor reads the archive API instead, which means:

- **The whole back catalogue**, not the last 20 posts, paginated 50 at a time.
- **Reactions and comment counts** per post, so you can rank a newsletter's archive by what actually landed.
- **Paywall status** on every post, via `audience` (`everyone` or `only_paid`) and an `is_paywalled` boolean.
- **Word count** as Substack itself reports it.

### About paywalled posts

Every post carries `audience`, so you always know whether it is free or paid. For paid posts Substack serves the publicly available portion, which for many publications is most of the article and for others is a short preview.

We report what we actually received rather than guessing: when the delivered text is materially shorter than the word count Substack declares, the row is flagged `content_truncated: true`.

**We do not attempt to bypass the paywall.** It is the publication's business model, breaking it would be a legal problem for you, and any such trick would break the moment Substack tightened it. If you need the full text of paid posts, subscribe to the publication.

### Input

```json
{
  "publications": ["bigtechnology", "platformer.news", "https://www.astralcodexten.com/"],
  "maxPostsPerPublication": 50,
  "includeBody": true,
  "audienceFilter": ""
}
```

| Field | What it takes |
| --- | --- |
| `publications` | A plain name (`bigtechnology`), a subdomain, or any full URL. Custom domains are supported. |
| `maxPostsPerPublication` | Hard cap per publication. Your main cost and runtime control. |
| `includeBody` | Fetch the full post text. **This costs one extra request per post**, see the note below. Turn it off for a fast metadata-only pass. |
| `audienceFilter` | Limit to free posts, paywalled posts, or leave blank for both. |

**Why `includeBody` matters for cost.** The archive endpoint returns 50 posts of metadata in a single request, but it does **not** include post bodies: it returns an empty `body_html` field at every page size. Real bodies come from the per-post endpoint, one request each. So a metadata-only run is one request per 50 posts, while a full-text run is one request per post. Both are supported; pick the one your use case needs.

### Output

```json
{
  "publication": "bigtechnology",
  "post_id": "209279848",
  "title": "When Artificial Intelligence Is Too Valuable To Sell",
  "subtitle": "The frontier labs are rethinking who gets access",
  "url": "https://www.bigtechnology.com/p/when-artificial-intelligence-is-too",
  "authors": ["Alex Kantrowitz"],
  "published_at": "2026-07-31T19:30:17.712Z",
  "post_type": "newsletter",
  "audience": "only_paid",
  "is_paywalled": true,
  "word_count": 1139,
  "reactions_total": 74,
  "reactions": { "❤": 74 },
  "comment_count": 0,
  "body_text": "It's time to delete the assumption that the frontier AI labs will always license...",
  "scraped_at": "2026-08-13T13:02:44.108Z"
}
```

| Field | Description |
| --- | --- |
| 📰 `publication` | Publication name |
| 🔑 `post_id` | Substack post ID, stable for deduplication across runs |
| 📄 `title` / `subtitle` | Post headline and standfirst |
| 🔗 `url` | Canonical post URL |
| 👤 `authors` | Bylined authors |
| 📅 `published_at` | ISO publish timestamp |
| 🏷️ `post_type` | `newsletter`, `podcast`, `thread` and so on |
| 🔓 `audience` | `everyone` (free) or `only_paid` (paywalled) |
| 🔒 `is_paywalled` | Boolean form of the above |
| 📏 `word_count` | Word count Substack reports for the full post |
| ❤️ `reactions_total` | All reactions summed into one sortable number |
| 😀 `reactions` | The raw reaction map keyed by emoji |
| 💬 `comment_count` | Number of comments |
| 📃 `body_text` | Post body as clean plain text |
| ✂️ `content_truncated` | True when only part of a paid post was publicly available |
| 🕒 `scraped_at` | ISO timestamp of capture |

**Why reactions appear twice.** Substack returns reactions as a map keyed by emoji (`{"❤": 74}`), which is faithful but impossible to sort on. We keep the raw map and add a summed integer so you can rank an archive by engagement without writing your own reducer.

### Common use cases

**Newsletter competitive research.** Pull a competitor's full archive, sort by `reactions_total`, and see exactly which topics their audience responded to.

**Content and topic research.** Search across the body text of a set of newsletters in your space to find who has covered a subject and how.

**Author and market intelligence.** Track publishing cadence, word counts and the free-versus-paid mix across publications to understand how a newsletter is monetising.

**AI and RAG pipelines.** Newsletter archives are dense, well-edited prose, which makes them strong retrieval material. Pull once, index, and keep current on a schedule.

**Media monitoring.** Watch a set of publications for mentions of a company, person or product.

### Pricing

| Event | Price | You pay when |
| --- | --- | --- |
| Post delivered | $0.0015 | A post lands in your dataset |

**$1.50 per 1,000 posts.** Publications that do not exist or return nothing cost nothing.

### Run it on a schedule

1. Run once with the input you want repeated, then click **Save as a task**.
2. In the Apify Console go to **Schedules → Create new**.
3. Pick a frequency and attach the saved task.
4. Wire the dataset to Sheets, Slack or a webhook from the **Integrations** tab.

Deduplicate on `post_id` so a repeated schedule never reprocesses the same post.

### FAQ

**Do I need a Substack account or subscription?**
No. The actor reads publicly available archive data. A subscription is only relevant if you want the full text of paid posts, which this actor does not bypass.

**Does it work with custom domains?**
Yes. Many established newsletters publish from their own domain rather than `*.substack.com`, and both are handled.

**How far back does it go?**
As far as the publication's archive allows, paginated 50 posts at a time up to your cap.

**Why is my run slower with `includeBody` on?**
Because each post body is a separate request. See the cost note above.

**Why did a paid post come back shorter than its word count?**
Because only part of it is public. That row is flagged `content_truncated: true`.

### Use from Claude, ChatGPT and any MCP agent

```
https://mcp.apify.com/?tools=themineworks/substack-scraper
```

Or call it programmatically:

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('themineworks/substack-scraper').call({
  publications: ['bigtechnology', 'platformer.news'],
  maxPostsPerPublication: 50,
  includeBody: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

Questions, or need a field we don't return yet? Reach out through the Apify profile.

# Actor input Schema

## `publications` (type: `array`):

Accepts a plain name (bigtechnology), a subdomain (bigtechnology.substack.com), or any full Substack URL. Custom domains work too.

## `maxPostsPerPublication` (type: `integer`):

Hard cap per publication. The archive returns 50 posts per request, so this is your main cost and runtime control.

## `includeBody` (type: `boolean`):

Return the post body converted to plain text. Free posts return in full. Paid posts return the public preview only, flagged with content\_truncated.

## `audienceFilter` (type: `string`):

Limit to free posts or paywalled posts. Leave blank for both.

## Actor input object example

```json
{
  "publications": [
    "bigtechnology",
    "platformer"
  ],
  "maxPostsPerPublication": 25,
  "includeBody": true,
  "audienceFilter": ""
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "bigtechnology",
        "platformer"
    ],
    "maxPostsPerPublication": 25
};

// Run the Actor and wait for it to finish
const run = await client.actor("themineworks/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "bigtechnology",
        "platformer",
    ],
    "maxPostsPerPublication": 25,
}

# Run the Actor and wait for it to finish
run = client.actor("themineworks/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "bigtechnology",
    "platformer"
  ],
  "maxPostsPerPublication": 25
}' |
apify call themineworks/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,themineworks/substack-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JPteGw0qtbnDhdwCE/builds/zXgQBd7nP5cnWWhz8/openapi.json