# Substack Scraper: Posts, Full Text & Newsletter Data (`green_trousers/substack-scraper`) Actor

Scrape any Substack newsletter, custom domains too: every post with full text as clean Markdown, likes, comments, restacks, tags, authors, paid/free flag and subscriber counts. Optional comments and Substack-wide keyword search. Fast JSON, no browser. Built for AI/RAG and research.

- **URL**: https://apify.com/green_trousers/substack-scraper.md
- **Developed by:** [Omar Prime](https://apify.com/green_trousers) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Posts, Full Text & Newsletter Data

**Turn any Substack newsletter into clean, structured data.** Paste newsletter URLs (custom domains work too) and get every post with its full text as **LLM-ready Markdown**, plain text and HTML, plus likes, comments, restacks, tags, authors, paid/free status and the newsletter's public subscriber count. Add a keyword to **search all of Substack** at once.

It reads the same public JSON that every Substack site loads in your browser, so it is **fast (100 posts in about 5 seconds), cheap, and doesn't break** when Substack changes its page design. No browser, no proxies, no login.

### What you get

- **Every post in the archive**, newest first: title, subtitle, URL, authors, publish date, word count, cover image, tags, section, podcast link.
- **Full text, three ways:** clean Markdown (headings, links, lists, images, captions kept; subscribe buttons and share widgets removed), plain text, and the original HTML.
- **Engagement:** likes (reactions), comment count, restacks.
- **Paid vs free:** `audience` (free / paid / founding / subscribers_only) and a `paywalled` flag.
- **Comments (optional):** public comments and replies with author, date, likes and thread position.
- **Newsletter profile:** name, author, description, public subscriber count ("Over 1,200,000 subscribers"), paid tier, language, launch date.
- **Search across Substack (optional):** find posts about any topic from any newsletter.

### Use cases

- **AI, RAG and fine-tuning:** feed newsletters into a vector database or a custom GPT as clean Markdown with source URLs and dates. Schedule it daily with `since: "1 day"` to keep the index fresh.
- **Creator and market research:** benchmark newsletters in your niche: posting cadence, free vs paid mix, what topics get the most likes, comments and restacks.
- **Competitive intelligence:** track what competitors, investors or analysts publish, the moment they publish it.
- **Media monitoring and PR:** search all of Substack for your brand, product or topic.
- **Lead generation and partnerships:** find active writers in a niche with their audience size for sponsorships, guest posts and outreach.
- **Personal archives:** back up your own newsletter, or a favorite one, as Markdown.

### Input

| Field | What it does | Default |
|---|---|---|
| `publications` | Newsletter URLs (`https://name.substack.com`), custom domains (`www.lennysnewsletter.com`), bare subdomains (`lenny`), author handles (`@lenny`) or single post URLs (`…/p/slug`) | 2 examples |
| `searchQuery` | Keyword search across all of Substack | empty |
| `maxSearchResults` | Max posts from the search | 20 |
| `maxPostsPerPublication` | Newest posts first; `0` = whole archive | 50 |
| `since` | Only posts on/after a date (`2026-01-01`) or within a window (`7 days`) | any date |
| `audience` | `all`, `free` or `paid` posts | all |
| `includeContent` | Full text as Markdown, text and HTML | true |
| `includeHtml` | Also keep the raw HTML (turn off for smaller datasets) | true |
| `includeComments` | Add comments and replies to each post | false |
| `maxCommentsPerPost` | Comment cap per post | 100 |
| `includePublicationInfo` | Add author, description and subscriber count; save a `PUBLICATIONS` record | true |

```json
{
  "publications": ["https://www.lennysnewsletter.com", "oneusefulthing", "@garymarcus"],
  "maxPostsPerPublication": 100,
  "since": "2026-01-01",
  "includeContent": true,
  "includeComments": false,
  "searchQuery": "AI agents",
  "maxSearchResults": 50
}
```

### Output

One row per post:

```json
{
  "publication": "One Useful Thing",
  "publication_url": "https://www.oneusefulthing.org",
  "publication_id": 1180644,
  "post_id": 204003560,
  "title": "The twilight of the chatbots",
  "subtitle": "How work changes along the exponential",
  "slug": "the-twilight-of-the-chatbots",
  "url": "https://www.oneusefulthing.org/p/the-twilight-of-the-chatbots",
  "type": "newsletter",
  "author": "Ethan Mollick",
  "authors": [{ "name": "Ethan Mollick", "handle": "oneusefulthing", "id": 846835 }],
  "published_at": "2026-06-30T22:18:12.226Z",
  "audience": "free",
  "paywalled": false,
  "word_count": 1271,
  "reactions": 808,
  "comment_count": 103,
  "restacks": 78,
  "cover_image": "https://substackcdn.com/image/fetch/...",
  "tags": [],
  "description": "How work changes along the exponential",
  "content_markdown": "If you feel like things are accelerating in AI, you are probably right. Better AI models...\n\n![](https://substackcdn.com/image/...)\n\n...",
  "content_text": "If you feel like things are accelerating in AI, you are probably right...",
  "content_html": "<p>If you feel like things are accelerating in AI...</p>",
  "content_word_count": 1255,
  "publication_author": "Ethan Mollick",
  "publication_subscribers": "Over 488,000 subscribers",
  "source": "publication",
  "scraped_at": "2026-10-22T18:52:49.558Z"
}
```

With `includeComments`, each post also has `comments`: `[{ comment_id, parent_id, depth, author_name, author_handle, body, published_at, edited_at, reactions, restacks }]`.

The key-value store also holds:

- `PUBLICATIONS`: one full profile per newsletter (name, subdomain, custom domain, author, description, subscriber count, paid tier, language, podcast, logo, created date, first post date).
- `SUMMARY`: posts found per input, and why an input failed (for example, a site that isn't on Substack).

### Pricing

Pay only for what you get. Full text is included in the post price.

| Event | Price | 1,000 = |
|---|---|---|
| Post (metadata + full text) | **$0.001** | $1.00 |
| Comment (optional) | **$0.0003** | $0.30 |

No extra run-start fee beyond Apify's minimal default. Inputs that aren't Substack newsletters cost nothing. Set a maximum cost per run in Apify and the Actor stops cleanly when it is reached.

### FAQ

**Does it get paid (paywalled) posts?**
It returns exactly what Substack shows any visitor who isn't a paying subscriber: the title, stats and the **free preview** of the post, with `paywalled: true` and `audience: "paid"`. It never tries to get around a paywall. Comments on paid posts are usually hidden from non-subscribers too, so they come back empty. Use `audience: "free"` to skip paid posts entirely.

**Do custom domains work?**
Yes. `www.lennysnewsletter.com`, `www.noahpinion.blog`, `newsletter.pragmaticengineer.com` and any other Substack on its own domain work the same as `name.substack.com`. Subdomains that redirect to a custom domain are followed.

**What's the difference between `word_count` and `content_word_count`?**
`word_count` is Substack's own count for the full post. `content_word_count` is the number of words actually returned, which is lower for paywalled previews.

**How fresh is the data?**
Live. Every run reads the newsletter at that moment. Schedule runs with `since: "1 day"` to collect only new posts.

**Is the Markdown good enough for an LLM?**
Yes. Substack's subscribe boxes, share buttons and image-expand clutter are removed, while headings, lists, links, quotes, images and captions are kept.

**Is subscriber count exact?**
Substack only shows rounded public figures ("Over 121,000 subscribers", "Tens of thousands of paid subscribers"), and that's what you get.

**Is this allowed?**
The Actor only reads public pages and public data that Substack serves to every visitor, at a polite rate, with no login. You are responsible for how you use the content: respect the authors' copyright and Substack's terms, and don't republish full posts without permission.

**Something missing?**
Open an issue on the Actor's Issues tab and it will usually be added quickly.

# Actor input Schema

## `publications` (type: `array`):

Newsletter URLs (https://name.substack.com), custom domains (www.lennysnewsletter.com), bare subdomains (lenny), author handles (@lenny) or single post URLs (…/p/post-slug).

## `searchQuery` (type: `string`):

Keyword search across every Substack newsletter (e.g. "AI agents", "creator economy"). Matching posts are returned in the same format. Leave empty to only scrape the newsletters above.

## `maxSearchResults` (type: `integer`):

Upper limit of posts returned from the search query.

## `maxPostsPerPublication` (type: `integer`):

Newest posts first. 0 = the entire archive.

## `since` (type: `string`):

Only posts published on or after this date (e.g. 2026-01-01) or within a relative window (e.g. 7 days). Great for daily or weekly scheduled runs. Empty = any date.

## `audience` (type: `string`):

Keep all posts, only free (public) posts, or only paid/subscriber posts. Paid posts include the free preview only.

## `includeContent` (type: `boolean`):

Adds the post body as clean Markdown, plain text and HTML. For paid posts you get the public free preview and paywalled: true. Turn off for metadata-only runs.

## `includeHtml` (type: `boolean`):

Also return the original body HTML (content_html). Turn off to keep datasets small when you only need Markdown/text.

## `includeComments` (type: `boolean`):

Adds public comments and replies to each post (charged per comment).

## `maxCommentsPerPost` (type: `integer`):

Upper limit of comments (including replies) per post when comments are on.

## `includePublicationInfo` (type: `boolean`):

Adds the newsletter's author, description and public subscriber count to each post, and saves a PUBLICATIONS record with full newsletter profiles. Free.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com",
    "oneusefulthing"
  ],
  "searchQuery": "",
  "maxSearchResults": 20,
  "maxPostsPerPublication": 50,
  "audience": "all",
  "includeContent": true,
  "includeHtml": true,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "includePublicationInfo": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "oneusefulthing"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("green_trousers/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "https://www.lennysnewsletter.com",
        "oneusefulthing",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("green_trousers/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com",
    "oneusefulthing"
  ]
}' |
apify call green_trousers/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,green_trousers/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/SdtEZbiL1FhMLbXZ4/builds/Uh9yd4AdZJCQ4ScvP/openapi.json