# Substack Scraper (`romajure/substack-scraper`) Actor

Scrape public Substack publications without login: posts, authors, dates, full text and HTML, paywall flag, likes, restacks, tags and comments. Pay per post.

- **URL**: https://apify.com/romajure/substack-scraper.md
- **Developed by:** [Romas Jurevicius](https://apify.com/romajure) (community)
- **Categories:** News, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Substack Scraper

Substack Scraper extracts posts from any **public Substack publication** without a Substack account, API key or browser. It reads the same public JSON endpoints that power Substack's own archive pages (`/api/v1/archive` and `/api/v1/posts/<slug>`), walks back through the archive page by page, and returns one clean JSON record per post: title, subtitle, authors, publication date, post type, full text and HTML, paywall flag, word count, cover image, likes, restacks, comment count, tags and, optionally, the public comment thread.

Typical uses: monitoring newsletters in your niche, building datasets for research or NLP, feeding posts into an AI agent or n8n/Make automation, archiving a publication, tracking competitors, or syncing new posts into a database or CMS.

The scraper works with publications on `*.substack.com` **and** on custom domains (for example `www.astralcodexten.com` or `www.noahpinion.blog`); subdomains that moved to a custom domain are followed automatically. Paywalled posts return only the free preview that Substack shows to logged-out readers, flagged with `isPaywalled: true`. Private (invite-only) publications and unknown subdomains produce a status record (`private`, `not_found`) instead of failing the run.

### How to scrape a Substack publication

1. Open the Actor in Apify Console and click **Try for free**.
2. In **Publications**, add one or more publications. Any of these forms work: `noahpinion`, `noahpinion.substack.com`, `https://www.astralcodexten.com`, `https://www.lennysnewsletter.com/p/some-post`.
3. Set **Max posts per publication** (default 10, newest first). Optionally set **Since date** to stop at a date, turn on **Include full body** to fetch each post's full text and HTML (off by default, so the run makes no per-post request), or turn on **Include comments** to get the public thread of every post.
4. Click **Start**. Results appear in the **Output** tab within seconds and can be downloaded as JSON, CSV, Excel or XML, or fetched through the API.

The archive is read 20 posts per page, so the default of 10 posts is one page load, and 50 posts means three page loads plus one request per post when the full body is on. Publications are processed in parallel with a short delay between requests to the same host.

#### How pagination works

Substack's archive endpoint lists posts newest first and accepts an `offset`. The Actor follows it until it has collected the requested number of posts, reaches the `sinceDate` cutoff, or receives an empty page (the beginning of the publication's history). Posts are de-duplicated by id, so a run that migrates between servers never produces the same post twice.

#### Custom domains

Many large publications serve their Substack from their own domain. You can pass either the Substack subdomain or the custom-domain URL. When a subdomain redirects to a custom domain the redirect is followed; when a custom domain is given without `www.` or is reachable only through its canonical URL, the Actor tries the `www.` variant and the homepage's canonical link before reporting `not_found`.

### Substack Scraper pricing

This Actor uses Apify's **pay-per-event** model. There is no subscription and no per-result platform fee on top:

- **Actor start**: a flat fee of $0.00005 per run.
- **Post scraped**: **$0.002 per post** pushed to your dataset (that is $2 per 1,000 posts).
- **Comment scraped**: **$0.0005 per comment** returned inside a post's `comments` array (only when **Include comments** is on).

Status records for publications that do not exist, are private or are empty are free. You can cap spending with **Maximum cost per run**; when the cap is reached the Actor stops cleanly and keeps everything scraped so far. Apify's free plan credit is enough to scrape thousands of posts for evaluation.

### Input example

```json
{
    "publications": ["astralcodexten", "noahpinion", "https://www.lennysnewsletter.com"],
    "maxPostsPerPublication": 50,
    "sinceDate": "2026-01-01",
    "includeBody": true,
    "includeComments": false,
    "maxCommentsPerPost": 50,
    "proxyConfiguration": { "useApifyProxy": true }
}
```

#### Input fields

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `publications` | array of strings | required | Subdomains, `*.substack.com` hosts or custom-domain URLs of public publications. |
| `maxPostsPerPublication` | integer | 10 | Newest N posts to collect per publication. |
| `sinceDate` | ISO date string | none | Stop once posts are older than this date. |
| `includeBody` | boolean | false | Fetch each post for its full `text` and `bodyHtml`. |
| `includeComments` | boolean | false | Fetch the public comment thread into `comments`. |
| `maxCommentsPerPost` | integer | 50 | Cap on comments (replies included) per post. |
| `proxyConfiguration` | object | Apify Proxy | Standard Apify proxy settings. |

### Output example

One dataset item per post:

```json
{
    "publication": "https://www.noahpinion.blog",
    "publicationName": "Noahpinion",
    "publicationStatus": "ok",
    "postId": 219538380,
    "slug": "maybe-the-rapture-is-already-here",
    "url": "https://www.noahpinion.blog/p/maybe-the-rapture-is-already-here",
    "title": "Maybe the Rapture is already here",
    "subtitle": "In the future, we're all heavenbanned.",
    "authors": [{ "name": "Noah Smith", "handle": "noahpinion" }],
    "publishedAt": "2026-10-09T09:18:32.051Z",
    "type": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "wordCount": 1577,
    "text": "Back when I was a teenager, a lot of people were very into the Left Behind series...",
    "bodyHtml": "<div class=\"captioned-image-container\"><figure>...</figure></div><p>Back when I was a teenager...</p>",
    "coverImage": "https://substackcdn.com/image/fetch/...png",
    "likes": 458,
    "commentCount": 45,
    "restacks": 63,
    "tags": [],
    "scrapedAt": "2026-10-10T19:30:12.401Z"
}
```

`text` is a readable, markdown-ish rendering of the post (headings keep `#` markers, lists use `- `, footnote anchors become `[n]`, subscribe buttons are removed). `type` is Substack's post type (`newsletter`, `podcast`, `thread`, `video`), `audience` is `everyone` or `only_paid` (founding tiers appear as served). When **Include comments** is on, `comments` holds `{ "id", "parentId", "author", "handle", "date", "body", "likes" }` objects, flattened depth-first. Publications that could not be scraped produce a single record with `publicationStatus` set to `not_found`, `private`, `empty` or `error` and a human-readable `message`.

### Integrations

- **API**: start runs and read results with the [Apify API](https://docs.apify.com/api/v2) or the JavaScript and Python clients. Run the Actor synchronously with `run-sync-get-dataset-items` to get posts in one call.
- **MCP / AI agents**: the Actor is exposed through the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp), so Claude, ChatGPT and other agents can call it as a tool with the same input.
- **n8n and Make**: use the Apify node or module, pick this Actor, pass the input JSON above and map the dataset items into your workflow.
- **Schedules and webhooks**: run it hourly or daily from Apify Schedules with a `sinceDate` of yesterday and let a webhook push new posts to Slack, Notion, Google Sheets, a database or a CRM.

### Substack Scraper FAQ

#### Does it work for paywalled posts?

Partially. Substack's public API returns the free preview of paid posts (typically the first few paragraphs) to anyone who is not a paying subscriber, and that is what you get, flagged with `isPaywalled: true`. The Actor never logs in and cannot read the locked part.

#### Does it work for private or invite-only publications?

No. Those publications do not expose a public archive. The Actor returns a `private` status record so your workflow can skip them.

#### Can I scrape a publication on its own domain?

Yes. Pass the custom-domain URL (for example `https://www.lennysnewsletter.com`). Some publications have retired their original `*.substack.com` subdomain entirely; in that case pass the custom domain, because the subdomain now redirects to a profile page and the Actor reports `not_found` with an explanatory message.

#### Can I get comments?

Yes, turn on **Include comments**. Each post then costs one extra request and $0.0005 per comment returned, up to **Max comments per post**. Replies are included and flattened, with `parentId` pointing at the parent comment.

#### How fresh is the data?

Every run reads Substack's live API, so results are as current as the publication's own archive page, including likes, restacks and comment counts at the time of the run.

#### Can I scrape many publications at once?

Yes. Add as many publications as you like; they are processed in parallel with built-in rate limiting and automatic retries.

### Related actors

Same author, same pay-per-event model and the same no-login approach. They combine well in one pipeline (for example calendar events plus prediction odds plus channel chatter around a release):

- [Telegram Channel Scraper](https://apify.com/romajure/telegram-channel-scraper) - posts, views, media and links from public Telegram channels, no login.
- [Economic Calendar Scraper](https://apify.com/romajure/economic-calendar) - Forex Factory and Investing.com macro events with UTC times, impact, actual, forecast and previous values.
- [Prediction Market Odds Scraper](https://apify.com/romajure/prediction-market-odds) - Polymarket and Kalshi markets with 0-1 prices, implied probabilities, bid/ask, volume and orderbooks.
- [Product Hunt Scraper](https://apify.com/romajure/product-hunt-scraper) - daily, weekly and monthly leaderboards, topic pages, search and product launches with votes, makers and website URLs.

### Legal and compliance

This Actor only reads publicly available data that Substack serves to anyone without logging in, and it does not bypass any access control or paywall. You are responsible for using the data in line with [Substack's Terms of Use](https://substack.com/tos), the rights of the authors whose content you collect, applicable law and, where personal data is involved (for example comment authors), GDPR or other privacy regulations. Please read Apify's article on the [legality of web scraping](https://blog.apify.com/is-web-scraping-legal/) if you are unsure.

### Support

Found a publication that does not parse correctly, or need an extra field? Open an issue on the Actor's **Issues** tab in Apify Console or contact the author through the Actor page. Bug reports that include the publication URL are usually fixed within days.

# Actor input Schema

## `publications` (type: `array`):

Publications to scrape. Accepts subdomains (`noahpinion`), Substack hosts (`noahpinion.substack.com`) or full URLs of publications on their own domain (`https://www.astralcodexten.com`). All forms are normalized to the publication's base URL; subdomains that moved to a custom domain are followed automatically. Defaults to `noahpinion` when omitted.

## `maxPostsPerPublication` (type: `integer`):

Stop after this many posts have been collected from a publication (newest first). The archive is read 20 posts per page.

## `sinceDate` (type: `string`):

Optional ISO 8601 date or date-time (for example `2026-01-01` or `2026-01-01T12:00:00Z`). Posts published before this are skipped and pagination stops once the archive passes it.

## `includeBody` (type: `boolean`):

Fetch each post individually to return its full content as plain text (`text`) and HTML (`bodyHtml`). When off, `text` holds only the short preview from the archive and no per-post request is made.

## `includeComments` (type: `boolean`):

Fetch the public comment thread of every post and return it in `comments` (one extra request per post, charged per comment).

## `maxCommentsPerPost` (type: `integer`):

Upper bound on the number of comments (replies included, flattened) returned per post when **Include comments** is on.

## `proxyConfiguration` (type: `object`):

Proxy settings. Apify Proxy is recommended; Substack sits behind Cloudflare and rate-limits bursts from a single IP.

## Actor input object example

```json
{
  "publications": [
    "noahpinion"
  ],
  "maxPostsPerPublication": 10,
  "includeBody": false,
  "includeComments": false,
  "maxCommentsPerPost": 50,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "noahpinion"
    ],
    "maxPostsPerPublication": 10,
    "includeBody": false,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("romajure/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["noahpinion"],
    "maxPostsPerPublication": 10,
    "includeBody": False,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("romajure/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "noahpinion"
  ],
  "maxPostsPerPublication": 10,
  "includeBody": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call romajure/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,romajure/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/7mAOyza5yXdu3kusO/builds/zyHvkqTYD0lML7LNk/openapi.json