# Substack Scraper – Posts, Engagement & Metadata (`rowfeed/substack-scraper`) Actor

Extract every post from any Substack publication's archive: titles, subtitles, dates, word counts, reactions, comments and URLs. No login, no persondata, full article text opt-in. Clean JSON for researchers, journalists and AI agents.

- **URL**: https://apify.com/rowfeed/substack-scraper.md
- **Developed by:** [Rowfeed](https://apify.com/rowfeed) (community)
- **Categories:** News, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

Pull every post from any **Substack publication's archive** as clean JSON: title, subtitle, publish date, word count, reaction/comment/restack counts and the public post URL, no login and no browser. Counts only — comment and reaction *text* is not included. Built for **researchers and journalists** tracking a beat across newsletters, **RAG/LLM ingestion pipelines** that need a structured feed of a publication's back catalog, **market/media analysts** watching engagement trends across writers, and **dashboards and alerting** that poll a publication for new posts on a schedule.
Plain HTTPS calls against Substack's own archive API (`/api/v1/archive`) with retries and a silent-failure check, so a scheduled run keeps working even when one publication in the list turns out not to be on Substack at all.

### What you get

- **Every post as one row** – id, title, subtitle, slug, canonical URL, publish date, type (newsletter/podcast/thread), audience, word count, reactions, comments, restacks, a short description/preview, and the raw Substack API fields for everything else (cover image, podcast metadata, section, SEO title/description, and more).
- **Any publication, two ways to name it** – a short name (`astralcodexten`), a custom domain, or a full URL; redirects are followed automatically, so `astralcodexten.substack.com` and its real domain `www.astralcodexten.com` each reach the same publication. Pick one form per publication: listing both in the same run fetches and bills it twice (see Pricing).
- **Filters that keep a run's cost predictable** – a per-publication post cap, a `postedAfter` date cutoff, and a whole-word `searchTerm` match on title/subtitle (so searching "NFL" never matches "inflation" the way a naive substring search would).

### No personal data, ever

The Actor never outputs subscriber, contact or author-profile data. Substack's archive API returns a `publishedBylines` object per post (author name, id, profile photo, bio) — this Actor drops that field entirely, along with anything else in the raw response whose field name matches `email`, `phone`, `subscriber`, `address`, `ip_address`, `password` or `token`, before the row is ever written. What's left is publication-level content metadata: post titles, dates and engagement counts, the same thing you'd see browsing the publication's public archive page. That makes the output safe to store, join and hand to a model without a data-processing agreement.

### Sample row

A real row from a default run against `astralcodexten` (`https://www.astralcodexten.com`):

```json
{
  "publication": "astralcodexten",
  "host": "astralcodexten.substack.com",
  "id": 210677400,
  "title": "Your Book Review: The Tale Of Genji",
  "subtitle": "Finalist #8 in the Book Review Contest",
  "slug": "your-book-review-the-tale-of-genji",
  "canonical_url": "https://www.astralcodexten.com/p/your-book-review-the-tale-of-genji",
  "post_date": "2026-09-04T19:34:24.551Z",
  "type": "newsletter",
  "audience": "everyone",
  "wordcount": 9885,
  "reaction_count": 212,
  "comment_count": 106,
  "restacks": 8,
  "description": "Finalist #8 in the Book Review Contest",
  "truncated_body_text": null,
  "cover_image": "https://substackcdn.com/image/fetch/...jpeg",
  "podcast_url": null,
  "podcast_duration": null,
  "section_name": null,
  "scraped_at": "2026-09-10T18:24:12+00:00"
}
```

Every row also carries the rest of Substack's own archive fields (SEO title/description, section, cover-image flags, podcast metadata and so on) — 65 fields per post before any are dropped for privacy. `description` and `truncated_body_text` (a short plain-text preview) are always included; the full article isn't.

### Full article text (opt-in)

Substack's archive listing can include `body_html` (rendered article HTML) and `body_json` (the same content as structured rich-text JSON) — the publication's actual copyrighted words, not just metadata about the post. Both are off by default. Turn on `includeBodyHtml` to add them to every row. Note that Substack's archive endpoint itself only populates full body content for some posts/publications; when it's empty there, this Actor can't invent it either.

### Filters

| Input | Default | What it does |
|---|---|---|
| `publications` | `["astralcodexten", "slowboring", "garymarcus"]` | Publications to scrape: short name, custom domain, or full URL. |
| `maxPostsPerPublication` | `100` | Stop after this many posts per publication (newest first). |
| `postedAfter` | `""` | Keep only posts published on or after this ISO date. |
| `searchTerm` | `""` | Keep only posts whose title or subtitle contains this whole word/phrase (case-insensitive, word-boundary match — not a naive substring). |
| `includeBodyHtml` | `false` | Add `body_html` and `body_json` (full article content) to each row. |

### Pricing

Pay per event, no subscription: **$1 per 1,000 posts**. A default run (3 publications, up to 100 posts each) costs a few cents to a few dimes depending on how many posts each publication actually has. A publication that isn't reachable or isn't on Substack produces a single unbilled error row, so one bad entry among several never wastes budget on the rest — the run finishes and you get the posts from the good ones. If *every* publication in your list fails, the run is marked failed rather than quietly returning nothing, and you are charged for nothing. Error rows are never billed either way.

Listing the same publication twice in two different forms (`example` and `www.example.com`) counts as two publications and is fetched, and billed, twice; the Actor can only tell they are the same after following the redirect. List each publication once.

### Use it from your tools

- **API and SDKs** – call it via the Apify API or the official Python/JavaScript clients: one call to start the run, one to fetch the post list as JSON or CSV.
- **Schedules** – run it daily or weekly inside Apify and pipe newly-published posts into Google Sheets, a webhook or your own storage.
- **n8n, Make and Zapier** – trigger runs and feed extracted posts into a workflow through Apify's integration for each.
- **AI agents and MCP** – eligible for agentic use via Apify's MCP server with pay-per-event pricing, so an agent can call it mid-task and pay only for the posts it actually pulls.

### Details

- **Source**: `https://<publication-host>/api/v1/archive`, Substack's own public archive endpoint, the same one the publication's website uses to render its archive page. No authentication, no proxies, no browser.
- **Not a Substack publication** – if a given domain returns something other than a JSON post list (an HTML page, a 404), the Actor never crashes; it logs an unbilled error row categorized `network`, `rate_limit`, `blocked`, `not_found` or `other` and continues with the rest of `publications`.
- **Reliability**: 429 and 5xx responses are retried with exponential backoff (5 tries); 404 is never retried; a 200 response that isn't actually a post list counts as a failure, not a silent empty success.
- **Run stats**: the `STATS` record in the run's key-value store holds post/error counts and per-category request-error counts.
- **Politeness**: 0.2 s between requests, 30 s timeout, redirects followed, a User-Agent that names the Actor.
- **Not affiliated with Substack.** Built against Substack's own public archive API; no login, no paywall bypass, no subscriber data.

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape: a short name (e.g. "astralcodexten" -> astralcodexten.substack.com), a custom domain, or a full URL. Redirects are followed, so the .substack.com and custom-domain form of the same publication both work.

## `maxPostsPerPublication` (type: `integer`):

Stop fetching a publication's archive after this many posts (newest first). Each row is one `post` event ($1 per 1,000).

## `postedAfter` (type: `string`):

Keep only posts published on or after this ISO date, e.g. 2024-01-01. Empty = no date filter.

## `searchTerm` (type: `string`):

Keep only posts whose title or subtitle contains this word or phrase, matched on whole-word boundaries (case-insensitive) so "NFL" does not match "inflation". Empty = no filter.

## `includeBodyHtml` (type: `boolean`):

Add body\_html (rendered article HTML) and body\_json (the same content as Substack's rich-text JSON) to each row. Off by default: this is the publication's copyrighted full-text content, not just metadata. description and truncated\_body\_text (a short preview) are always included.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten",
    "www.slowboring.com",
    "https://garymarcus.substack.com"
  ],
  "maxPostsPerPublication": 100,
  "postedAfter": "",
  "searchTerm": "",
  "includeBodyHtml": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("rowfeed/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("rowfeed/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call rowfeed/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rowfeed/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bannckzI0ShC7zcmh/builds/2fFb5e2TAZKkqHBi2/openapi.json