# Substack Intelligence Scraper - Posts & Comments (`webdata_labs/substack-intelligence-scraper`) Actor

\[$2.00 / 1K] Scrape public Substack posts, clean text, Markdown, comments, authors, and publication metadata. Monitor only new posts across custom domains.

- **URL**: https://apify.com/webdata\_labs/substack-intelligence-scraper.md
- **Developed by:** [WebData Labs](https://apify.com/webdata_labs) (community)
- **Categories:** Social media, News, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 substack posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### 🔎 What does Substack Intelligence Scraper do?

**Substack Intelligence Scraper** turns public [Substack](https://substack.com/) publications into structured post, comment, author, and newsletter datasets. It works with `*.substack.com` publications, custom domains, archive pages, and individual post URLs.

Use it as a fast Substack API alternative for competitive research, newsletter monitoring, content analysis, sponsorship research, and AI-ready corpora. The Actor uses Substack's public HTTP endpoints and server-rendered data, so it does not need a browser or customer-managed proxy settings.

Key capabilities include:

- Complete public archive pagination with `0` for unlimited depth
- Public post HTML, normalized plain text, and Markdown
- Nested comment threads plus optional analytics-friendly comment rows
- Publication, audience, author, tag, podcast, reaction, and restack metadata
- Date, post type, paywall, include-keyword, and exclude-keyword filters
- Direct collection from a single `/p/...` post URL
- Persistent `onlyNew` monitoring for scheduled runs
- Per-source and per-post error rows, so one bad URL does not discard a batch

### Ready-to-run examples

- [Monitor new Substack newsletter posts](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/monitor-substack-newsletter-posts)
- [Export Substack comments and replies](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/export-substack-comments)
- [Build a Substack RAG dataset](https://apify.com/webdata_labs/substack-intelligence-scraper/examples/build-substack-rag-dataset)

### 👥 Who is this Substack scraper for?

- Newsletter teams monitoring competitors and topic coverage
- Researchers comparing publishing cadence and reader engagement
- Sponsorship and partnership teams evaluating public publications
- Data journalists studying newsletter ecosystems
- AI teams preparing searchable text or RAG datasets
- Creators auditing their own public archives

### ⚙️ How to scrape Substack newsletters

1. Add one or more newsletter homepages, custom domains, archive pages, or post URLs.
2. Keep the safe default of 50 posts per publication or use `0` for a complete archive.
3. Choose whether to fetch public content and select HTML, text, Markdown, or all formats.
4. Optionally enable comments and select nested, separate-row, or combined output.
5. Apply date, content-type, paywall, or keyword filters.
6. Start the Actor and open the Output tab.
7. Download JSON, CSV, Excel, XML, RSS, or connect through the Apify API.

Apify can run the same configuration on a schedule, send a webhook when it finishes, and connect the dataset to Make, Zapier, Google Sheets, cloud storage, or your own application.

### 📥 Input

See the Input tab for every option. A typical run looks like this:

```json
{
    "urls": ["https://example.substack.com", "https://newsletter.example.com/p/a-public-post"],
    "maxPostsPerNewsletter": 100,
    "includeContent": true,
    "contentFormat": "all",
    "includeComments": true,
    "commentOutput": "both",
    "includePublicationInfo": true,
    "publicationOutput": "both",
    "contentType": "all",
    "onlyFree": false,
    "includeKeywords": ["AI", "product strategy"],
    "onlyNew": false
}
```

`maxPostsPerNewsletter` limits matching posts, not archive pages scanned. Keyword matching is case-insensitive and checks titles, subtitles, descriptions, public preview text, and tags.

### 📤 What Substack data can be extracted?

| Data              | Important fields                                                        |
| ----------------- | ----------------------------------------------------------------------- |
| Post identity     | `postId`, `title`, `subtitle`, `slug`, `url`                            |
| Timing and access | `publishedAt`, `updatedAt`, `postType`, `audience`, `isPaid`            |
| Public content    | `bodyHtml`, `bodyText`, `bodyMarkdown`, `truncatedBodyText`             |
| Engagement        | `reactionCount`, `reactions`, `commentCount`, `restacks`                |
| Discovery         | `tags`, `language`, `coverImage`, `description`, `wordCount`            |
| Authors           | `authors`, `authorName`, `authorHandle`, public bios and photos         |
| Publication       | name, URL, subscriber signals, branding, benefits, paid-plan status     |
| Comments          | body, date, author, reactions, pinned status, reply depth and parent ID |

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. The Output tab includes dedicated views for posts, comments, publications, and all records.

Example post row:

```json
{
    "itemType": "post",
    "postId": 123456789,
    "title": "A practical guide to product strategy",
    "url": "https://example.substack.com/p/product-strategy",
    "publishedAt": "2026-08-01T12:00:00.000Z",
    "postType": "newsletter",
    "isPaid": false,
    "wordCount": 2450,
    "tags": ["Product", "Strategy"],
    "reactionCount": 120,
    "commentCount": 18,
    "bodyText": "Public article content...",
    "bodyMarkdown": "## Public article content...",
    "publicationName": "Example Newsletter",
    "scrapedAt": "2026-08-09T12:00:00.000Z"
}
```

### 💵 How much does it cost to scrape Substack?

The recommended Store pricing is pay per event with no start fee: **$2.00 per 1,000 posts** and **$0.50 per 1,000 comments**, with lower prices for higher Apify subscription tiers. Publication and recoverable error rows are free. Comments are charged once when collected, whether you choose nested, separate, or combined comment output.

For cost-efficient runs:

- Start with `includeContent: false` to survey metadata quickly.
- Keep comments disabled unless reader discussion is required.
- Use a date window for recurring research.
- Keep the default 50-post cap for the first run.

### 🔁 How to monitor new Substack posts

Enable `onlyNew` and run the same input on an Apify schedule. The Actor stores emitted post IDs in a configuration-specific named key-value store. Later runs suppress previously delivered posts while preserving separate state for different URL and filter combinations.

For RSS delivery, request the last task dataset in RSS format and map `title`, `url`, `description`, and `publishedAt` to feed fields.

### 🛠️ Substack scraper API

Run the Actor from Node.js with `apify-client`:

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('webdata_labs/substack-intelligence-scraper').call({
    urls: ['https://on.substack.com'],
    maxPostsPerNewsletter: 20,
    contentFormat: 'text',
    onlyNew: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

The same input works through the REST API, Apify CLI, webhooks, schedules, and MCP integrations.

### ⚠️ Limits and responsible use

The Actor collects only data available without authentication. It does not log in, bypass access controls, or return paid post bodies. Paid posts can still expose public metadata and preview text. Substack can change undocumented public endpoints or page payloads, so recoverable failures are emitted as structured `error` rows.

Public comments and author profiles can contain personal data. Use the dataset only for a legitimate purpose and comply with Substack's terms, GDPR, CCPA, copyright rules, and other applicable laws. If you are unsure whether your use is legitimate, consult qualified legal counsel.

### ❓ FAQ and support

#### Does it support custom Substack domains?

Yes. Add the full custom-domain URL and the Actor discovers its canonical publication metadata.

#### Can it scrape a single post?

Yes. A URL containing `/p/post-slug` is treated as a direct post and does not expand into the full archive.

#### Why is a paid post body empty?

The Actor intentionally omits paid post bodies. Public metadata, engagement, tags, word count, and preview text can still be returned.

#### How are duplicate inputs handled?

Duplicate source strings, publication archives, and post IDs are deduplicated within each run. `onlyNew` extends post deduplication across scheduled runs with the same monitoring configuration.

For bugs or feature requests, use the Actor's Issues tab and include the public source URL plus the run ID. Do not include private credentials or subscription cookies.

# Actor input Schema

## `urls` (type: `array`):

Public Substack URLs to collect. Both \*.substack.com and custom publication domains are supported.

## `maxPostsPerNewsletter` (type: `integer`):

Maximum matching posts returned for each publication. Use 0 to collect the complete public archive.

## `sortOrder` (type: `string`):

Collect the newest posts first or the publication's top posts first.

## `contentType` (type: `string`):

Return every post type or only newsletters, podcasts, or Substack threads.

## `startDate` (type: `string`):

Optional ISO date or timestamp. A YYYY-MM-DD value includes that entire UTC day.

## `endDate` (type: `string`):

Optional ISO date or timestamp. A YYYY-MM-DD value includes that entire UTC day.

## `onlyFree` (type: `boolean`):

Exclude subscriber-only and paid posts from the result dataset.

## `includeKeywords` (type: `array`):

Keep posts whose title, subtitle, description, preview, or tags contain at least one phrase. Matching is case-insensitive.

## `excludeKeywords` (type: `array`):

Drop posts whose title, subtitle, description, preview, or tags contain any listed phrase.

## `includeContent` (type: `boolean`):

Fetch public post pages to extract the body. Paid post bodies are never returned.

## `contentFormat` (type: `string`):

Choose HTML, clean text, Markdown, or multiple body representations.

## `includeComments` (type: `boolean`):

Fetch public comments, nested replies, public author handles, reactions, and pinned status for each selected post.

## `commentOutput` (type: `string`):

Nest comments inside each post, emit analytics-friendly comment rows, or return both formats.

## `includePublicationInfo` (type: `boolean`):

Attach public publication identity and audience metadata to results.

## `publicationOutput` (type: `string`):

Embed publication fields in posts, emit one publication row, or return both representations.

## `onlyNew` (type: `boolean`):

Remember emitted post IDs for this source and filter combination and suppress them in later runs.

## Actor input object example

```json
{
  "urls": [
    "https://on.substack.com"
  ],
  "maxPostsPerNewsletter": 50,
  "sortOrder": "new",
  "contentType": "all",
  "onlyFree": false,
  "includeKeywords": [],
  "excludeKeywords": [],
  "includeContent": true,
  "contentFormat": "html_and_text",
  "includeComments": false,
  "commentOutput": "nested",
  "includePublicationInfo": true,
  "publicationOutput": "embedded",
  "onlyNew": false
}
```

# Actor output Schema

## `posts` (type: `string`):

Newsletter posts with engagement, authors, public content, and publication context.

## `comments` (type: `string`):

Flat public comment and reply records when separate comment output is enabled.

## `publications` (type: `string`):

Publication-level metadata when separate publication output is enabled.

## `allResults` (type: `string`):

Every post, comment, publication, and recoverable error row.

## `summary` (type: `string`):

Counts and monitoring status for the run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://on.substack.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("webdata_labs/substack-intelligence-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://on.substack.com"] }

# Run the Actor and wait for it to finish
run = client.actor("webdata_labs/substack-intelligence-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://on.substack.com"
  ]
}' |
apify call webdata_labs/substack-intelligence-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,webdata_labs/substack-intelligence-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/KQqfCNsPrPp1O3YXV/builds/vD6EmyEW5kAtcg9DR/openapi.json