# Substack Posts Scraper - Newsletter Data (`benthepythondev/substack-posts-scraper`) Actor

Scrape public Substack newsletter posts from one or many publications. Extract titles, authors, dates, full content, images, categories and post URLs.

- **URL**: https://apify.com/benthepythondev/substack-posts-scraper.md
- **Developed by:** [Ben](https://apify.com/benthepythondev) (community)
- **Categories:** Marketing, Business, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## Substack Posts Scraper - Newsletter Content and Publication Data

Turn public **Substack newsletter posts** into structured data without copying articles by hand. Provide one or many publication URLs and receive recent post titles, authors, publication dates, summaries, categories, images, canonical URLs, and full public content when the feed exposes it. The Actor is designed for newsletter monitoring, content research, media intelligence, creator discovery, and scheduled archives.

The scraper reads each publication's public RSS feed, so it does not require a Substack login, browser automation, cookies, or an API key. This low-overhead approach is fast enough for recurring workflows and keeps the default run inexpensive. Download the resulting dataset as JSON, CSV, Excel, XML, or RSS, or connect it to the Apify API, Make, Zapier, n8n, Google Sheets, and your own database.

### What the Actor does

- Accepts Substack home URLs, custom publication domains, or direct feed URLs
- Scrapes multiple publications in one run
- Returns the newest posts from each publication
- Extracts full public HTML and clean plain text when available
- Captures authors, dates, categories, images, summaries, and URLs
- Adds publication-level title, description, and URL to every row
- Deduplicates repeated posts across supplied feeds
- Fails clearly when no public posts are available

### Data extracted

Each result represents one newsletter post and can include:

- `post_id` - public feed identifier or canonical URL
- `title` - post headline
- `url` - direct public post URL
- `published_at` - normalized publication timestamp
- `author` - source-provided author name
- `subtitle` - readable summary or preview
- `content_text` - clean full-text content when enabled and publicly provided
- `content_html` - source HTML when enabled
- `categories` - feed categories or tags
- `image` - featured image or first public article image
- `publication_title` and `publication_description`
- `publication_url`, requested URL, and resolved feed URL
- source and scrape timestamp

The output reflects what the publication makes available in its public feed. Some publishers provide a complete article; others provide only an excerpt. The Actor does not bypass paywalls or fabricate omitted content.

### Input

| Field | Description | Default |
|---|---|---|
| `publicationUrls` | Publication home URLs, custom domains, or direct public feeds | `astralcodexten.substack.com` |
| `maxPostsPerPublication` | Maximum recent posts saved for each publication | `10` |
| `includeContent` | Include public HTML and plain-text content | `true` |

#### Example input

```json
{
  "publicationUrls": [
    "https://astralcodexten.substack.com",
    "https://example.substack.com"
  ],
  "maxPostsPerPublication": 25,
  "includeContent": true
}
````

### Example output

```json
{
  "post_id": "https://astralcodexten.substack.com/p/example-post",
  "title": "Example newsletter post",
  "url": "https://astralcodexten.substack.com/p/example-post",
  "published_at": "2026-07-09T14:00:00+00:00",
  "author": "Scott Alexander",
  "subtitle": "A short public preview of the post...",
  "content_text": "The public article text...",
  "categories": ["Open Thread"],
  "image": "https://substackcdn.com/image/fetch/...",
  "publication_title": "Astral Codex Ten",
  "publication_url": "https://astralcodexten.substack.com",
  "source": "Substack RSS"
}
```

The example demonstrates the field shape. Actual values depend on the public feed at the time of the run.

### Use cases

#### Newsletter and media monitoring

Run a stable publication list every hour or day, store `post_id` as the deduplication key, and notify a researcher when a new URL appears. This works well for competitor newsletters, policy publications, industry analysts, and independent media.

#### Creator and topic research

Combine title, subtitle, content, author, and category fields to study what a newsletter covers over time. Export the data to a notebook or warehouse for topic classification, entity extraction, keyword trends, or citation analysis.

#### Content operations

Feed new public posts into an internal reading queue, editorial dashboard, or knowledge base. Keep the canonical URL attached so readers always return to the original publication.

#### Public archive and change tracking

Create dated snapshots of currently public feed entries. Compare consecutive datasets to identify newly published posts. Respect copyright and retention requirements; structured access does not grant republication rights.

### Pricing

The Actor uses transparent pay-per-event pricing: a very small one-time start event and **$0.003 per result** saved to the default dataset. One result is one newsletter post. The default is limited to ten posts per publication, and `maxPostsPerPublication` provides a predictable spending control for larger runs.

### Reliability and data quality

Public RSS is substantially lighter and more stable than automating a logged-in website. Requests are retried for temporary failures, duplicate post IDs are suppressed, dates are normalized when possible, and a zero-item run fails visibly instead of looking successful. If a publication has disabled or changed its feed, the run log identifies the affected URL while continuing with other publications.

For recurring pipelines, preserve `post_id`, `url`, and `published_at`. Treat missing fields as source omissions. The Actor never guesses author names, categories, or publication dates.

### Responsible use

This Actor collects only content exposed through public publication feeds. It does not bypass subscriptions or paywalls. You are responsible for complying with copyright, privacy, platform terms, and applicable law. Use article text for permitted analysis and internal workflows, and link back to the original publisher rather than republishing content without authorization.

### FAQ

**Do I need a Substack account?**\
No. The Actor reads public publication feeds.

**Does it work with custom Substack domains?**\
Yes, when the custom domain exposes the standard public `/feed` route. You can also paste a direct feed URL.

**Can it scrape paid posts?**\
It returns only the title, preview, or content that the public feed exposes. It does not unlock subscriber-only text.

**Can I scrape several newsletters at once?**\
Yes. Add each publication to `publicationUrls`; the result limit applies to each publication.

**Why is full content missing for some rows?**\
Publication owners choose whether feeds include the complete article or an excerpt.

**How should I deduplicate scheduled runs?**\
Use `post_id`, falling back to `url`, as the stable unique key.

**Can I use the output in Make, Zapier, or n8n?**\
Yes. Use the default dataset API or the built-in Apify integrations.

### You might also like

- [RSS Feed Scraper](https://apify.com/benthepythondev/rss-feed-scraper) for general RSS and Atom feeds
- [Google News Scraper](https://apify.com/benthepythondev/google-news-scraper) for news monitoring
- [Dev.to Articles Scraper](https://apify.com/benthepythondev/devto-articles-scraper) for developer publishing data
- [Website Content Crawler](https://apify.com/apify/website-content-crawler) for full-site crawling

***

**Keywords:** Substack scraper, Substack posts scraper, newsletter scraper, Substack RSS, scrape newsletters, newsletter monitoring, Substack API alternative, creator content data, public newsletter archive, Substack export, newsletter analytics.

# Actor input Schema

## `publicationUrls` (type: `array`):

Substack home URLs, custom publication domains, or direct public feed URLs.

## `maxPostsPerPublication` (type: `integer`):

Maximum recent posts returned from each public feed.

## `includeContent` (type: `boolean`):

Include the full public HTML and a plain-text version when present in the feed.

## Actor input object example

```json
{
  "publicationUrls": [
    "https://astralcodexten.substack.com"
  ],
  "maxPostsPerPublication": 10,
  "includeContent": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publicationUrls": [
        "https://astralcodexten.substack.com"
    ],
    "maxPostsPerPublication": 10,
    "includeContent": true
};

// Run the Actor and wait for it to finish
const run = await client.actor("benthepythondev/substack-posts-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publicationUrls": ["https://astralcodexten.substack.com"],
    "maxPostsPerPublication": 10,
    "includeContent": True,
}

# Run the Actor and wait for it to finish
run = client.actor("benthepythondev/substack-posts-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publicationUrls": [
    "https://astralcodexten.substack.com"
  ],
  "maxPostsPerPublication": 10,
  "includeContent": true
}' |
apify call benthepythondev/substack-posts-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=benthepythondev/substack-posts-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Substack Posts Scraper - Newsletter Data",
        "description": "Scrape public Substack newsletter posts from one or many publications. Extract titles, authors, dates, full content, images, categories and post URLs.",
        "version": "1.0",
        "x-build-id": "oUmaa1YsQNnqJEumg"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/benthepythondev~substack-posts-scraper/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-benthepythondev-substack-posts-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/benthepythondev~substack-posts-scraper/runs": {
            "post": {
                "operationId": "runs-sync-benthepythondev-substack-posts-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/benthepythondev~substack-posts-scraper/run-sync": {
            "post": {
                "operationId": "run-sync-benthepythondev-substack-posts-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "publicationUrls": {
                        "title": "Substack publication URLs",
                        "type": "array",
                        "description": "Substack home URLs, custom publication domains, or direct public feed URLs.",
                        "default": [
                            "https://astralcodexten.substack.com"
                        ],
                        "items": {
                            "type": "string"
                        }
                    },
                    "maxPostsPerPublication": {
                        "title": "Maximum posts per publication",
                        "minimum": 1,
                        "maximum": 100,
                        "type": "integer",
                        "description": "Maximum recent posts returned from each public feed.",
                        "default": 10
                    },
                    "includeContent": {
                        "title": "Include full post content",
                        "type": "boolean",
                        "description": "Include the full public HTML and a plain-text version when present in the feed.",
                        "default": true
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```