# Substack Newsletter Scraper & Author Extractor (`pure_matai/substack-newsletter-scraper`) Actor
Paste Substack URLs to extract structured newsletter data. Export posts, author profiles, and publication details to CSV, Excel, JSON, or XML. Process 1,000 posts for $0.50. No coding or browser needed. Ideal for newsletter research, PR monitoring, sponsorship research, competitive intelligence.
- **URL**: https://apify.com/pure\_matai/substack-newsletter-scraper.md
- **Developed by:** [AzulGanzes Technologies](https://apify.com/pure_matai) (community)
- **Categories:** Developer tools, AI, News
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $0.50 / 1,000 per exported posts
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Substack Newsletter Scraper & Author Extractor
**Turn 3 hours of manual newsletter monitoring into 60 seconds. 1,000 posts for $0.50.**
---
---
### Why You Need This
A PR agency monitoring 50 newsletters manually spends **3 hours a week** reading, copying, and pasting. At a $50/hr billing rate, that's **$125/week** in lost billable time.
This tool does the same job in **60 seconds** for **$0.50**.
**No browser. No manual reading. No coding.** Paste your Substack URLs, click Start, and download a structured dataset ready for your CRM, newsletter research, or competitive analysis.
**Public content only. No login required. No paywall bypass. Robots.txt respected.**
---
### What You Get
| Export Formats | Output Fields |
|---|---|
| JSON, CSV, Excel (XLSX), XML | 21 fields per post: newsletter metadata, post details, engagement stats, full content, author profile, media links |
Sample output (CSV):
```csv
newsletterName,newsletterUrl,title,publishedAt,authorName,wordCount,likeCount
Lenny's Newsletter,https://lenny.substack.com,"How to run effective brainstorming sessions",2026-07-10,Lenny Rachitsky,2400,847
````
***
### How to Run It
1. **Click Start** — Lenny's Newsletter is pre-filled. First 10 posts are free. No typing required.
2. **Configure filters** — date range, keyword match, post limit (optional).
3. **Results appear** — in under 60 seconds.
4. **Download your dataset** — CSV, Excel, JSON, or XML. Ready for your CRM.
The tool checks `robots.txt` before every crawl, respects rate limits, and never accesses subscriber-only content.
***
### Pricing
| What You Pay | Price | Example Run |
|---|---|---|
| Per post exported | $0.00050 | **1,000 posts = $0.50** |
| Full content fetch | +$0.00030/post | **1,000 posts with article text = $0.80** |
| Free preview | First 10 posts | **$0.00** |
***
### For Everyone Else
| You're a... | Here's what this tool does for you |
|---|---|
| **Newsletter Sponsor** | Export engagement stats (likes, comments, restacks) across 20 candidates. Sort by audience activity. Know exactly which newsletter to sponsor. |
| **VC or Investor** | Feed 30 thought-leader newsletters through keyword filters. Get a structured deal-flow intelligence feed — not a flooded inbox. |
| **Recruiter** | Identify domain experts by newsletter. Export author bios and public profiles for direct outreach. |
| **Audience Research Team** | Enrich B2B contact databases with newsletter authorship signals. Every writer is a potential buyer, partner, or influencer. |
***
### Example Use Cases
**Weekly PR Coverage Report**
```
newsletterUrls: ["client1.substack.com", "client2.substack.com"]
keywordFilter: "ClientBrandName"
includeContent: false
maxPosts: 0
```
Exports every mention of your client. Run weekly. ~$0.30/run.
**Newsletter Sponsorship Audit**
```
newsletterUrls: ["candidate1.substack.com", "candidate2.substack.com"]
includeContent: false
maxPosts: 100
```
Compare engagement across candidates. ~$0.10 per newsletter.
**VC Deal Flow Monitor**
```
newsletterUrls: ["analyst1.substack.com", "analyst2.substack.com"]
keywordFilter: "fundraising|funding|series A"
dateFrom: "2026-01-01"
```
Track funding announcements. ~$0.50/run.
***
### Screenshots
***
### Technical Details
The tool uses Substack's public RSS feeds and JSON API — no browser automation, no login, no reverse engineering.
| Component | Responsibility |
|---|---|
| `src/fetcher.py` | RSS parsing, JSON API, robots.txt enforcement, rate-limit handling |
| `src/parser.py` | Extracts 21 fields from raw data |
| `src/exporter.py` | Outputs JSON, CSV, XLSX, XML |
| `src/utils.py` | URL validation, domain extraction |
**Depends on:** Python 3.11+, `apify`, `httpx`, `beautifulsoup4`, `lxml`, `openpyxl`
# Actor input Schema
## `newsletterUrls` (type: `array`):
One or more Substack newsletter URLs (e.g. https://lenny.substack.com). Custom domains pointing to Substack are supported.
## `maxPosts` (type: `integer`):
Maximum number of posts to export per newsletter. Default is 10 for the free preview. Set to 0 to fetch all available.
## `includeContent` (type: `boolean`):
Fetch full article HTML and plain text. Disable for faster, cheaper metadata-only runs.
## `dateFrom` (type: `string`):
Only include posts published on or after this date (YYYY-MM-DD). Leave empty for no filter.
## `dateTo` (type: `string`):
Only include posts published on or before this date (YYYY-MM-DD). Leave empty for no filter.
## `keywordFilter` (type: `string`):
Case-insensitive substring match on post title. Only matching posts are returned.
## `includeAuthors` (type: `boolean`):
Fetch author profile data (name, bio, avatar) for each newsletter.
## `proxy` (type: `object`):
Proxy settings for large runs. Apify proxy recommended for runs over 1,000 posts.
## Actor input object example
```json
{
"newsletterUrls": [
"https://lenny.substack.com"
],
"maxPosts": 10,
"includeContent": true,
"includeAuthors": true,
"proxy": {
"useApifyProxy": true
}
}
```
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"newsletterUrls": [
"https://lenny.substack.com"
],
"proxy": {
"useApifyProxy": true
}
};
// Run the Actor and wait for it to finish
const run = await client.actor("pure_matai/substack-newsletter-scraper").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"newsletterUrls": ["https://lenny.substack.com"],
"proxy": { "useApifyProxy": True },
}
# Run the Actor and wait for it to finish
run = client.actor("pure_matai/substack-newsletter-scraper").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"newsletterUrls": [
"https://lenny.substack.com"
],
"proxy": {
"useApifyProxy": true
}
}' |
apify call pure_matai/substack-newsletter-scraper --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=pure_matai/substack-newsletter-scraper",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Substack Newsletter Scraper & Author Extractor",
"description": "Paste Substack URLs to extract structured newsletter data. Export posts, author profiles, and publication details to CSV, Excel, JSON, or XML. Process 1,000 posts for $0.50. No coding or browser needed. Ideal for newsletter research, PR monitoring, sponsorship research, competitive intelligence.",
"version": "0.1",
"x-build-id": "RqWRBwgBVN9b87Ys6"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/pure_matai~substack-newsletter-scraper/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-pure_matai-substack-newsletter-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/pure_matai~substack-newsletter-scraper/runs": {
"post": {
"operationId": "runs-sync-pure_matai-substack-newsletter-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/pure_matai~substack-newsletter-scraper/run-sync": {
"post": {
"operationId": "run-sync-pure_matai-substack-newsletter-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"newsletterUrls"
],
"properties": {
"newsletterUrls": {
"title": "Newsletter URLs",
"type": "array",
"description": "One or more Substack newsletter URLs (e.g. https://lenny.substack.com). Custom domains pointing to Substack are supported.",
"items": {
"type": "string"
}
},
"maxPosts": {
"title": "Max posts per newsletter",
"minimum": 0,
"type": "integer",
"description": "Maximum number of posts to export per newsletter. Default is 10 for the free preview. Set to 0 to fetch all available.",
"default": 10
},
"includeContent": {
"title": "Include full article content",
"type": "boolean",
"description": "Fetch full article HTML and plain text. Disable for faster, cheaper metadata-only runs.",
"default": true
},
"dateFrom": {
"title": "Posts from date",
"pattern": "^\\d{4}-\\d{2}-\\d{2}$",
"type": "string",
"description": "Only include posts published on or after this date (YYYY-MM-DD). Leave empty for no filter."
},
"dateTo": {
"title": "Posts to date",
"pattern": "^\\d{4}-\\d{2}-\\d{2}$",
"type": "string",
"description": "Only include posts published on or before this date (YYYY-MM-DD). Leave empty for no filter."
},
"keywordFilter": {
"title": "Keyword filter",
"type": "string",
"description": "Case-insensitive substring match on post title. Only matching posts are returned."
},
"includeAuthors": {
"title": "Include author profiles",
"type": "boolean",
"description": "Fetch author profile data (name, bio, avatar) for each newsletter.",
"default": true
},
"proxy": {
"title": "Proxy configuration",
"type": "object",
"description": "Proxy settings for large runs. Apify proxy recommended for runs over 1,000 posts.",
"properties": {
"useApifyProxy": {
"title": "Use Apify proxy",
"type": "boolean",
"description": "Route requests through Apify proxy to avoid rate limiting."
}
},
"default": {
"useApifyProxy": true
}
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```