# Substack Posts Scraper 📚 (`easyapi/substack-posts-scraper`) Actor

Scrape Substack posts and articles by keywords. Extract comprehensive post data including title, author, publication details, podcast information, reactions, and more. Perfect for content analysis and research.

- **URL**: https://apify.com/easyapi/substack-posts-scraper.md
- **Developed by:** [EasyApi](https://apify.com/easyapi) (community)
- **Categories:** Social media, Integrations, Other
- **Stats:** 278 total users, 30 monthly users, 99.9% runs succeeded, 10 bookmarks
- **User rating**: 2.60 out of 5 stars

## Pricing

from $4.99 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Posts Scraper 📚

Powerful scraper for extracting posts and articles from Substack based on keywords. Get detailed information about posts, publications, and authors with advanced search capabilities.

### Features ✨

- 🔍 Search posts by keywords
- 📊 Extract comprehensive post metadata
- 🎙️ Support for podcast episodes data
- 👥 Get author and publication details
- ❤️ Capture engagement metrics (reactions, comments)
- 🔄 Auto-scrolling for pagination
- ⚡ High-performance with Puppeteer
- 🛡️ Built-in anti-detection mechanisms

### Output Data Structure 📋

The actor provides rich post data including:

- Post title, subtitle, and description
- Publication details
- Author information
- Podcast episode data (if applicable)
- Cover images and media
- Engagement metrics
- Tags and categories
- Publication timestamps
- And much more!

### Usage 💡

Simply provide:

1. Keywords to search for
2. Maximum number of items to scrape (optional)

The actor will automatically:

- Search Substack for your keywords
- Scroll through results
- Extract detailed post information
- Handle pagination
- Export structured JSON data

### Use Cases 🎯

- Content Research
- Market Analysis
- Topic Monitoring
- Audience Engagement Analysis
- Content Aggregation
- Newsletter Analytics
- Competitive Analysis

### Limitations ⚠️

- Respects Substack's terms of service
- Public posts only
- Rate limiting applied for stability

#### Input Example

A full explanation of an input example in JSON.

```json
{
    "keywords": [
        "ai"
    ],
    "maxItems": 50
}
```

#### Output sample

The results will be wrapped into a dataset which you can always find in the **Storage** tab. Here's an excerpt from the data you'd get if you apply the input parameters above:

And here is the same data but in JSON. You can choose in which format to download your data: JSON, JSONL, Excel spreadsheet, HTML table, CSV, or XML.

```json
[
    {
        "keyword": "ai",
        "id": 156491923,
        "editor_v2": false,
        "publication_id": 2270667,
        "title": "New AI image models, free AI music generators, GPT can THINK now, new top AI models, DeepSeek Janus",
        "social_title": null,
        "search_engine_title": null,
        "search_engine_description": null,
        "type": "podcast",
        "slug": "new-ai-image-models-free-ai-music",
        "post_date": "2025-02-04T23:10:38.722Z",
        "audience": "everyone",
        "podcast_duration": 2684.9436,
        "video_upload_id": null,
        "podcast_upload_id": "06e2c81a-16e8-4c32-a936-a1f89d596005",
        "write_comment_permissions": "everyone",
        "should_send_free_preview": false,
        "free_unlock_required": false,
        "default_comment_sort": null,
        "canonical_url": "https://aisearch.substack.com/p/new-ai-image-models-free-ai-music",
        "section_id": null,
        "top_exclusions": [],
        "pins": [],
        "is_section_pinned": false,
        "section_slug": null,
        "section_name": null,
        "reactions": {
            "❤": 0
        },
        "restacked_post_id": null,
        "restacked_post_slug": null,
        "restacked_pub_name": null,
        "restacked_pub_logo_url": null,
        "position": 1,
        "subtitle": "Welcome to the AI Search podcast. Here are the top highlights in AI this week.",
        "cover_image": "https://substack-post-media.s3.amazonaws.com/public/images/1d231857-e4ee-468b-b626-0deb428ee7d6_1400x1400.png",
        "cover_image_is_square": true,
        "cover_image_is_explicit": false,
        "podcast_episode_image_url": "https://substack-post-media.s3.amazonaws.com/public/images/1d231857-e4ee-468b-b626-0deb428ee7d6_1400x1400.png",
        "podcast_episode_image_info": {
            "url": "https://substack-post-media.s3.amazonaws.com/public/images/1d231857-e4ee-468b-b626-0deb428ee7d6_1400x1400.png",
            "isDefaultArt": false,
            "isDefault": false
        },
        "podcast_url": "https://api.substack.com/api/v1/audio/upload/06e2c81a-16e8-4c32-a936-a1f89d596005/src",
        "videoUpload": null,
        "podcastFields": {
            "post_id": 156491923,
            "podcast_episode_number": null,
            "podcast_season_number": null,
            "podcast_episode_type": null,
            "should_syndicate_to_other_feed": null,
            "syndicate_to_section_id": null,
            "hide_from_feed": false,
            "free_podcast_url": null,
            "free_podcast_duration": null
        },
        "podcast_preview_upload_id": null,
        "podcastUpload": {
            "id": "06e2c81a-16e8-4c32-a936-a1f89d596005",
            "name": "news-24.mp3",
            "created_at": "2025-02-04T23:09:13.865Z",
            "uploaded_at": "2025-02-04T23:09:23.872Z",
            "publication_id": 2270667,
            "state": "transcoded",
            "post_id": 156491923,
            "user_id": 191014175,
            "duration": 2684.9436,
            "height": null,
            "width": null,
            "thumbnail_id": 1,
            "preview_start": null,
            "preview_duration": null,
            "media_type": "audio",
            "primary_file_size": "42959560",
            "is_mux": false,
            "mux_asset_id": null,
            "mux_playback_id": null,
            "mux_preview_asset_id": null,
            "mux_preview_playback_id": null,
            "mux_rendition_quality": null,
            "mux_preview_rendition_quality": null,
            "explicit": false,
            "copyright_infringement": null,
            "src_media_upload_id": null,
            "live_stream_id": null,
            "transcription": {
                "media_upload_id": "06e2c81a-16e8-4c32-a936-a1f89d596005",
                "created_at": "2025-02-04T23:10:05.279Z",
                "requested_by": 191014175,
                "status": "transcribed",
                "modal_call_id": "fc-01JK9KMQDG2KRT327JTSR976DS",
                "approved_at": "2025-02-04T23:12:43.876Z",
                "transcript_url": "s3://substack-video/video_upload/post/156491923/06e2c81a-16e8-4c32-a936-a1f89d596005/1738710621/transcription.json",
                "attention_vocab": null,
                "speaker_map": null,
                "captions_map": {
                    "en": {
                        "url": "s3://substack-video/video_upload/post/156491923/06e2c81a-16e8-4c32-a936-a1f89d596005/1738710621/en.vtt",
                        "language": "en",
                        "original": true
                    }
                },
                "cdn_url": "https://substackcdn.com/video_upload/post/156491923/06e2c81a-16e8-4c32-a936-a1f89d596005/1738710621/transcription.json?Expires=1739315665&Key-Pair-Id=APKAIVDA3NPSMPSPESQQ&Signature=kkyvVNWtOpL1VD3Y4n0LkIioiMU3r10tYrBXXlrNC927uzauCnLvzbp4j4-VU5UBwCm5HTayghaSnVeUPWfr6GPD-YTYgeHufNrkwCtmgqinTner3DwKh7z4EsvxbTkH58qXOAR82qLG8MHuu~iSTsXJ5CARuEeGPTW121bHK74poh6QH6jMT3iW-8qqRv4VP4aioSWL8OQyolUxoalTWSiejR6RE9RTxdRUMUbg8pk60GN3nzq3NTRff0qiZtnwJuvh~-A0L4FiTCiNtdFJsHOfYmcieyRydEDj7rHLsgY7yzuFXnsQx2qau9aoF79XAsJ5s4T1EySb~vg7fMTPNQ__",
                "cdn_unaligned_url": "https://substackcdn.com/video_upload/post/156491923/06e2c81a-16e8-4c32-a936-a1f89d596005/1738710621/unaligned_transcription.json?Expires=1739315665&Key-Pair-Id=APKAIVDA3NPSMPSPESQQ&Signature=TpwJtOqaEBWXzOc2orXqEOjPumCJZzfg0wCNb22pdu~T8GQ2PCRcJaPAd6qZZzEPMFLsHbtiEvxC23ZIiD4D6ft94TvlQVYJV~TKKfTy0nC8Ut77ni9FJvzfRQhfYw1tUPdQ2mQEx2s5~TVCKlplUoaceWJ03B55xSURcT9apy4-8X2MjZk57O9Z-almjQ2QtkvxOUxNWvGiM1HN4RGKCPNfu211OpEn1rVMbmU~0WdZ3Sz7QaaXRnJbc3~tQKkev4MWXyq-E8lYa88lpNFj1LJnpF9piIAwfJUtkPym8SXdqiMLFHj0B7pBc4Th15eTUeL1MgdM145boHfnuAlP~w__",
                "signed_captions": [
                    {
                        "language": "en",
                        "url": "https://substackcdn.com/video_upload/post/156491923/06e2c81a-16e8-4c32-a936-a1f89d596005/1738710621/en.vtt?Expires=1739315665&Key-Pair-Id=APKAIVDA3NPSMPSPESQQ&Signature=JpSxDJtOUrhIB698DRiCsap2PZTwPNbQaH1zBYxivUyFGmr8oTHIU80z0SELPqmnsaJmpRTgXjuAKJMbBBMeZlg7FwCNAPxijb9jhS5Oai-SZrDKH4jG21RJwh2TiF0Yg0yp9kF0z8xpK56RzqeS7JCHUJG8iVQmjqSNrrvCVtOGBGN3fBIHlr7Z3RKioxD0grIAatAkDbCwjkc49~~XWa9hd-awKhqByx2o2w6uHlNfuC5ZRRpwnaiX8Ju6rxU9MwW24QQCKCpjbND6S6kaGJ-Z~N90shdh-fD31FymCJ4quq6M1JaEyonZIjcwyUc6FduiKeQi25x-MvUIITtrow__",
                        "original": true
                    }
                ]
            }
        },
        "podcastPreviewUpload": null,
        "voiceover_upload_id": null,
        "voiceoverUpload": null,
        "has_voiceover": false,
        "description": "Welcome to the AI Search podcast. Here are the top highlights in AI this week.",
        "body_json": null,
        "body_html": null,
        "truncated_body_text": "INSANE AI news: OpenAI o3-mini, DeepSeek Janus-Pro, Qwen2.5-Max, Riffusion FUZZ, YuE AI music generator, Doubao 1.5 Pro, Google Daily Listen, Tulu 3",
        "wordcount": 38,
        "postTags": [
            {
                "id": "06ec7467-035f-41d3-aa2a-f2dafd005ba2",
                "publication_id": 2270667,
                "name": "research",
                "slug": "research",
                "hidden": false
            },
            {
                "id": "134583b8-fd61-4289-83f8-2768f0e74637",
                "publication_id": 2270667,
                "name": "machine learning",
                "slug": "machine-learning",
                "hidden": false
            },
            {
                "id": "3b762592-4665-4885-b8c3-5c01a13fbd93",
                "publication_id": 2270667,
                "name": "artificial intelligence",
                "slug": "artificial-intelligence",
                "hidden": false
            },
            {
                "id": "aae204af-3183-4835-9184-ac27d860a342",
                "publication_id": 2270667,
                "name": "science",
                "slug": "science",
                "hidden": false
            },
            {
                "id": "d07870f1-c05d-4f6b-9a7e-c94b7e0ba2c5",
                "publication_id": 2270667,
                "name": "tech",
                "slug": "tech",
                "hidden": false
            },
            {
                "id": "e9d24f2a-02b0-4774-bc85-8b311ff1ab12",
                "publication_id": 2270667,
                "name": "ai",
                "slug": "ai",
                "hidden": false
            }
        ],
        "teaser_post_eligible": true,
        "postCountryBlocks": [],
        "coverImagePalette": {
            "Vibrant": {
                "rgb": [
                    60,
                    180,
                    252
                ],
                "population": 3621
            },
            "DarkVibrant": {
                "rgb": [
                    109,
                    52,
                    68
                ],
                "population": 14
            },
            "LightVibrant": {
                "rgb": [
                    100,
                    196,
                    252
                ],
                "population": 5
            },
            "Muted": {
                "rgb": [
                    164,
                    87,
                    108
                ],
                "population": 5
            },
            "DarkMuted": {
                "rgb": [
                    86,
                    47,
                    61
                ],
                "population": 115
            },
            "LightMuted": {
                "rgb": [
                    218,
                    174,
                    194
                ],
                "population": 99
            }
        },
        "publishedBylines": [
            {
                "id": 191014175,
                "name": "AI Search",
                "handle": "aisearch",
                "previous_name": null,
                "photo_url": "https://substack-post-media.s3.amazonaws.com/public/images/e1ef43b4-d382-41ad-8e0b-86080c6f0b2a_1400x1400.png",
                "bio": "Stay up to date with AI news, tech, & research",
                "profile_set_up_at": "2024-01-18T18:32:07.954Z",
                "publicationUsers": [
                    {
                        "id": 2288577,
                        "user_id": 191014175,
                        "publication_id": 2270667,
                        "role": "admin",
                        "public": true,
                        "is_primary": false,
                        "publication": {
                            "id": 2270667,
                            "name": "AI Search",
                            "subdomain": "aisearch",
                            "custom_domain": null,
                            "custom_domain_optional": false,
                            "hero_text": "Welcome to the AI Search newsletter. We bring you the highlights in AI every week. No fluff, just the interesting stuff. \n\nSubscribe and get a FREE cheat sheet on the top 50 most useful AI tools!",
                            "logo_url": "https://substack-post-media.s3.amazonaws.com/public/images/2905d20a-608c-4fa8-9bdf-7af0c3792e1a_1280x1280.png",
                            "author_id": 191014175,
                            "theme_var_background_pop": "#FF0000",
                            "created_at": "2024-01-18T18:32:47.729Z",
                            "rss_website_url": null,
                            "email_from_name": "AI Search",
                            "copyright": "AI Search",
                            "founding_plan_name": null,
                            "community_enabled": false,
                            "invite_only": false,
                            "payments_state": "disabled",
                            "language": null,
                            "explicit": false,
                            "is_personal_mode": false
                        }
                    }
                ],
                "is_guest": false,
                "bestseller_tier": null
            }
        ],
        "reaction": null,
        "reaction_count": 0,
        "comment_count": 0,
        "child_comment_count": 0,
        "is_geoblocked": false,
        "hasCashtag": false,
        "scrapedAt": "2025-02-10T05:26:49.500Z"
    },
    ...
]
```

### Related Actors

- 📚 [Substack Publications Scraper](https://apify.com/easyapi/substack-publications-scraper) - Extract detailed newsletter information from Substack with comprehensive publication metrics
- 📚 [Substack People Scraper](https://apify.com/easyapi/substack-people-scraper) - Extract comprehensive Substack author and publication data using keywords
- 🔍 [Substack Notes Scraper](https://apify.com/easyapi/substack-notes-scraper) - Extract notes and comments from Substack's search results with engagement metrics
- 📊 [Substack Leaderboard Scraper](https://apify.com/easyapi/substack-leaderboard-scraper) - Get insights about top newsletters including subscriber counts and pricing
- 📄 [Article Content Extractor](https://apify.com/easyapi/article-content-extractor) - Extract clean article content and metadata from any web page
- 🔍 [Medium Posts Search Scraper](https://apify.com/easyapi/medium-posts-search-scraper) - Extract comprehensive article data from Medium's search results
- 📝 [Medium User Posts Scraper](https://apify.com/easyapi/medium-user-posts-scraper) - Extract detailed post data from Medium user profiles
- 🎯 [Medium Publications Search Scraper](https://apify.com/easyapi/medium-publications-search-scraper) - Extract publication details from Medium's search results
- 📰 [Google News Scraper](https://apify.com/easyapi/google-news-scraper) - Collect up to 5000 news articles with flexible search options
- 🔬 [Nature Search Results Scraper](https://apify.com/easyapi/nature-search-results-scraper) - Extract comprehensive research article data from Nature.com
- 📚 [arXiv Search Scraper](https://apify.com/easyapi/arxiv-search-scraper) - Extract research paper data from arXiv search results
- 📚 [PubMed Search Scraper](https://apify.com/easyapi/pubmed-search-scraper) - Scrape research papers and academic articles from PubMed
- 📚 [Goodreads Book Scraper](https://apify.com/easyapi/goodreads-book-scraper) - Extract comprehensive book data from Goodreads search results
- 📚 [Goodreads Review Scraper](https://apify.com/easyapi/goodreads-review-scraper) - Extract detailed book reviews from Goodreads
- 📚 [Wattpad Story Scraper](https://apify.com/easyapi/wattpad-story-scraper) - Scrape stories from Wattpad with advanced filtering options

# Actor input Schema

## `keywords` (type: `array`):

List of keywords to search for on Substack

## `maxItems` (type: `integer`):

Maximum number of posts to scrape (default: 50)

## `proxyConfiguration` (type: `object`):

Optionally, use Apify Proxy or custom HTTP proxy to avoid IP address-based blocking.

## Actor input object example

```json
{
  "keywords": [
    "ai"
  ],
  "maxItems": 50,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("easyapi/substack-posts-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("easyapi/substack-posts-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call easyapi/substack-posts-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,easyapi/substack-posts-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/asNMcstjylwoMkaWc/builds/nUxSeSN8C2VgA8Td7/openapi.json