📰 Substack Newsletter & Post Scraper — Apify Actor

Extract posts, author bios, likes, comments, and engagement metrics from any Substack publication — fast, via direct JSON API. No headless browsers. No overhead.

---

✨ Features

• Direct API access — hits Substack's internal /api/v1/archive and /api/v1/publication endpoints directly
• Zero browser overhead — pure async HTTP via httpx; runs in ~128 MB RAM
• Multi-publication — scrape as many Substack slugs or URLs as you need in one run
• Pagination handled — automatically pages through the full archive
• Clean output — Pydantic-validated schema, ready for downstream pipelines
• Apify-native — batched push_data, structured input/output schemas, Docker-ready

---

📁 Project Structure

.
├── .actor/
│   ├── actor.json          # Actor manifest
│   ├── input_schema.json   # Input configuration schema
│   └── output_schema.json  # Output view / table schema
├── src/
│   ├── __init__.py
│   ├── main.py             # Apify Actor entry point
│   ├── models.py           # Pydantic data models
│   └── substack_client.py  # Async HTTP client + pagination
├── Dockerfile
├── requirements.txt
└── README.md

---

🔌 API Endpoints Used

Purpose	Endpoint
Publication metadata	GET https://{slug}.substack.com/api/v1/publication
Posts archive	GET https://{slug}.substack.com/api/v1/archive?sort=new&limit=50&offset={n}

All requests use standard browser headers — no authentication required.

---

⚙️ Input Parameters

Field	Type	Default	Description
subdomains	string[]	["lennysnewsletter", "thealgorithmicbridge"]	List of Substack slugs or full URLs to scrape
maxPostsPerSubdomain	integer	50	Max posts per publication. Set to 0 for unlimited

Example Input

{
  "subdomains": [
    "lennysnewsletter",
    "thealgorithmicbridge",
    "https://pragmaticengineer.substack.com"
  ],
  "maxPostsPerSubdomain": 100
}

---

📦 Output Schema

Each item in the dataset represents one Substack post:

Field	Type	Description
id	int	Internal Substack post ID
publication_name	str	Human-readable publication name
publication_subdomain	str	Slug (e.g. lennysnewsletter)
publication_url	str	Root URL of the publication
title	str	Article title
subtitle	str / null	Article subtitle
slug	str	URL slug of the post
post_url	str	Direct link to the article
published_at	str / null	ISO 8601 publish date
canonical_url	str / null	Canonical URL
audience	str / null	Access tier (everyone, paid, etc.)
cover_image_url	str / null	Cover image URL
likes_count	int	Number of heart reactions
comments_count	int	Number of comments
word_count	int / null	Approximate word count
reading_time_minutes	int / null	Estimated reading time in minutes
authors	str[]	List of author names
body_excerpt	str / null	Short description / excerpt

Example Output Item

{
  "id": 123456789,
  "publication_name": "Lenny's Newsletter",
  "publication_subdomain": "lennysnewsletter",
  "publication_url": "https://lennysnewsletter.substack.com",
  "title": "How to build a product people love",
  "subtitle": "Lessons from 50 top product leaders",
  "slug": "how-to-build-a-product-people-love",
  "post_url": "https://lennysnewsletter.substack.com/p/how-to-build-a-product-people-love",
  "published_at": "2024-03-15T09:00:00.000Z",
  "canonical_url": "https://lennysnewsletter.substack.com/p/how-to-build-a-product-people-love",
  "audience": "everyone",
  "cover_image_url": "https://substackcdn.com/image/fetch/...",
  "likes_count": 1842,
  "comments_count": 94,
  "word_count": 2300,
  "reading_time_minutes": 10,
  "authors": ["Lenny Rachitsky"],
  "body_excerpt": "After interviewing 50 of the world's best product leaders..."
}

---

🚀 Running Locally

1. Install dependencies

$pip install -r requirements.txt

2. Run as a standalone script

$python3 -m src.main

The actor will use default input (lennysnewsletter + thealgorithmicbridge, 50 posts each) when no Apify input is available.

3. Run with Docker

docker build -t substack-scraper .
docker run substack-scraper

---

🏗️ Architecture Notes

src/models.py

Defines SubstackPost as a Pydantic BaseModel. All fields are strictly typed and optional where the API may omit them. model_dump(mode="json") serialises cleanly for Actor.push_data.

src/substack_client.py

• clean_subdomain() normalises slugs, full URLs, and .substack.com domains
• fetch_substack_posts() is an async generator — memory-efficient for large archives
• Uses a single httpx.AsyncClient per publication (connection pooling)
• Gracefully handles publication metadata errors and empty pages

src/main.py

• Iterates over all requested subdomains sequentially
• Buffers results in batches of 25 before calling Actor.push_data to minimise API round-trips

---

📋 Requirements

• Python 3.11+
• apify >= 2.0.0
• httpx >= 0.27.0
• pydantic >= 2.0.0

---

📄 License

MIT