📰 Substack Newsletter & Full Article Intelligence Scraper

Scrape full article text, clean Markdown, voiceover audio streams (MP3), word counts, reading times, and engagement metrics from any Substack newsletter for AI RAG pipelines, NotebookLM, Notion & Obsidian.

---

⚡ What Does This Actor Do?

Substack hosts the world's highest-quality independent analysis across Tech, AI, Macro Finance, Strategy, and Culture (authors like Lenny Rachitsky, Packy McCormick's Not Boring, Mario Gabriele's The Generalist, Alex Xu's ByteByteGo, Gergely Orosz's The Pragmatic Engineer, etc.).

This Actor provides an automated, lightweight API to extract both archive metadata and full formatted post contents:

• Any Substack Publication: Supports publication handles (thegeneralist, bytebytego, lenny), direct substack URLs (https://stratechery.substack.com), and custom domains (https://www.generalist.com).
• AI-Ready Formats:
  • contentMarkdown: Formatted Markdown with clean headers, bullet points, quotes, and links (ready for Obsidian, Notion, or NotebookLM).
  • contentText: Pure plain text stripped of HTML tags (perfect for LLM context windows and vector embeddings).
  • contentHtml: Raw structured article HTML.
• Voiceover & Podcast Audio: Extracts direct MP3 voiceover / narration links (audioStreamUrl) when available for automated transcription via Whisper.
• Engagement & Social Proof: Reactions count, comments count, estimated reading time, and word counts.
• Aggregated Publication Summary: Includes a publicationSummary record with average word counts, top 3 most popular articles, and audio availability.

---

💡 Top Use Cases

1. NotebookLM & AI Knowledge Base Feeding: Ingest weekly analysis from top thought leaders into Google NotebookLM or custom RAG pipelines.
2. Automated Newsletter Monitoring & Competitive Intelligence: Track strategy updates, market breakdowns, and company teardowns automatically via Make/n8n.
3. Audio Extraction & Whisper Transcriptions: Automatically download audio voiceover streams for podcast repurposing or text summarization.
4. Personal Knowledge Management (PKM): Sync full markdown posts into Obsidian vaults, Readwise, or Notion databases.

---

📥 Input Parameters

Parameter	Type	Default	Description
publications	Array of Strings	["thegeneralist", "bytebytego"]	Substack handles or URLs (e.g. thegeneralist, https://www.generalist.com).
maxArticlesPerPublication	Integer	15	Max recent articles to retrieve per newsletter (1 to 100).
extractFullText	Boolean	true	Fetch complete article body formatted in clean Markdown, Plain Text and HTML.
onlyFreeArticles	Boolean	false	Exclude paid subscriber paywalled stubs.
includeSummary	Boolean	true	Include an aggregate publicationSummary record.

---

📤 Output Format

1. Full Article Record (article)

{
  "recordType": "article",
  "id": "220756708",
  "title": "Pete Florence: When Robots Improvise",
  "subtitle": "As intelligence scales, robots are learning more like humans.",
  "slug": "pete-florence-when-robots-improvise",
  "publication": "thegeneralist",
  "publicationUrl": "https://thegeneralist.substack.com",
  "articleUrl": "https://www.generalist.com/p/pete-florence-when-robots-improvise",
  "postDate": "2026-10-20T12:04:23.312Z",
  "authors": ["Mario Gabriele"],
  "audience": "everyone",
  "isPaidOnly": false,
  "wordCount": 4250,
  "readingTimeMinutes": 19.3,
  "reactionCount": 342,
  "commentCount": 28,
  "hasAudio": true,
  "audioStreamUrl": "https://api.substack.com/api/v1/audio/upload/9e69a2ac-b748-46c2-967f-16fda4d7ff21/src",
  "contentMarkdown": "# Pete Florence: When Robots Improvise\n\nAs intelligence scales, robots are learning...",
  "contentText": "Pete Florence: When Robots Improvise\n\nAs intelligence scales, robots are learning..."
}

2. Aggregated Publication Summary (publicationSummary)

{
  "recordType": "publicationSummary",
  "publication": "thegeneralist",
  "totalArticlesScraped": 15,
  "freeArticlesCount": 14,
  "paidArticlesCount": 1,
  "averageWordCount": 3850,
  "totalReactions": 4120,
  "totalComments": 315,
  "audioArticlesCount": 11,
  "topArticles": [
    {
      "title": "Pete Florence: When Robots Improvise",
      "reactions": 342,
      "wordCount": 4250,
      "url": "https://www.generalist.com/p/pete-florence-when-robots-improvise"
    }
  ]
}

---

💰 Pay-Per-Event Pricing

• $0.0025 per extracted article ($2.50 per 1,000 articles).
• Free Overview Summary: The aggregated publication metrics record is included free of charge.
• Instant Speed: Pure HTTP TLS emulation without heavy browser overhead.