📰 Newsletter Scraper — Substack Archives as Clean, LLM-Ready Data

Extract full newsletter content from Substack — entire archives, individual posts and publication metadata — as clean, structured data with Markdown, HTML and plain-text versions plus word and token counts. It's the content layer you need for AI/LLM training sets, competitive research and content archiving, captured straight from the public web with no copy-pasting. Export to JSON/CSV/Excel, run on a schedule, call via API, or connect to Make, Zapier or n8n.

📚 What is the Newsletter Scraper?

It turns any Substack newsletter into a structured dataset. Give it a newsletter homepage URL (or a single post URL) and choose a mode — full archive, single post, or metadata only — and it returns each post with its full content in your chosen format, engagement metrics, images and a built-in token count for LLM cost planning. Beehiiv and Ghost detection is built in, with those platform scrapers in active development; Substack (including custom domains) is fully supported today.

What data does it extract?

• Title, subtitle and author plus the post URL
• Published date and scraped_at timestamp
• Content in three formats — content_markdown, content_html, content_text
• Word count and LLM token count (word_count, token_count)
• Images array with url, alt_text, width, height, caption
• Engagement metadata — likes, comments, shares, views (when available)
• Premium / paywall flag (is_premium), tags, category and language

⬇️ Input

Pick a mode, point it at a newsletter, and choose your content format:

Field	Description
newsletterUrl	Newsletter homepage, e.g. https://platformer.substack.com
scrapeMode	archive (all posts), single-post, or newsletter (metadata only)
postUrl	URL of a specific post (used in single-post mode)
maxPosts	Maximum posts to scrape (0 = unlimited)
outputFormat	markdown, html, text, or all
includeImages	Extract image URLs and details (default on)
includeMetadata	Extract engagement metrics — likes, comments, shares
sinceDate	Only scrape posts published after YYYY-MM-DD
delaySeconds	Polite delay between requests (default 1.0)

Example input

{
  "newsletterUrl": "https://platformer.substack.com",
  "scrapeMode": "archive",
  "outputFormat": "markdown",
  "maxPosts": 50,
  "includeImages": true,
  "includeMetadata": true,
  "delaySeconds": 1.0
}

⬆️ Output

Every post is one clean row (view as a table, or export JSON / CSV / Excel):

{
  "title": "How to get the most out of your 1-on-1s",
  "subtitle": "A tactical guide for managers and ICs",
  "url": "https://platformer.substack.com/p/how-to-get-the-most-out-of-your",
  "author": "Casey Newton",
  "published_date": "2026-01-15T10:00:00",
  "content_markdown": "# How to get the most out of your 1-on-1s\n\nA tactical guide...",
  "content_html": "<article><h1>How to get the most out...</h1></article>",
  "content_text": "How to get the most out of your 1-on-1s...",
  "word_count": 2450,
  "token_count": 3267,
  "images": [
    {
      "url": "https://cdn.substack.com/image/example.jpg",
      "alt_text": "Meeting room illustration",
      "width": 800,
      "height": 600,
      "caption": null
    }
  ],
  "metadata": { "likes": 245, "comments": 18, "shares": 8, "views": null },
  "is_premium": false,
  "tags": ["management", "career"],
  "category": null,
  "language": "en",
  "scraped_at": "2026-06-27T18:00:00"
}

In single-post mode you get one post; in newsletter mode you get publication metadata (name, description, platform, author, subscriber and post counts when available).

💡 Use cases

• 🤖 AI & LLM training: build domain-specific training sets and RAG corpora in clean Markdown, with token counts ready for cost planning.
• 📊 Content research: track topics, posting cadence and writing style across newsletters over time.
• 🔍 Competitive intelligence: monitor competitor newsletters and benchmark engagement (likes, comments, shares).
• 🗄️ Content archiving: back up your own newsletter or preserve historical posts in a searchable, portable format.

❓ FAQ

How do I scrape a newsletter? Paste the newsletter homepage URL, pick a scrapeMode (archive, single-post or newsletter), choose an outputFormat, then Run. You get structured posts with full content, metrics and images.

Do I need an API key or login? No. It reads publicly visible newsletter posts — just paste the URL. Paid/paywalled posts are detected and flagged, but full paid content (which requires authentication) is not retrieved.

Which platforms are supported? Substack is fully supported today, including custom domains. Beehiiv and Ghost are auto-detected with their scrapers in active development.

What content formats can I get? Markdown (LLM-optimized), HTML (original), plain text, or all three at once via outputFormat.

How accurate is the token count? It's an estimate using a standard GPT-style ratio (roughly 1 token ≈ 0.75 words) — great for planning LLM costs. For exact counts, run a tokenizer on the output text.

Can I get only recent posts? Yes — set sinceDate (YYYY-MM-DD) to keep only posts published after that date, and use maxPosts to cap the run.

Does it capture engagement metrics? Yes — with includeMetadata on, likes, comments, shares and views are captured when the platform exposes them.

Why did I only get a limited number of posts? Substack archive pages surface a limited set of posts by default — that's a platform behavior, not a scraper limit.

Can I run it on a schedule or via API? Yes — schedule recurring runs in Apify, call it via the API/SDK, or connect it to Make, Zapier or n8n.

Is scraping newsletters legal? It extracts publicly available content — the same posts anyone can read in a browser. Respect copyright, don't republish without permission, follow the newsletter's terms, and use the data responsibly.

🔗 You might also like

• Spotify Podcast Scraper — podcast episodes & metadata
• Podcast Intelligence Aggregator — multi-source podcast data
• Dev.to Articles Intelligence — developer articles
• YouTube Transcript Scraper — video transcripts for LLMs

---

Keywords: newsletter scraper, Substack scraper, newsletter archive, LLM training data, Markdown content extractor, content research, competitive intelligence, RAG dataset, token count, email newsletter data, content scraping API, AI training dataset, newsletter content extractor, Substack archive scraper.