Substack Scraper & API – Export All Posts, Newsletter Archives & Search (Markdown/CSV)

Scrape public Substack data fast and cheaply: full newsletter archives, individual posts, and Substack-wide keyword search — with post bodies as HTML, Markdown or plain text, engagement metrics (reactions, comments, restacks) and publication metadata (author, description, public subscriber-count text, sections). Works with *.substack.com and custom domains. No login, no browser, no proxy needed.

🚀 Quick start (60 seconds)

1. Click Try for free – the input is prefilled with a working example.
2. Press Start. A small run finishes in seconds and costs fractions of a cent.
3. Open the Output tab (table view) and export CSV, Excel, JSON or connect Google Sheets.

{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 10,
  "bodyFormat": "markdown"
}

Sample output

title	publicationName	publishedAt	reactionCount	commentCount	isPaywalled
How to tell if a startup has a moat	Lenny's Newsletter	2026-10-13	350	2	true
(free post) …	Astral Codex Ten	2026-10-10	412	380	false

Why this Substack scraper / Substack API alternative?

• 3 input types in one run – publication URLs / bare subdomains (astralcodexten), post URLs, and search keywords. Results are de-duplicated across inputs.
• Markdown & plain-text bodies – ready for LLMs, RAG pipelines and NLP, not just raw HTML.
• Paywall-safe – paywalled posts are returned with metadata and isPaywalled: true; their body is never collected. Option to skip them entirely.
• Privacy by design – never collects subscriber emails, support emails or contact data. Subscribe forms are stripped from bodies.
• Cost controls – per-publication cap, per-keyword cap, global maxPostsTotal, date range, post-type filter. Metadata-only mode costs half.
• Polite & reliable – Substack's public JSON API with HTML fallback, rate limiting (maxRequestsPerMinute), retries with backoff on 429/5xx, low default concurrency.
• Clean, flat, consistent schema – same fields for every source, plus ready-made dataset views (Overview, Content, Publications).

Use cases – export Substack posts, research & AI datasets

• Content & competitor research – track what top newsletters in your niche publish, how often, and what gets engagement.
• Newsletter sponsorship research – shortlist publications by topic and public subscriber-count tier.
• AI / LLM datasets – export free long-form posts as Markdown for RAG, summarization or fine-tuning (respect authors' rights).
• Media monitoring – schedule daily keyword searches ("climate policy", "OpenAI") and pipe new posts to Slack, Sheets or webhooks.
• Academic & data journalism – build archives of newsletter discourse with dates, tags and engagement.

Input

Field	Type	Default	Description
publications	string[]	–	Newsletter URL, custom domain, or bare subdomain
postUrls	string[]	–	Individual post URLs (…/p/slug)
searchKeywords	string[]	–	Substack-wide post search
maxPostsPerPublication	int	50	0 = full archive
maxSearchResultsPerKeyword	int	20	Up to 1000
maxPostsTotal	int	0	Global cap (0 = none)
dateFrom / dateTo	date	–	YYYY-MM-DD
postTypes	enum[]	all	newsletter, podcast, thread
onlyFree	bool	false	Skip paywalled posts
bodyFormat	enum	none	none, html, markdown, text, all
includeReactions	bool	true	Per-emoji reaction breakdown
includePublicationMetadata	bool	true	Nested publication object
maxConcurrency / maxRequestsPerMinute	int	4 / 90	Politeness
proxyConfiguration	object	off	Only if you hit 429s

Example input

{
  "publications": ["https://www.lennysnewsletter.com", "astralcodexten", "https://newsletter.pragmaticengineer.com"],
  "searchKeywords": ["artificial intelligence"],
  "maxPostsPerPublication": 20,
  "maxSearchResultsPerKeyword": 20,
  "dateFrom": "2025-01-01",
  "bodyFormat": "markdown"
}

Output

One dataset item per post. Example (trimmed):

{
  "postId": 219482066,
  "title": "How to tell if a startup has a moat",
  "subtitle": "A framework for assessing whether the company you're considering joining is built to last",
  "url": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
  "publishedAt": "2026-10-13T12:45:41.160Z",
  "type": "newsletter",
  "audience": "only_paid",
  "isPaywalled": true,
  "wordCount": 4153,
  "excerpt": "Hey there, I'm Lenny. Each week, I share deeply researched product, growth, and career advice...",
  "tags": ["Career"],
  "authors": [{ "name": "Dan Hockenmaier", "handle": "danhock", "bio": "Strategy, growth, and marketplaces.", "photoUrl": "https://substackcdn.com/..." }],
  "authorName": "Dan Hockenmaier",
  "reactionCount": 350,
  "commentCount": 2,
  "restacks": 18,
  "reactions": { "❤": 350 },
  "bodyMarkdown": null,
  "bodyNote": "Paywalled post: body not collected",
  "publicationName": "Lenny's Newsletter",
  "publication": {
    "name": "Lenny's Newsletter",
    "subdomain": "lenny",
    "customDomain": "www.lennysnewsletter.com",
    "description": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.",
    "authorName": "Lenny Rachitsky",
    "subscriberCountText": "Millions of subscribers",
    "subscriberCountApprox": "1.2M+",
    "paidSubscriberText": "Tens of thousands of paid subscribers",
    "categories": ["..."],
    "paymentsEnabled": true
  },
  "source": "publication",
  "sourceInput": "https://www.lennysnewsletter.com",
  "scrapedAt": "2026-10-21T19:26:38.210Z"
}

Free posts additionally carry bodyHtml / bodyMarkdown / bodyText depending on bodyFormat. Download as JSON, CSV, Excel, XML or via API.

⭐ Unique: Monitor mode, CSV-ready output & webhook summary

Track a list of newsletters daily: monitorMode: "onlyNew" returns only posts published since the last run (old posts aren't output or charged). "changes" also re-emits posts whose reactions/comments/restacks changed – great for engagement tracking.

• Monitor mode (monitorMode: all | changes | onlyNew, plus monitorKey) – cross-run dedupe and field-level diffs (changeType, changedFields, changes, firstSeenAt). Skipped items cost $0.
• Flat / CSV-ready output (outputFormat: "flat") – one column per field, perfect for Excel and Google Sheets.
• Run summary – a free SUMMARY record in the run's key-value store (also linked in the Output tab) and optionally POSTed to summaryWebhookUrl.
• Changes dataset view – see what's new/changed at a glance.

💰 Pricing (pay per result)

Event	Price
Run start	$0.001
Post (metadata)	$0.75 / 1,000
Post with full body (HTML/Markdown/text)	$1.50 / 1,000

That's ≥25% below the typical Substack Actors in the Store ($1.00–$5.00 / 1,000). Platform usage is included.

🎁 Sweetener: in Monitor mode you only pay for new/changed items, and duplicates within a run are never charged. Run summary + webhook are free.

API example

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('YOUR_USERNAME/substack-scraper').call({
  publications: ['astralcodexten'], maxPostsPerPublication: 100, bodyFormat: 'markdown',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

FAQ

Can it scrape paywalled posts? No. It returns their public metadata (title, excerpt, word count, engagement) and marks isPaywalled: true. It never bypasses paywalls or uses logins.

Does it collect emails or personal data? No subscriber lists, emails or contact details are collected. Only public author bylines (name, handle, bio, photo) shown on each post.

Custom domains? Yes – any Substack-hosted domain. Sites that left Substack (e.g. moved to Ghost) are reported in the log and skipped.

Subscriber counts? Substack publicly shows only rounded text (e.g. "1.2M+", "Tens of thousands of paid subscribers"); we return exactly that when shown.

Comments? Comment and reply counts are included. Full comment text is intentionally not collected (commenter personal data).

Is scraping Substack legal? This actor reads only publicly available pages/APIs that Substack serves to any visitor. You are responsible for complying with Substack's Terms, copyright and data-protection laws (GDPR/CCPA) in how you use the data.

🔌 Integrations & API

Works with Zapier, Make, n8n, Google Sheets, Slack, Airbyte, LangChain/LlamaIndex and the Apify MCP server (AI agents).

JavaScript

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('scrapeworkshq/substack-scraper').call({"publications": ["https://www.lennysnewsletter.com"], "maxPostsPerPublication": 10, "bodyFormat": "markdown"});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Python

from apify_client import ApifyClient
client = ApifyClient('<APIFY_TOKEN>')
run = client.actor('scrapeworkshq/substack-scraper').call(run_input={"publications": ["https://www.lennysnewsletter.com"], "maxPostsPerPublication": 10, "bodyFormat": "markdown"})
items = client.dataset(run['defaultDatasetId']).list_items().items

cURL (sync, returns items)

curl -X POST 'https://api.apify.com/v2/acts/scrapeworkshq~substack-scraper/run-sync-get-dataset-items?token=<APIFY_TOKEN>&format=csv' \
  -H 'content-type: application/json' -d '{"publications": ["https://www.lennysnewsletter.com"], "maxPostsPerPublication": 10, "bodyFormat": "markdown"}'

❓ More FAQ

Does it get paywalled content?
No. Paywalled posts are returned with metadata and isPaywalled: true, never their body.

Is Markdown good for LLMs/RAG?
Yes – choose Markdown as Post body; it is clean, heading-preserving text ready for chunking.

How does Monitor mode work?
The Actor remembers the IDs and key fields of items it returned before (in a named key-value store in your account, keyed by monitorKey). Next run, each item gets changeType = new / changed / unchanged plus a changes diff. In Only new / Only new or changed modes, the rest is skipped and not charged.

How do I get a clean CSV / Excel / Google Sheet?
Set Output format → Flat / CSV-ready. Nested objects become dot.columns and lists are joined with |, so every export is one row per item with no JSON blobs.

Can I get notified when a run finishes?
Set Summary webhook URL (Zapier/Make/n8n catch hook, Slack workflow, your API). You receive one compact JSON: counts, new/changed, price drops/rises, dataset ID. Or use Apify's native webhooks/integrations.

How do I limit cost?
Use the max-results inputs and Apify's Maximum cost per run. The Actor stops cleanly when the limit is reached.

Found a bug or need a field? Open an issue on the Issues tab – we usually reply within 24 h.

Changelog

• 2026-10-21 – Monitor mode (new/changed only, free skips), flat CSV-ready output, SUMMARY record + webhook, Changes view, lower prices.
• 1.0 – Initial release: archives, post URLs, keyword search, HTML/Markdown/text bodies, publication metadata, pay-per-event.