Substack Scraper to Markdown — for AI, RAG & LLMs

Turn any Substack newsletter into clean Markdown (or text / HTML) with full metadata and optional ready-made RAG chunks. Works with *.substack.com and custom domains (e.g. lennysnewsletter.com).

• ✅ Clean Markdown: subscribe buttons, share widgets and image wrappers removed — ready for LLMs
• ✅ RAG chunking on paragraph boundaries with section headings and overlap
• ✅ Metadata: title, subtitle, date, authors, tags, word count, reactions, comments, cover image, podcast URL
• ✅ Filters: date, keywords, max posts
• ✅ Monitoring: return only posts published since the last run — schedule it and get a newsletter feed
• ✅ Respects paywalls: paid posts return metadata and a short public teaser only

Use cases

• Build a RAG knowledge base / chatbot from your favourite newsletters
• AI agents that need fresh expert content (product, tech, finance, politics…)
• Newsletter research and analytics: posting frequency, engagement, topics
• Content monitoring: get every new post from a list of newsletters into Slack, Notion, Google Sheets
• Backing up your own Substack as Markdown

Input

Field	Description
publications	Publication URLs, names (astralcodexten) or single post URLs (…/p/slug).
maxPostsPerPublication	Newest first (default 50).
postedAfter	YYYY-MM-DD.
keywords	Match in title, subtitle or tags.
includeContent	Full body of free posts (default on).
contentFormat	markdown (default), text, html.
chunkSize / chunkOverlap	RAG chunks in characters (0 = off).
includePaidPosts	Add paywalled posts as metadata + teaser.
onlyNewPosts / monitorId	Only posts not returned in previous runs.

{
  "publications": ["https://www.lennysnewsletter.com", "astralcodexten"],
  "maxPostsPerPublication": 20,
  "postedAfter": "2026-01-01",
  "chunkSize": 2000
}

Output

{
  "publication": "www.lennysnewsletter.com",
  "postId": 215694124,
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "postDate": "2026-09-29T13:15:57.512Z",
  "audience": "everyone",
  "isPaywalled": false,
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny" }],
  "tags": [],
  "wordcount": 1276,
  "reactionCount": 289,
  "commentCount": 7,
  "contentFormat": "markdown",
  "content": "Two years ago, we held the first-ever Lenny and Friends Summit…",
  "chunks": [{ "index": 0, "heading": null, "text": "…" }]
}

A per-publication report is saved to the SUMMARY record of the run's key-value store.

Pricing

Pay per post: a small fee per run plus a fee per post in the dataset. With monitoring on you pay only for new posts.

Legal

The Actor reads only publicly available posts through Substack's public web endpoints and never extracts paywalled content. Posts remain the copyright of their authors — make sure your use (e.g. internal research, personal knowledge base) complies with applicable law and the authors' terms.

FAQ

Does it work with custom domains? Yes — any Substack publication, e.g. https://www.lennysnewsletter.com.

Can I scrape paid posts? No. Paywalled posts return metadata and the short public teaser only (enable includePaidPosts). The Actor never extracts paywalled content.

How do I build a RAG knowledge base? Set chunkSize to 1500–3000. Each post gets chunks[] with text and section heading — load them straight into your vector database with url and title as metadata.

Can I get only new posts? Enable onlyNewPosts and schedule the Actor; each run returns posts published since the previous run.

Related tools

• Career Sites Jobs Feed (11 ATS)
• Shopify Products & Price Monitor
• Guide: Substack API on GitHub