Substack Newsletter, Archive & Comments Scraper avatar

Substack Newsletter, Archive & Comments Scraper

Pricing

from $1.20 / 1,000 substack post returneds

Go to Apify Store
Substack Newsletter, Archive & Comments Scraper

Substack Newsletter, Archive & Comments Scraper

Extract Substack newsletter posts, full clean text/Markdown content, engagement metrics, and comments from any publication or custom domain using Substack's cursor-paginated JSON API.

Pricing

from $1.20 / 1,000 substack post returneds

Rating

0.0

(0)

Developer

Pulse Extract

Pulse Extract

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

3 days ago

Last modified

Categories

Features

Limited permissions

Share

Extract complete Substack newsletter archives, full clean article text, RAG-ready Markdown, engagement metrics (reactionCount, commentCount, restacks), and post comments from any Substack publication, custom domain (like lennysnewsletter.com), single-post URL, or global keyword search.

Unlike legacy scrapers breaking with HTTP 400 on Substack's retired offset= parameter, this Actor uses Substack's 2026 nextCursor pagination API — running purely over fast HTTP/2 with zero browser and zero proxy costs.


Key Features

  • Works Across All Substack Formats: Custom domains (https://www.lennysnewsletter.com), *.substack.com subdomains, bare publication handles (lennysnewsletter), single post URLs (/p/slug), and global keyword searches (searchQueries).
  • 2026 Cursor Pagination: Paginates deep archives smoothly via nextCursor without HTTP 400 offset errors.
  • RAG & AI-Ready Markdown: Strips subscribe/widget clutter cleanly and converts headings, lists, blockquotes, and code blocks into structured contentMarkdown alongside plain contentText.
  • Optional Comment Extraction: Enable includeComments: true to fetch reader comments (commentId, authorName, date, body, reactionCount).
  • Pre-Push Filtering: Filter by audienceFilter (everyone vs only_paid), publishedWithinDays, and keywords before records are pushed and billed.

Example Input

{
"publications": [
"https://www.lennysnewsletter.com",
"https://newsletter.pragmaticengineer.com"
],
"searchQueries": [],
"sortBy": "new",
"maxPostsPerSource": 20,
"includeContent": true,
"includeComments": false,
"maxCommentsPerPost": 20,
"keywords": [],
"audienceFilter": "all",
"publishedWithinDays": 0
}

Example Output Item

{
"sourceInput": "https://www.lennysnewsletter.com",
"publicationDomain": "www.lennysnewsletter.com",
"publicationName": "www.lennysnewsletter.com",
"postId": 224972682,
"slug": "35-premium-tools-free-for-a-yearnow",
"title": "35 premium tools free for a year",
"subtitle": "...",
"postType": "newsletter",
"audience": "everyone",
"isPaywalled": false,
"publishedAt": "2026-11-17T18:02:02.477Z",
"authors": ["Lenny Rachitsky"],
"tags": [],
"wordCount": 1752,
"reactionCount": 208,
"commentCount": 12,
"restacks": 15,
"canonicalUrl": "https://www.lennysnewsletter.com/p/35-premium-tools-free-for-a-yearnow",
"coverImageUrl": "https://substackcdn.com/...",
"contentText": "...",
"contentMarkdown": "## ...",
"scrapedAt": "2026-11-19T03:25:00+00:00"
}

Pricing & Billing Transparency

This Actor uses Pay-Per-Event (PPE) billing:

  • Actor Start: $0.00005 per run.
  • Per Post Returned (post-scraped): $0.0012 per post ($1.20 per 1,000 posts).
  • Optional Comments (comment-scraped): $0.0004 per extracted comment (only charged when includeComments is true).

Use Cases & Integrations

  • AI Knowledge Bases & Competitive Research: Ingest full contentMarkdown archives of top industry newsletters into vector databases via n8n, Make, or LangChain.
  • Trend & Audience Monitoring: Run weekly with publishedWithinDays: 7 and includeComments: true to spot high-engagement topics and reader pain points.
  • AI Agents & MCP: Connect through Apify's MCP server so Claude or Cursor can query newsletter insights cleanly.

FAQ

Can this bypass paid subscriber paywalls?
No. It extracts all public metadata and free preview/article bodies returned by Substack's public API; paywalled posts report isPaywalled: true and any public preview text.