Substack Scraper

Substack Scraper

Extract data from any Substack newsletter — posts with full HTML content, comments with nested replies, and publication metadata including subscriber counts. Unlimited archive depth (no 12-post cap), works with both *.substack.com and custom-domain newsletters. Uses Substack's public JSON API — no browser, no proxy.

Scrape full post HTML, engagement metrics, threaded comments, author details and publication subscriber counts from any Substack — by URL or by keyword. Filter by type, date, and paywall status. Export JSON, CSV, Excel.

ParseBird Substack Suite   •  Newsletter intelligence at scale
📰  Substack Scraper ➤ You are here	🏆  Leaderboard Scraper Top bestseller & rising newsletters by category

Copy to your AI assistant

Copy this block into ChatGPT, Claude, Cursor, or any LLM to start using this actor.

Apify Actor: parsebird/substack-scraper — scrapes Substack newsletters via the public JSON API (no browser/proxy). Call: ApifyClient("TOKEN").actor("parsebird/substack-scraper").call(run_input={"urls": ["https://www.lennysnewsletter.com"], "keywords": ["artificial intelligence"], "maxPostsPerNewsletter": 20, "includeContent": true, "includeComments": false, "maxCommentsPerPost": 0, "includePublicationInfo": true, "contentType": "all", "onlyFree": false, "startDate": "2024-01-01", "endDate": "2026-12-31"}). Inputs: urls (array — homepage / custom domain / /archive / .../p/slug post URL), keywords (array — discover newsletters by topic), maxSearchResultsPerKeyword (int, default 20, max 100), maxPostsPerNewsletter (int, default 100, 0 = full archive), includeContent (bool, default true), includeComments (bool, default false), maxCommentsPerPost (int, default 0 = all), includePublicationInfo (bool, default true), contentType (all|newsletter|podcast|thread), onlyFree (bool), startDate/endDate (YYYY-MM-DD). Output: `default` dataset = posts (postId, title, url, publishedAt, updatedAt, postType, isPaid, wordcount, bodyHtml, truncatedBodyText, tags, reactionCount, commentCount, restacks, authorName/Handle, publicationName, subscriberCount, comments[] with nested replies); `publications` dataset = newsletter metadata. API docs: https://docs.apify.com/api/v2  Token: https://console.apify.com/settings/integrations

What does Substack Scraper do?

Give it newsletter URLs, keywords, or both. It walks each newsletter's archive through Substack's public API and returns a structured record per post.

Per post:

• Identity — postId, title, subtitle, slug, canonical url
• Timestamps — publishedAt, updatedAt
• Type & paywall — postType (newsletter / podcast / thread), audience, isPaid
• Content — bodyHtml (full HTML for free posts), truncatedBodyText (preview for every post), wordcount (even for paid)
• Media — coverImage, podcastUrl, podcastDuration, hasVoiceover
• Engagement — reactionCount, commentCount, childCommentCount, restacks
• Author — authorName, authorHandle, authorBio, authorPhotoUrl, authorId
• Publication — publicationName, publicationId, publicationUrl, subscriberCount
• Comments (optional) — full nested threads: id, body, date, editedAt, name, handle, photoUrl, reactionCount, restacks, isAuthor, isPinned, replies[]

Per publication (publications dataset): id, name, subdomain, customDomain, baseUrl, authorName, authorHandle, authorBio, authorPhotoUrl, logoUrl, heroText, language, paymentsEnabled, subscriberCount, monthlyPrice, annualPrice.

Input parameters

Parameter	Type	Default	Description
urls	array	[]	Newsletter URLs — homepage, custom domain, /archive, or a single post URL.
keywords	array	[]	Topics to discover newsletters across public Substack.
maxSearchResultsPerKeyword	integer	20	Matching posts to process per keyword (max 100).
maxPostsPerNewsletter	integer	100	Max posts per newsletter, newest first. 0 = full archive.
includeContent	boolean	true	Include the full HTML body. Disable for metadata-only (faster, cheaper).
includeComments	boolean	false	Fetch comments for each post. One extra API call per post.
maxCommentsPerPost	integer	0	Cap top-level comment threads per post (0 = all). Replies always included.
includePublicationInfo	boolean	true	Emit newsletter metadata to the publications dataset.
contentType	select	all	all, newsletter, podcast, or thread.
onlyFree	boolean	false	Only include free posts.
startDate / endDate	string	—	Only posts within this date range (YYYY-MM-DD).

Output example

{
  "postId": 186226252,
  "title": "How to build AI product sense",
  "subtitle": "The secret is using Cursor for non-technical work",
  "slug": "how-to-build-ai-product-sense",
  "url": "https://www.lennysnewsletter.com/p/how-to-build-ai-product-sense",
  "publishedAt": "2026-02-03T13:45:58.303Z",
  "updatedAt": "2026-05-16T03:04:37.084Z",
  "postType": "newsletter",
  "audience": "everyone",
  "isPaid": false,
  "wordcount": 5867,
  "coverImage": "https://substackcdn.com/image/fetch/...",
  "tags": ["AI"],
  "reactionCount": 298,
  "commentCount": 31,
  "childCommentCount": 15,
  "restacks": 20,
  "hasVoiceover": false,
  "bodyHtml": "<div class=\"body markup\">...</div>",
  "truncatedBodyText": "👋 Hey there, I'm Lenny...",
  "authorName": "Tal Raviv",
  "authorHandle": "talsraviv",
  "publicationName": "Lenny's Newsletter",
  "subscriberCount": "1,200,000",
  "comments": [
    {
      "id": 209331673,
      "body": "This article creates a whole new paradigm for learning...",
      "date": "2026-02-03T15:34:25.318Z",
      "name": "Jack Cohen",
      "handle": "jackcohen10",
      "reactionCount": 9,
      "isAuthor": false,
      "replies": [
        { "id": 209340123, "body": "Thanks Jack!", "name": "Tal Raviv", "isAuthor": true, "replies": [] }
      ]
    }
  ],
  "scrapedAt": "2026-02-06T02:07:09.750Z"
}

Use cases

• Content archival — back up an entire newsletter, including update history and word counts
• Competitive research — track a competitor's posting cadence, topics, and engagement
• Audience analysis — mine comment threads for sentiment, questions, and top contributors
• Newsletter discovery — find every Substack in a niche by keyword, with subscriber counts
• Training data / RAG — clean HTML post bodies ready to chunk and embed
• Dashboards — feed reactions, comments and restacks into BI tools

How it works

1. Each URL is classified — homepage / custom domain / archive → the newsletter's full archive; a .../p/slug URL → just that post.
2. Keywords are resolved to publications through Substack's publication search.
3. For every newsletter, the archive is paginated with no depth limit; posts are filtered by type, date and paywall status.
4. With includeContent, each post's full HTML body is fetched.
5. With includeComments, the post's comment tree (with nested replies) is fetched and flattened counts are billed.
6. With includePublicationInfo, the newsletter's metadata is read from its homepage and written to the publications dataset.
7. Progress is checkpointed so a platform migration mid-run resumes without duplicates.

Pricing

Event	Free	Bronze	Silver	Gold
post (metadata only)	$0.00099	$0.00079	$0.00059	$0.00039
post-with-content	$0.00199	$0.00179	$0.00149	$0.00109
comment (per comment incl. replies)	$0.00049	$0.00039	$0.00039	$0.00029

Each post fires exactly one of post or post-with-content depending on includeContent. Each saved comment (including every nested reply) fires one comment event. Example: 1,000 posts with content and no comments on the Free plan ≈ $1.99.

FAQ

Do I need a login or a proxy? No. The actor uses Substack's public JSON API directly.

Does it work with custom domains? Yes — pass https://www.example.com just like a *.substack.com URL.

Can I get the full archive, not just 12 posts? Yes. Set maxPostsPerNewsletter to 0 for the complete archive.

Can I get paywalled post content? Substack only serves the preview for paid posts. You always get truncatedBodyText, wordcount, and all metadata; bodyHtml is populated for free posts.

Why are some newsletters' subscriber counts missing? subscriberCount is only present when the publication chooses to display it publicly.

How does keyword discovery work? Each keyword is matched against Substack's publication directory. The matching newsletters' recent posts are then scraped, up to maxSearchResultsPerKeyword posts per keyword split across those newsletters.

Can I schedule runs? Yes. Use Apify Schedules to run at any interval and only new posts will be added (de-duplicated by post ID within a run).

Legal and compliance

This actor collects publicly available data from Substack's public API — published posts, public comments, and publicly displayed publication metadata. It does not access private drafts, subscriber lists, or paywalled content. You are responsible for using the exported data in compliance with Substack's terms, applicable laws (including copyright and data-protection rules), and the rights of the content's authors.