Substack Scraper: posts, full text, likes, comments and newsletters

Scrape any Substack newsletter, including ones on custom domains. Get a publication's full archive, single posts with their full text, the comments under each post, and search all of Substack for posts or newsletters, or pull the top newsletters in a category. Every post comes with likes, comment count, restacks, word count, authors and paywall status. No login and no API key.

What data you get

• Posts: title, subtitle, URL, slug and ID; publish and update dates (ISO 8601); post type (article, podcast, thread); audience (everyone, only_paid, only_subscribers)
• Author name plus every byline (name, handle, profile link, guest flag)
• Publication name, URL and ID
• Likes, reactions by emoji, comment count, restacks and word count
• Tags, language, cover image, podcast duration, the short public preview text
• Full post text as clean plain text (optional HTML). Paid posts give only the free public preview that Substack shows to everyone, flagged with textIsPreviewOnly
• Comments (optional) as separate results: text, date, author name and handle, likes, restacks, number of replies, parentId and depth for threads, and whether the post's author wrote it
• Publications (from search or category leaderboards): name, URL, subdomain, custom domain, description, author, subscriber count as shown by Substack (e.g. "Over 58,000 subscribers"), paid subscriber band, language, launch date and logo
• type (post, comment or publication) and source (which input found the result) on every item

Sample result

{
  "type": "post",
  "id": 215694124,
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "postType": "newsletter",
  "audience": "everyone",
  "isPaid": false,
  "publishedAt": "2026-09-29T13:15:57.512Z",
  "authorName": "Lenny Rachitsky",
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny", "profileUrl": "https://substack.com/@lenny", "isGuest": false }],
  "publicationName": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "likes": 291,
  "comments": 7,
  "restacks": 5,
  "wordCount": 1276,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/9fed347c-..._9213x6142.jpeg",
  "text": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice… I’m excited to share that every main-stage talk from the recent Lenny & Friends Summit is now online…",
  "textIsPreviewOnly": false,
  "source": "publication:www.lennysnewsletter.com"
}

{
  "type": "comment",
  "id": 349085941,
  "postId": 215694124,
  "postUrl": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "parentId": null,
  "depth": 0,
  "text": "Thank you Lenny for putting this together…",
  "publishedAt": "2026-09-29T23:34:47.245Z",
  "authorName": "Product Mindset",
  "authorHandle": "productmindset",
  "likes": 4,
  "replies": 0
}

How to use

Field	What it does
publications	Newsletters to scrape in full: garymarcus, garymarcus.substack.com or a custom domain like https://www.lennysnewsletter.com
postUrls	Specific posts, e.g. https://www.astralcodexten.com/p/why-does-ozempic-cure-all-diseases or https://substack.com/home/post/p-221631819
searchQueries	Keywords searched across all of Substack
searchType	posts (default) or publications
categories	Category leaderboards, e.g. technology, business, finance, culture, politics
categoryRanking	all (top overall) or paid (top paid)
sort	Archive order: new (default) or top (most popular first)
maxItems	Stop after this many results in total (default 50)
maxPostsPerPublication	Per-newsletter cap so one big archive does not use the whole budget (0 = no cap)
includeFullText	Post body as plain text (default on). Turn off for a fast metadata-only run
includeHtml	Also return the body HTML
includeComments	Save each post's comments and replies as comment results
maxCommentsPerPost	Most-liked comments first (default 50, 0 = all)
onlyFree	Skip paid and subscriber-only posts
postType	all, newsletter, podcast or thread
startDate / endDate	Only posts published in this range (YYYY-MM-DD)

Example inputs:

{ "publications": ["garymarcus", "https://www.lennysnewsletter.com"], "maxPostsPerPublication": 100, "maxItems": 200 }

{ "postUrls": [{ "url": "https://www.astralcodexten.com/p/why-does-ozempic-cure-all-diseases" }], "includeComments": true, "maxCommentsPerPost": 0 }

{ "searchQueries": ["ai agents"], "searchType": "publications", "categories": ["technology"], "maxItems": 100 }

A full archive with text costs one request per post, so 1,000 posts take a few minutes. Turn off includeFullText for a fast run: the archive then gives about 20 posts per request with all stats.

Pricing

$2 per 1,000 results, charged only for results saved to your dataset (a post, a comment or a publication each count as one result). The full 500-post archive of a newsletter costs $1.00. 100 posts with 20 comments each cost about $4.20.

Use cases

• Archive and analyse a newsletter's whole back catalogue: which topics and headlines get the most likes, comments and restacks
• Competitor and creator research: compare posting frequency, engagement and paywall strategy
• Find newsletters in a niche through search and category leaderboards, with subscriber counts, for sponsorships and partnerships
• Collect free post text for RAG pipelines, summarisation and AI training sets
• Audience research: read what commenters say and which comments get the most likes
• Monitor newsletters on a schedule with startDate for new posts

FAQ

Is it legal? The scraper only reads public data that any visitor sees without logging in. It never gets around the paywall: paid posts return the free preview Substack shows to everyone. It does not collect emails or other contact details. You are responsible for how you use the data, including personal data and copyright.

Does it work with custom domains? Yes. Pass the domain (www.lennysnewsletter.com, astralcodexten.com); the scraper finds the right address itself.

Why are subscriber counts text? Substack only publishes rounded figures ("Over 1,200,000 subscribers"). subscribersOrderOfMagnitude gives a number you can sort by.

Why did I get fewer results than maxItems? The archive, search or leaderboard ran out, or your filters (onlyFree, postType, dates) removed posts. Comments count toward maxItems too.

What is not included? Paid post bodies, Substack Notes and chat, and subscriber lists.

Can I run it on a schedule? Yes. Use Apify Schedules with startDate to collect new posts daily and send them to Google Sheets, Slack, Make, Zapier or a webhook with Apify integrations.

API

Call it from any language with the Apify API or client libraries (apify-client for JavaScript and Python), or connect it to an AI agent through the Apify MCP server.