Substack Scraper

Scrape posts, full article text, likes, restacks, comments and podcast episodes from any Substack newsletter, including publications on custom domains, with no login and no API key. Built for weekly or daily monitoring: a date filter, a Substack-side keyword search and a since-last-run checkpoint mean a scheduled run collects only what is new. $0.50 per 1,000 posts (full text included) and $0.20 per 1,000 comments; nothing charged for errors, summaries or empty runs. The first 10 posts each month are free for every account.

Why this one

	This actor
Input	A subdomain (lenny), a custom domain (https://www.lennysnewsletter.com), or a single post URL. Redirects are followed, so you never need to know which one a publication uses.
Full text	bodyText (clean plain text, paragraphs kept) and bodyHtml for every free post. Paywalled posts return Substack's public preview and are flagged bodyIsPreviewOnly: true, never silently truncated.
Engagement	Likes, restacks, comment count, word count, audience (everyone, only_paid, founding).
Comments	Optional, threaded: parentCommentId, depth, author handle, date, likes.
Podcasts and video	podcastUrl, podcastDuration, videoUploadId on episode posts.
Scheduled monitoring	onlyNewSinceLastRun stores the newest post date per publication and stops there next time. Quiet weeks cost only the start fee.
Reliability	One free summary record per publication with status and stoppedBy; unreachable or non-Substack sites produce an error record instead of a failed run; tested against live publications every day.
Price	$0.0005 per post, $0.0002 per comment, Apify's default $0.00005 start fee. No platform usage charged on top.

What people use it for

• Research and competitive analysis: what the leading newsletters in a niche publish, how often, and which posts get the most likes and comments.
• Weekly digest: every new post from a list of publications delivered to Slack, Notion, e-mail or a Google Sheet, through an Apify schedule and integration.
• AI and RAG pipelines: clean article text for a vector database or an agent, with the podcast audio URL when a post is an episode. Works through the Apify MCP server.
• Your own publication's analytics: likes, restacks, comments and word counts per post over time, exported to a sheet.
• Comment mining: who comments, how often, and what readers ask for.

Example tasks

Ready-made configurations you can open, copy and run as they are:

• Download a Substack newsletter's full archive with article text: every post of one publication with the full article text, likes, restacks and comment counts.

Input

Field	Type	Default	Description
publications	array	required	Newsletter URLs, subdomains or single post URLs (/p/slug).
maxPostsPerPublication	integer	50	Newest posts first. Up to 100,000.
search	string	–	Keyword filter applied by Substack's own archive search. You pay only for matching posts.
postedAfter	string	–	ISO date (2026-01-01) or a relative value (7d, 12h, 2w). Pagination stops at older posts.
onlyNewSinceLastRun	boolean	false	Checkpoint mode for schedules: remembers the newest post date per publication and stops there on the next run.
checkpointStoreName	string	substack-scraper-state	Key-value store that holds the checkpoints. Use different names for independent monitors.
includeBody	boolean	true	Fetch each post for bodyText and bodyHtml (one extra request per post, included in the price).
includeComments	boolean	false	Also output comment records for every post.
proxyConfiguration	object	none	Optional Apify Proxy. Substack rarely blocks; enable only on HTTP 403 or 429.

{
    "publications": ["https://www.lennysnewsletter.com", "stratechery"],
    "maxPostsPerPublication": 100,
    "postedAfter": "2026-01-01",
    "includeBody": true,
    "includeComments": true,
    "onlyNewSinceLastRun": true
}

Output

Four record types share one dataset, told apart by type.

post (charged, $0.0005 each). A real row from Lenny's Newsletter, long fields shortened:

{
  "type": "post",
  "publication": "www.lennysnewsletter.com",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "publicationName": "Lenny's Newsletter | Substack",
  "publicationDescription": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders. Click to read Lenny's Newsletter, a Substack publication with millions of subscribers.",
  "postId": 219482066,
  "slug": "how-to-tell-if-a-startup-has-a-moat",
  "url": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
  "title": "How to tell if a startup has a moat ",
  "subtitle": "A framework for assessing whether the company you’re considering joining is built to last",
  "description": "A framework for assessing whether the company you’re considering joining is built to last",
  "postType": "newsletter",
  "postDate": "2026-10-13T12:45:41.160Z",
  "audience": "only_paid",
  "isPaywalled": true,
  "authors": [
    "Dan Hockenmaier"
  ],
  "authorHandles": [
    "danhock"
  ],
  "likes": 284,
  "comments": 1,
  "restacks": 7,
  "wordCount": 4153,
  "coverImage": "https://substackcdn.com/image/fetch/$s_!7Uce!,f_auto,q_auto:…",
  "podcastUrl": null,
  "podcastDuration": null,
  "videoUploadId": null,
  "bodyHtml": "<p><em><span>👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth,…",
  "bodyText": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice. For more: Lenny’s Jobs | Lenny’s Podcast | Lennybot | How I AI | Become an AI-Native Builder and my other favorite AI/PM co…",
  "bodyWordCount": 2261,
  "bodyIsPreviewOnly": true
}

Field	Type	Notes
postId, slug, url	number, string, string	Substack's post ID, slug and canonical URL.
title, subtitle, description	string	As published.
postType	string	newsletter, podcast, thread or video.
postDate	string	ISO 8601 UTC.
audience, isPaywalled	string, boolean	everyone, only_paid or founding; paywalled when not everyone.
authors, authorHandles	arrays	All bylines.
likes, comments, restacks, wordCount	numbers	Engagement and length as reported by Substack.
bodyText, bodyHtml, bodyWordCount, bodyIsPreviewOnly	string, string, number, boolean	Full text when includeBody is on; preview-only flag for paywalled posts.
coverImage, podcastUrl, podcastDuration, videoUploadId	misc	Media on the post.
publication, publicationUrl, publicationName, publicationDescription	strings	Publication metadata on every record.

comment (charged, $0.0002 each, only when includeComments is on):

{
  "type": "comment",
  "publication": "www.lennysnewsletter.com",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "publicationName": "Lenny's Newsletter | Substack",
  "publicationDescription": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders. Click to read Lenny's Newsletter, a Substack publication with millions of subscribers.",
  "postId": 215694124,
  "postUrl": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "commentId": 349085941,
  "parentCommentId": null,
  "depth": 0,
  "date": "2026-09-29T23:34:47.245Z",
  "name": "Product Mindset",
  "handle": "productmindset",
  "body": "Thank you Lenny for putting this together. It feels amazing to see the world's best product thinkers wrestling with the same questions out l…",
  "likes": 4,
  "isAuthorReply": false
}

summary (free, one per publication at the end of the run). status is COMPLETE, PARTIAL (stopped by your spending limit), EMPTY or ERROR; stoppedBy is maxPosts, postedAfter, checkpoint, endOfArchive, chargeLimit, singlePost or error:

{
  "type": "summary",
  "publication": "www.lennysnewsletter.com",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "status": "COMPLETE",
  "postsScraped": 4,
  "commentsScraped": 7,
  "newestPostDate": "2026-10-13T12:45:41.160Z",
  "oldestPostDate": "2026-09-15T12:45:28.445Z",
  "stoppedBy": "maxPosts",
  "checkpointUsed": null
}

error (free): {"type": "error", "publication": "https://example.com", "error": "Publication not reachable (HTTP 404) at https://example.com"}

Pricing and cost examples

Try it free. The first 10 posts each calendar month are free for every Apify account, with nothing to enter: the actor keeps count in a small key-value store in your account (free-slice-substack-scraper) and charges only from the 11th on. The free results are real results you keep; the counter resets on the first of each month.

Job	Items	Cost
Full archive of one newsletter, 300 posts with full text	300 posts	$0.15
Weekly monitor of 40 newsletters averaging 2 new posts each, checkpoint on	80 posts a week	$0.04 a week, under $0.20 a month
The same monitor with comments on (about 15 per post)	80 posts + 1,200 comments	$0.28 a week
Research dataset: 50 newsletters × 200 posts with text	10,000 posts	$5.00

Platform compute is included in the per-item price. A run that finds nothing new costs only Apify's $0.00005 start fee.

Discounts for paid Apify plans, applied automatically to the buyer's plan: Bronze 10% off, Silver 25% off, Gold and enterprise tiers 40% off the prices above. Free-plan users pay the listed price.

Scheduling: a weekly digest in two minutes

1. Create a task from this actor with your publications and onlyNewSinceLastRun: true.
2. Add an Apify schedule (for example every Monday at 8:00).
3. Add an integration on the task: e-mail, Slack, Google Sheets, Notion via Make or n8n, or a webhook. Each run delivers only new posts; the summary records tell you per publication whether the run was complete.

The first checkpoint run fetches up to maxPostsPerPublication posts; later runs stop at the previous newest post. To start over, change checkpointStoreName or delete the key-value store.

Run it from your code

Node.js:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('agentsgearbox/substack-scraper').call({
    publications: ['https://www.lennysnewsletter.com'], maxPostsPerPublication: 20, includeBody: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.filter((i) => i.type === 'post').map((i) => [i.title, i.likes, i.bodyWordCount]));

Python:

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("agentsgearbox/substack-scraper").call(run_input={
    "publications": ["https://www.lennysnewsletter.com"], "maxPostsPerPublication": 20, "includeBody": True})
posts = [i for i in client.dataset(run["defaultDatasetId"]).iterate_items() if i["type"] == "post"]

cURL (synchronous, returns the dataset):

curl -X POST "https://api.apify.com/v2/acts/agentsgearbox~substack-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"publications": ["stratechery"], "maxPostsPerPublication": 10}'

For AI agents

Give your agent hands. Instead of fetching pages, getting blocked and reasoning over HTML, it calls this actor as a tool and gets verified results back as JSON, priced per result. Fetching 500 newsletter posts one page at a time is 500 fetches of 10,000 to 20,000 tokens each. This actor returns 500 posts with full text for $0.25.

• MCP, this actor only: add https://mcp.apify.com?tools=agentsgearbox/substack-scraper to Claude Desktop, Claude Code, Cursor or any MCP client (with your Apify token), then ask for "the last 20 posts of lennysnewsletter.com with their full text".

• MCP, all agentsgearbox actors: https://mcp.apify.com?tools=agentsgearbox/telegram-channel-scraper,agentsgearbox/google-play-reviews-scraper,agentsgearbox/substack-scraper,agentsgearbox/google-news-scraper,agentsgearbox/brand-name-generator,agentsgearbox/ai-readiness-audit,agentsgearbox/company-lookup,agentsgearbox/app-intelligence,agentsgearbox/app-store-reviews-scraper,agentsgearbox/company-jobs-scraper,agentsgearbox/apple-podcasts-scraper,agentsgearbox/remote-jobs-scraper

• No Apify account? This actor is eligible for Apify's agentic payments: an agent can buy a spend-capped token through agi.apify.com (x402, MPP or Skyfire) and call the API directly.
• Prices are per result and known in advance, so the agent can set a spending limit; error records come back as data instead of failed runs.

Limitations, stated plainly

• Paywalled posts return exactly what Substack shows a non-subscriber: the free preview, flagged bodyIsPreviewOnly. The actor never bypasses a paywall.
• Substack Notes (the social feed) are not covered; posts, podcasts, threads and comments are.
• Comment threads on some publications are visible only to subscribers; those return no comment records.
• search uses Substack's own archive search, which matches words rather than substrings.

Troubleshooting

Symptom	Cause and fix
error record "Publication not reachable"	The domain is misspelled or is not a Substack publication. Open the URL in a browser and check it says "substack" in the page footer.
0 posts with the checkpoint on	Nothing new since the last run. Check the summary record: stoppedBy: "checkpoint".
bodyIsPreviewOnly: true on many posts	The publication is mostly paid. Filter on audience = "everyone" if you only want full text.
HTTP 403 or 429 in the log	Rare; enable Apify Proxy in proxyConfiguration.
Run stopped with PARTIAL	Your spending limit for the run was reached. Raise it or lower maxPostsPerPublication.

Track record

• Published 13 October 2026 and tested every day since by an automated run against live publications; every test run has succeeded. The Store's run statistics on this page reflect those runs with a delay of up to a day.
• Built without a browser, so a run cannot fail on Chrome crashes or memory limits, which is the usual failure mode of browser-based scrapers in this category.
• Changes are released only after a local test and a real platform run, and every actor from this account is monitored daily, so a breaking change on the target site is normally fixed the same day.

More from agentsgearbox

• Telegram Channel Scraper: public Telegram channels as JSON, no API key.
• Google Play Reviews Scraper: Google Play reviews with replies and versions, any market.
• Google News Scraper: Google News with real publisher URLs and full text.
• Brand & Business Name Generator: brand names verified live on domains, handles and app stores.
• AI Readiness Audit: how well AI search engines and agents can read a site: crawler access, llms.txt, structured data, content without JavaScript, scored with fixes.
• Company Lookup: one row per domain with legal identity and VAT validation, contacts, socials, tech stack, open jobs, apps, news and Wikidata facts.
• App Intelligence: App Store and Google Play in one run: search, top charts, full app details and each app's twin on the other store.
• App Store Reviews Scraper · iOS + Google Play, Any Country: Reviews of any iOS or Android app from both stores in one run: stars, text, version, votes, developer replies, since-last-run checkpoint, rating trend per app..
• Company Jobs Scraper: every open job on a company's careers page: nine applicant tracking systems, one schema, salaries where stated, hiring signals free.
• Apple Podcasts Scraper: Apple Podcasts search, charts, episodes with audio links and listener reviews, any country.
• Remote Jobs Scraper: remote jobs from six boards and the Hacker News hiring thread in one schema, deduplicated, only-new mode for alerts.

All of them are pay-per-event, run without a browser, and are compared with their alternatives on agentsgearbox.com.

Support

Open an issue on this actor's Issues tab, usually answered within a day, or email support@agentsgearbox.com. More actors and daily comparisons of scrapers for 200+ websites at agentsgearbox.com.

Legal and fair use

The actor reads pages and endpoints Substack serves publicly. Respect authors' copyright and the privacy of commenters when you store or republish data.