Substack Scraper – Posts, Newsletters & Free Full Text

Scrape any Substack newsletter, including publications on custom domains. You get each post's title, subtitle, authors, publish date, free/paid audience, word count, likes, comments, restacks, cover image and tags. Free posts also come with their full text as Markdown, HTML or plain text. You can also search Substack publications by keyword.

• Every public post, custom domains included. Scrape the whole archive or the newest N posts, filtered by date, audience and post type. lenny, https://lenny.substack.com and https://www.astralcodexten.com all work.
• Fast and paywall-safe. The Actor makes plain HTTP requests to Substack's public JSON endpoints, with no browser, and 100 posts take a few seconds. Requests are anonymous and never unlock paid content.
• $1.00 per 1,000 posts plus $0.005 per run. Failed inputs and posts dropped by your filters are free.

Quick start

This is the prefilled input. It returns the 3 newest posts from Lenny's Newsletter in a few seconds, for about $0.008.

{ "publications": ["lenny"], "maxPostsPerPublication": 3 }

Sample output

title	publicationName	publishedAt	audience	wordCount	reactionCount	commentCount
Your Book Review: Lee Kuan Yew's Memoirs	Astral Codex Ten	2026-09-25	everyone	5,683	198	81
Fall Meetups 2026 - Call For Meetup Organizers	Astral Codex Ten	2026-09-25	everyone	2,086	52	18
Hidden Open Thread 452.5	Astral Codex Ten	2026-09-23	only_paid	20	10	45

Price comparison (Apify Store, September 2026)

Actor	Price per 1,000 posts	Monthly users
Substack Scraper (this Actor)	$1.00 (+ $0.005 per run)	new
automation-lab/substack-scraper	$1.15	134

Related Actors: Bluesky Scraper and Google News Scraper. Together they cover newsletters, social posts and news headlines for media monitoring.

What can this Substack scraper do?

Feature
Scrape all posts of a publication (the whole archive or the newest N)	✅
Filter by date range (2026-01-01, or relative like 30 days)	✅
Filter by audience (free / paid) and post type (newsletter / podcast / thread)	✅
Full post content for free posts: Markdown, HTML, both, or plain text	✅
Likes / reactions, comment count, restacks, word count	✅
Author names, handles, photos, guest-author flag	✅
Publication info: name, description, author, logo, language, custom domain, first post date	✅
Individual post URLs	✅
Keyword search for publications (best-effort, see FAQ)	✅
Paid-post full text	❌ Never. We don't bypass paywalls.

Use cases

• Newsletter research and competitor tracking. See what top Substacks in your niche publish, how often, and which posts get the most likes and comments.
• Content datasets for AI and RAG. Clean Markdown of free posts with metadata, ready for embeddings or LLM summarisation.
• Media monitoring. Schedule the Actor daily with dateFrom: "1 day" to collect new posts from a list of newsletters.
• Lead generation and influencer discovery. Search publications by keyword, then rank them by engagement.
• Archiving and backup of your own publication's public posts.

Input example

{
  "publications": ["https://www.astralcodexten.com", "lenny", "noahpinion.substack.com"],
  "postUrls": ["https://www.slowboring.com/p/the-map-is-melting"],
  "maxPostsPerPublication": 20,
  "dateFrom": "2026-06-01",
  "audience": "all",
  "includeContent": true,
  "contentFormat": "markdown"
}

Keyword search:

{ "searchQueries": ["climate", "product management"], "maxSearchResults": 20, "scrapeSearchResults": false }

Field	Description
publications	A subdomain (lenny), a https://x.substack.com URL, or a custom domain. Post URLs pasted here are detected automatically.
postUrls	Specific posts (https://<publication>/p/<slug>).
searchQueries / maxSearchResults / scrapeSearchResults	Find publications by keyword, and optionally scrape their posts too.
maxPostsPerPublication	Newest first. 0 scrapes the whole archive. Default is 20.
dateFrom / dateTo	YYYY-MM-DD or relative dates (30 days, 6 months).
audience	all, free or paid.
postTypes	newsletter, podcast, thread. Leave empty for all types.
includeContent / contentFormat	Full body of free posts as markdown, html, both or text.
includePublicationInfo	Adds publication fields to every post.
maxConcurrency, proxyConfiguration	Advanced. The defaults are fine; a proxy is rarely needed.

Output example

Each post is one dataset item. The example below is shortened:

{
  "type": "post",
  "postId": 208730073,
  "title": "Announcing Lenny’s Jobs: The best place in the world to find, vet, and land your dream job",
  "subtitle": "Where product managers, engineers, designers, and growth/marketing professionals discover high-quality open roles",
  "slug": "announcing-lennys-jobs-the-best-place",
  "url": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place",
  "postType": "newsletter",
  "publishedAt": "2026-08-18T15:40:06.921Z",
  "audience": "everyone",
  "isPaid": false,
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny", "id": 1849774, "isGuest": false }],
  "authorNames": "Lenny Rachitsky",
  "wordCount": 933,
  "reactionCount": 379,
  "reactions": { "❤": 379 },
  "commentCount": 18,
  "restacks": 9,
  "coverImage": "https://substackcdn.com/image/fetch/...png",
  "tags": ["Career"],
  "previewText": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice...",
  "publicationName": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "publicationSubdomain": "lenny",
  "publicationCustomDomain": "www.lennysnewsletter.com",
  "publicationAuthor": "Lenny Rachitsky",
  "contentIncluded": true,
  "contentMarkdown": "My goal is to help you succeed in your career. ...",
  "scrapedAt": "2026-09-27T22:23:33+00:00"
}

For paid posts, contentIncluded is false and contentNote explains why. Only metadata and previewText are returned. Search results have "type": "publication" and include the publication name, URL, author, description and a subscriber-count label. A run summary, with counts and any failed inputs, is saved to the OUTPUT record of the key-value store. See SAMPLE_OUTPUT.json for complete items.

Pricing

Pay per event. You pay only for results:

Event	Price
Run start	$0.005
Post scraped	$0.001 ($1 per 1,000 posts)
Publication search result	$0.002

Examples: 100 posts cost $0.105, and 10,000 posts cost about $10. Set a maximum cost per run in Apify. The Actor stops cleanly when it reaches the limit and never saves an item it can't charge for.

How it compares (Apify Store, September 2026)

Actor	Monthly users	Start fee	Notes
This Actor	new	$0.005	$0.001/post, custom domains, date filters, keyword search, Markdown
automation-lab Substack scraper	134	see listing	$1.15 per 1,000 posts
fatihtahta Substack scraper	35	see listing
easyapi Substack scraper	23 (rated 1.9★)	$0.09

Use with AI agents (Apify MCP)

This Actor works as a tool for Claude, ChatGPT, Cursor and other MCP clients through the Apify MCP server (https://mcp.apify.com). Add the Actor to your MCP configuration. An agent can then ask for things like "get the last 10 free posts from Noahpinion as Markdown" and receive structured JSON. The inputs are simple strings and numbers, so they're easy for an LLM to fill in. contentFormat: "markdown" gives compact, token-efficient text for summarising or RAG.

You can also call it from code with the Apify API or client (apify-client for Python/JS):

from apify_client import ApifyClient
run = ApifyClient("<TOKEN>").actor("gazidev/substack-scraper").call(run_input={"publications": ["lenny"], "maxPostsPerPublication": 5})

FAQ

Does it scrape paid or paywalled posts? No. The Actor sends anonymous requests only, with no cookies or logins, and it never outputs the body of a post whose audience isn't everyone. For paid posts you get the metadata and Substack's public teaser text.

Does it work with custom domains? Yes. Enter the domain (for example https://www.astralcodexten.com). *.substack.com addresses that redirect to a custom domain are followed automatically.

How many posts can I get? Up to the whole public archive. Set maxPostsPerPublication: 0 for no limit, and use dateFrom/dateTo to narrow the range.

Are comments included? Comment counts and reaction counts are included. The text of individual comments isn't.

How does keyword search work? It uses the same public search that substack.com uses. That endpoint is undocumented, so results and ranking may change. Substack's older dedicated publication-search endpoint now always returns empty results, so it isn't used.

Some inputs failed. Was I charged? No. Errors such as a site not on Substack or a post returning 404 are logged and listed in the OUTPUT summary. They're never charged.

Is scraping Substack legal? This Actor only reads publicly available data that any logged-out visitor can see. You're responsible for how you use it. Respect authors' copyright, and don't republish full articles without permission.

Is it affiliated with Substack? No. This is an independent tool and isn't endorsed by Substack Inc.