Substack Scraper — Newsletter Posts, Authors, Likes & Free Content

Scrape any Substack newsletter, including publications on custom domains. Get every post's title, subtitle, author(s), publish date, free/paid audience, word count, likes (reactions), comment count, restacks, cover image and tags. Free posts also come with their full text as Markdown, HTML or plain text. You can also search Substack publications by keyword.

• ⚡ Fast and cheap. Plain HTTP requests to Substack's public JSON endpoints, with no browser. 100 posts take a few seconds.
• 💲 $0.002 per post plus a $0.005 start fee per run. You don't pay for failed inputs or for posts your filters drop.
• 🔒 Paywall-safe. Requests are anonymous and never unlock paid content. For paid posts you get metadata and Substack's public teaser text only.
• 🌐 Works with custom domains such as www.lennysnewsletter.com or www.astralcodexten.com, as well as *.substack.com addresses and bare subdomains like lenny.

What can this Substack scraper do?

Feature
Scrape all posts of a publication (the whole archive or the newest N)	✅
Filter by date range (2026-01-01, or relative like 30 days)	✅
Filter by audience (free / paid) and post type (newsletter / podcast / thread)	✅
Full post content for free posts: Markdown, HTML, both, or plain text	✅
Likes / reactions, comment count, restacks, word count	✅
Author names, handles, photos, guest-author flag	✅
Publication info: name, description, author, logo, language, custom domain, first post date	✅
Individual post URLs	✅
Keyword search for publications (best-effort, see FAQ)	✅
Paid-post full text	❌ Never. We don't bypass paywalls.

Use cases

• Newsletter research and competitor tracking. See what top Substacks in your niche publish, how often, and which posts get the most likes and comments.
• Content datasets for AI and RAG. Clean Markdown of free posts with metadata, ready for embeddings or LLM summarisation.
• Media monitoring. Schedule the Actor daily with dateFrom: "1 day" to collect new posts from a list of newsletters.
• Lead generation and influencer discovery. Search publications by keyword, then rank them by engagement.
• Archiving and backup of your own publication's public posts.

Input example

{
  "publications": ["https://www.astralcodexten.com", "lenny", "noahpinion.substack.com"],
  "postUrls": ["https://www.slowboring.com/p/the-map-is-melting"],
  "maxPostsPerPublication": 20,
  "dateFrom": "2026-06-01",
  "audience": "all",
  "includeContent": true,
  "contentFormat": "markdown"
}

Keyword search:

{ "searchQueries": ["climate", "product management"], "maxSearchResults": 20, "scrapeSearchResults": false }

Field	Description
publications	A subdomain (lenny), a https://x.substack.com URL, or a custom domain. Post URLs pasted here are detected automatically.
postUrls	Specific posts (https://<publication>/p/<slug>).
searchQueries / maxSearchResults / scrapeSearchResults	Find publications by keyword, and optionally scrape their posts too.
maxPostsPerPublication	Newest first. 0 scrapes the whole archive. Default is 20.
dateFrom / dateTo	YYYY-MM-DD or relative dates (30 days, 6 months).
audience	all, free or paid.
postTypes	newsletter, podcast, thread. Leave empty for all types.
includeContent / contentFormat	Full body of free posts as markdown, html, both or text.
includePublicationInfo	Adds publication fields to every post.
maxConcurrency, proxyConfiguration	Advanced. The defaults are fine; a proxy is rarely needed.

Output example

Each post is one dataset item. The example below is shortened:

{
  "type": "post",
  "postId": 208730073,
  "title": "Announcing Lenny’s Jobs: The best place in the world to find, vet, and land your dream job",
  "subtitle": "Where product managers, engineers, designers, and growth/marketing professionals discover high-quality open roles",
  "slug": "announcing-lennys-jobs-the-best-place",
  "url": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place",
  "postType": "newsletter",
  "publishedAt": "2026-08-18T15:40:06.921Z",
  "audience": "everyone",
  "isPaid": false,
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny", "id": 1849774, "isGuest": false }],
  "authorNames": "Lenny Rachitsky",
  "wordCount": 933,
  "reactionCount": 379,
  "reactions": { "❤": 379 },
  "commentCount": 18,
  "restacks": 9,
  "coverImage": "https://substackcdn.com/image/fetch/...png",
  "tags": ["Career"],
  "previewText": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice...",
  "publicationName": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "publicationSubdomain": "lenny",
  "publicationCustomDomain": "www.lennysnewsletter.com",
  "publicationAuthor": "Lenny Rachitsky",
  "contentIncluded": true,
  "contentMarkdown": "My goal is to help you succeed in your career. ...",
  "scrapedAt": "2026-09-27T22:23:33+00:00"
}

For paid posts, contentIncluded is false and contentNote explains why. Only metadata and previewText are returned. Search results have "type": "publication" and include the publication name, URL, author, description and a subscriber-count label. A run summary, with counts and any failed inputs, is saved to the OUTPUT record of the key-value store. See SAMPLE_OUTPUT.json for complete items.

Pricing

Pay per event. You pay only for results:

Event	Price
Run start	$0.005
Post scraped	$0.002 ($2 per 1,000 posts)
Publication search result	$0.002

Examples: 100 posts cost $0.205, and 10,000 posts cost about $20. Set a maximum cost per run in Apify. The Actor stops cleanly when it reaches the limit and never saves an item it can't charge for.

How it compares (Apify Store, September 2026)

Actor	Monthly users	Start fee	Notes
This Actor	new	$0.005	$0.002/post, custom domains, date filters, keyword search, Markdown
automation-lab Substack scraper	118	see listing
fatihtahta Substack scraper	35	see listing
easyapi Substack scraper	23 (rated 1.9★)	$0.09

Use with AI agents (Apify MCP)

This Actor works as a tool for Claude, ChatGPT, Cursor and other MCP clients through the Apify MCP server (https://mcp.apify.com). Add the Actor to your MCP configuration. An agent can then ask for things like "get the last 10 free posts from Noahpinion as Markdown" and receive structured JSON. The inputs are simple strings and numbers, so they're easy for an LLM to fill in. contentFormat: "markdown" gives compact, token-efficient text for summarising or RAG.

You can also call it from code with the Apify API or client (apify-client for Python/JS):

from apify_client import ApifyClient
run = ApifyClient("<TOKEN>").actor("gazidev/substack-scraper").call(run_input={"publications": ["lenny"], "maxPostsPerPublication": 5})

FAQ

Does it scrape paid or paywalled posts? No. The Actor sends anonymous requests only, with no cookies or logins, and it never outputs the body of a post whose audience isn't everyone. For paid posts you get the metadata and Substack's public teaser text.

Does it work with custom domains? Yes. Enter the domain (for example https://www.astralcodexten.com). *.substack.com addresses that redirect to a custom domain are followed automatically.

How many posts can I get? Up to the whole public archive. Set maxPostsPerPublication: 0 for no limit, and use dateFrom/dateTo to narrow the range.

Are comments included? Comment counts and reaction counts are included. The text of individual comments isn't.

How does keyword search work? It uses the same public search that substack.com uses. That endpoint is undocumented, so results and ranking may change. Substack's older dedicated publication-search endpoint now always returns empty results, so it isn't used.

Some inputs failed. Was I charged? No. Errors such as a site not on Substack or a post returning 404 are logged and listed in the OUTPUT summary. They're never charged.

Is scraping Substack legal? This Actor only reads publicly available data that any logged-out visitor can see. You're responsible for how you use it. Respect authors' copyright, and don't republish full articles without permission.

Is it affiliated with Substack? No. This is an independent tool and isn't endorsed by Substack Inc.