Substack Scraper — Posts, Comments & Newsletter Search

Turn Substack into a clean dataset. Search posts by keyword across all of Substack, pull the full archive of any newsletter (including custom domains), or fetch specific posts. You get the full text, author, date, likes, comment count, paywall status and, if you want them, every comment and reply as structured rows. No login, no browser, no proxy setup.

Who uses this

• Media, PR & comms teams track what newsletters say about a brand, a competitor or a topic, every day.
• Researchers & analysts build corpora of long-form writing for discourse analysis, trend tracking and sentiment.
• AI / RAG builders feed clean plain-text posts into LLM pipelines, vector stores and summarizers.
• Creators & growth teams benchmark rival newsletters: what they publish, how often, and which posts get the most likes and comments.
• Community & audience researchers mine comment threads to see what readers actually care about.

What data you get

One row per post (rowType: "post"), plus one row per comment when comments are enabled (rowType: "comment").

Post rows

Field	Description
title, subtitle, url, slug, postId	Post identity
postDate	Publish time (ISO-8601, UTC)
authorName, authorHandle, authorId, coAuthors	Byline(s)
likeCount, commentCount, restackCount	Engagement
audience, isPaywalled	everyone, only_paid or founding
wordCount	Full length of the post
bodyText, bodyHtml	Full content as clean plain text and as HTML
contentTruncated	true when only the free preview of a paid post was available
tags, sectionName, coverImage, description, language	Metadata
podcastUrl, podcastDurationSec	For podcast episodes
publicationId, publicationName, publicationUrl	The newsletter
source, scrapedAt	Which input produced the row, and when

Comment rows

Field	Description
commentId, postId, postTitle, url	Which post the comment belongs to
commentBody, commentDate, editedAt	The comment
authorName, authorHandle, authorId	Commenter
depth, parentCommentId, replyCount	Thread structure: rebuild the tree, or keep top-level comments only
likeCount, isDeleted	Engagement / status

Use cases

1. Daily media monitoring. Search "your brand" with Published within: Past 24 hours on a schedule, and push new mentions to Slack or a sheet.
2. Competitor newsletter benchmarking. Scrape 10 rival newsletters with Most popular first and compare topics, length, likes and comment volume.
3. Incremental archive sync. Run once with no date filter to backfill a newsletter, then daily with Published after set to yesterday to pick up only new posts.
4. Reader-voice research. Enable Include comments on a newsletter's top posts to collect thousands of reader opinions for analysis.
5. LLM knowledge base. Export bodyText for a set of newsletters into your vector store for grounded Q&A.

Input

Parameter	Default	Description
mode	search	search (keywords), publications (newsletter archives), or posts (URLs)
searchQueries	—	Keywords, one per line (search mode)
dateRange	any	day, week, month, year: only recent posts (search mode)
publications	—	Subdomain (astralcodexten), x.substack.com URL or custom domain (www.noahpinion.blog)
sortBy	new	new or top (most popular first), for archives
postUrls	—	Specific post URLs (…/p/<slug>)
publishedAfter	—	YYYY-MM-DD; skip older posts (archives stop as soon as they reach older posts)
includeContent	true	Add bodyHtml + bodyText
includeComments	false	Add one row per comment and reply
maxCommentsPerPost	100	Comment cap per post (0 = all)
commentSort	best_first	best_first or most_recent_first
maxPostsPerSource	50	Post cap per keyword or newsletter (0 = no cap)
maxItems	200	Stop after this many rows in total (0 = no limit)

Example input — keyword search, last week

{
    "mode": "search",
    "searchQueries": ["artificial intelligence", "semiconductors"],
    "dateRange": "week",
    "maxPostsPerSource": 100,
    "maxItems": 500
}

Example input — two newsletters with comments

{
    "mode": "publications",
    "publications": ["astralcodexten", "https://www.noahpinion.blog"],
    "sortBy": "top",
    "includeComments": true,
    "maxCommentsPerPost": 50,
    "maxPostsPerSource": 20,
    "maxItems": 0
}

Example output

[
  {
    "rowType": "post",
    "postId": 216809249,
    "title": "Open Thread 452",
    "url": "https://www.astralcodexten.com/p/open-thread-452",
    "postDate": "2026-09-21T22:05:59.990Z",
    "postType": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "wordCount": 414,
    "likeCount": 61,
    "commentCount": 390,
    "restackCount": 1,
    "authorName": "Scott Alexander",
    "authorHandle": "astralcodexten",
    "tags": [],
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/cab976a2-…_1536x1024.png",
    "language": "en",
    "bodyText": "This is the weekly visible open thread. Post about anything you want, ask random questions, whatever. …",
    "contentTruncated": false,
    "publicationId": 89120,
    "publicationName": "Astral Codex Ten",
    "publicationUrl": "https://www.astralcodexten.com",
    "source": "url",
    "scrapedAt": "2026-09-23T01:38:30Z"
  },
  {
    "rowType": "comment",
    "commentId": 342672468,
    "postId": 216809249,
    "postTitle": "Open Thread 452",
    "parentCommentId": null,
    "depth": 0,
    "commentDate": "2026-09-21T22:15:53.598Z",
    "commentBody": "Since point 4 implies that normally all coauthors of a finalist review get a free subscription, …",
    "authorName": "Reader Name",
    "authorHandle": "reader-handle",
    "likeCount": 14,
    "replyCount": 2,
    "isDeleted": false,
    "publicationName": "Astral Codex Ten"
  }
]

The dataset has two ready-made views: Posts and Comments.

Scheduling, webhooks & integrations

• Schedule the actor (Console → Schedules) daily with dateRange: "day" or a rolling publishedAfter date for monitoring.
• Webhooks can call your endpoint when a run succeeds; fetch the dataset as JSON, CSV, Excel or XML.
• Connect directly to Google Sheets, Slack, Zapier, Make, Airbyte or S3 through Apify integrations.

Use it from AI assistants (MCP)

This actor works as a tool through the Apify MCP server, so Claude, ChatGPT, Cursor and other MCP clients can call it directly:

"Find Substack posts from the past week about GLP-1 drugs and summarize the main arguments."

The agent calls the actor with mode: "search", dateRange: "week" and reads back bodyText.

FAQ

Do I need a Substack account? No. The actor reads the same public data Substack shows logged-out visitors.

Can it get paid-only posts? It returns their metadata and the free preview that Substack shows publicly. Paid-only rows are flagged with isPaywalled: true and contentTruncated: true. The actor does not bypass paywalls.

How far back can archives go? All the way. Archives are paginated until the first post, unless you set maxPostsPerSource or publishedAfter.

Does it work with custom domains? Yes. Pass the custom domain (e.g. www.noahpinion.blog) or the *.substack.com subdomain; both resolve to the same newsletter.

How fresh is the data? Live. Every run queries Substack at run time.

How reliable is it? It uses Substack's own JSON API over plain HTTP, with no headless browser to break. Transient errors are retried automatically, and progress is checkpointed so a restarted run resumes without duplicate rows.

Is scraping Substack legal? The actor only collects publicly available information. You are responsible for using the data in line with applicable laws (e.g. GDPR for personal data in comments) and Substack's terms.

Something broke or you need a field? Open an issue on the actor's Issues tab.