Substack Scraper — Posts, Comments & Subscriber Counts

Scrape any Substack newsletter into clean, structured data — full post content, threaded comments, tags, authors, podcast episodes and exact free-subscriber counts. Works with *.substack.com subdomains and custom domains, follows publication moves automatically, and detects paywalled posts honestly. Pure HTTP against Substack's public JSON API — no browser, fast and cheap.

How it works

✨ Why use this scraper?

• Any URL shape — newsletter homepages, custom domains (www.noahpinion.blog), /archive, /about, or direct post URLs (/p/{slug}). Bare names work too ("noahpinion").
• Full post content — bodyHtml + clean bodyText, word counts, cover images, tags, authors with bios, podcast audio URLs and durations.
• Threaded comments — one row per comment with parentCommentId + depth preserved, joinable to posts via postId.
• Exact subscriber counts — the publication row carries the precise freeSubscriberCount (e.g. 454000), not just "454K+".
• Honest paywall handling — paid posts return their free preview flagged bodyIsPreviewOnly: true; filter them out entirely with onlyFree.
• Date-window scraping — startDate + newest-first sort stops paginating at the cutoff, so monitoring runs stay small and cheap.

🎯 Use cases

Audience	What they pull
Newsletter analysts & investors	Subscriber counts, posting cadence, engagement (reactions/comments) across a portfolio of publications
Content & SEO teams	Full archives as clean text for topic research, repurposing, or LLM fine-tuning corpora
Community managers	Comment threads with authors and reply structure for sentiment and superfan analysis
Podcast researchers	Episode metadata + audio URLs from podcast-type posts
Lead generation	Publication rows with author names, handles and opt-in contact-email enrichment
Media monitors	Daily startDate-windowed runs over a watchlist of newsletters

📥 Supported inputs

Input	Example	What it does
Newsletter homepage	https://noahpinion.substack.com	Walks the archive, newest first
Custom domain	https://www.noahpinion.blog	Same — custom domains serve the same API
Archive / About page	https://thezvi.substack.com/archive	Treated as the publication root
Direct post URL	https://…/p/some-post-slug	Scrapes just that post
Bare name (publications field)	noahpinion	Expands to the subdomain homepage

Not supported: Substack Notes, Chat, private/logged-in-only publications, and full text of paywalled posts (previews only — see FAQ).

🔄 How it works

1. Classify each start URL — publication root vs direct post.
2. Resolve moves — {name}.substack.com publications that migrated to a custom domain are followed automatically (e.g. Astral Codex Ten).
3. Walk the archive API page by page, applying your filters; with startDate + newest-first sort it stops at the cutoff.
4. Fetch full content per post (the archive API only carries metadata), plus comments and the publication info row when enabled.
5. Push rows — post, comment, and publication row types, joinable via postId / publicationUrl.

⚙️ Input parameters

Field	Type	Default	Description
startUrls	array	—	Substack URLs (homepage, custom domain, /archive, /about, /p/… post)
publications	array	—	Bare subdomains/domains, each expands to a homepage
maxPostsPerNewsletter	integer	50	Posts per newsletter; 0 = full archive
maxItems	integer	1000	Hard cap on post rows for the whole run
sortOrder	string	new	new (newest first) or top (most popular)
includeBody	boolean	true	Full bodyHtml + bodyText (one extra API call per post)
includeComments	boolean	false	Emit comment rows per post
maxCommentsPerPost	integer	100	Comment rows per post incl. replies; 0 = unlimited
includePublicationInfo	boolean	true	One publication row per newsletter
contentType	string	all	all / newsletter / podcast / thread
startDate / endDate	string	—	YYYY-MM-DD publish-date window
onlyFree	boolean	false	Skip paywalled posts entirely
enrichEmails	boolean	false	Opt-in author/publication contact-email discovery (billed per email found)
maxConcurrency	integer	5	Posts processed in parallel
proxy	object	built-in	Explicit override only; leave empty for the built-in residential routing

Example input:

{
    "startUrls": ["https://www.noahpinion.blog", "https://astralcodexten.substack.com"],
    "maxPostsPerNewsletter": 50,
    "includeComments": true,
    "maxCommentsPerPost": 100,
    "includePublicationInfo": true,
    "startDate": "2026-01-01",
    "sortOrder": "new"
}

📊 Output overview

The dataset mixes three row types, distinguished by rowType:

rowType	One row per	Join keys
post	post (newsletter / podcast / thread)	postId, publicationUrl
comment	comment, replies flattened with parentCommentId + depth	postId, postSlug
publication	newsletter (when includePublicationInfo)	publicationId, baseUrl

📦 Output samples

post row (trimmed, real run):

{
    "rowType": "post",
    "postId": 207376523,
    "slug": "book-review-power-and-progress-874",
    "url": "https://www.noahpinion.blog/p/book-review-power-and-progress-874",
    "title": "Book Review: \"Power and Progress\"",
    "contentType": "newsletter",
    "audience": "everyone",
    "isPaid": false,
    "bodyIsPreviewOnly": false,
    "postDate": "2026-07-16T12:01:32.192Z",
    "bodyHtml": "<p>…full HTML…</p>",
    "bodyText": "Daron Acemoglu…",
    "wordCount": 7414,
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/…",
    "tags": [],
    "authors": [{ "id": 8243895, "name": "Noah Smith", "handle": "noahpinion", "bio": "Econ blogger", "twitter": "Noahpinion" }],
    "reactionCount": 208,
    "commentCount": 89,
    "restacks": 14,
    "publicationUrl": "https://www.noahpinion.blog",
    "scrapedAt": "2026-07-19T21:03:44.118Z"
}

comment row:

{
    "rowType": "comment",
    "commentId": 296191763,
    "postId": 207376523,
    "postSlug": "book-review-power-and-progress-874",
    "parentCommentId": null,
    "depth": 0,
    "authorName": "Russ S. Chien",
    "authorHandle": "chieninsights",
    "date": "2026-07-17T04:54:30.484Z",
    "bodyText": "I haven't read this book Power and Progress, but I couldn't agree more…",
    "reactionCount": 10,
    "repliesCount": 2,
    "publicationUrl": "https://www.noahpinion.blog"
}

publication row:

{
    "rowType": "publication",
    "publicationId": 35345,
    "name": "Noahpinion",
    "subdomain": "noahpinion",
    "customDomain": "www.noahpinion.blog",
    "baseUrl": "https://www.noahpinion.blog",
    "authorName": "Noah Smith",
    "authorHandle": "noahpinion",
    "freeSubscriberCount": 454000,
    "subscriberCountText": "454K+",
    "language": "en",
    "createdAt": "2020-09-25T21:19:04.505Z"
}

🗂 Key output fields

Post identity & content: postId, slug, url, title, subtitle, description, bodyHtml, bodyText, wordCount, coverImage, tags[] Engagement: reactionCount, commentCount, childCommentCount, restacks Monetisation signals: audience (everyone / only_paid / founding), isPaid, bodyIsPreviewOnly Podcast: contentType: "podcast", podcastUrl, podcastDurationSec People: authors[] (id, name, handle, photo, bio, twitter), comment authorName / authorHandle Publication: freeSubscriberCount (exact), subscriberCountText, customDomain, authorBio, createdAt, copyright Enrichment (opt-in): contactEmail, contactWebsite, emailEnrichment on the publication row

❓ FAQ

Q: Can it scrape paywalled posts? No — paywalled posts return only the free preview Substack itself exposes, flagged bodyIsPreviewOnly: true (some return no body at all). We don't bypass the paywall or accept Substack accounts. Use onlyFree: true to skip them entirely.

Q: How do I scrape a full archive? Set maxPostsPerNewsletter: 0 and raise maxItems. Archives paginate 20 posts per API call.

Q: Does it work for newsletters on their own domains? Yes — custom domains serve the same API. Publications that moved from {name}.substack.com to a custom domain are followed automatically.

Q: Where does the subscriber count come from? The publication's own About page metadata — the exact free-subscriber figure (454000), plus the display string (454K+). A handful of publications hide it; then both fields are null.

Q: How are comment replies represented? Flattened — every comment is its own row with parentCommentId and depth, so you can rebuild the tree or just count engagement.

Q: What does email enrichment do? Opt-in: it discovers the author/publication's own website and reads its public contact pages. You're billed per email actually found, never for misses.

Q: Substack Notes / Chat? Not covered — this actor is posts, comments, and publication metadata.

💬 Support

Found an issue or need a field added? Open an issue on the actor's Issues tab — response time is typically under a day.

🛠 Additional services

Need a custom pipeline (scheduled monitoring, delivery to Sheets/BigQuery, cross-newsletter dashboards) or a scraper for another site? Contact us via the Issues tab or muhamed.didovic@gmail.com.

🔎 Explore more scrapers

• G2 Reviews + AI Intelligence
• Trustpilot Scraper
• Gumroad Scraper — Products, Sellers & Creator Email
• Naver Blog Scraper

---

⚠️ Disclaimer

This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Substack Inc. or any of its subsidiaries or affiliates. All trademarks mentioned are the property of their respective owners.

The scraper accesses only publicly available pages and public API endpoints — no authenticated endpoints, no paywall bypass, and no content behind a Substack login. Paywalled posts yield only the public preview Substack itself serves to anonymous visitors. Users are responsible for ensuring their use complies with Substack's Terms of Service, applicable data-protection law (GDPR, CCPA, etc.), and any contractual obligations of their own organisation.

---

SEO Keywords

substack scraper, scrape substack, substack api, substack newsletter scraper, substack posts scraper, substack comments scraper, substack subscriber count, newsletter scraper, newsletter data extraction, substack archive export, substack to csv, substack to json, newsletter market research, creator economy data, substack podcast scraper, newsletter monitoring, substack analytics, apify substack, substack rss alternative, author contact email finder