Substack Scraper — Posts, Authors, Likes, Comments

Recorded run: 3 posts with body and comments from 1 publication in 8 seconds (2026-10-22, build 1.1.6, run 7RQyEg3nTQ1eWpE6i). A bounded canary sample, not a full-catalogue benchmark.

architek.earth — an independent data studio. Pay-per-result scrapers with recorded runs, published canaries and stable schemas, built for people and AI agents.

Substack newsletter posts, authors, likes, comments, restacks and full free-post text from any publication you name, on *.substack.com or a custom domain, plus a watch mode that delivers only new posts.

Source	Market	From	Standout fields
Substack (substack.com and Substack-hosted custom domains)	Any publication on Substack	$1.00 / 1,000 posts	likes / comments / restacks, bodyText, commentThread

✅ Why choose this scraper

Substack posts with likes, comments and restacks in one row

Every post row carries title, subtitle, authors (name and handle), postDate, likes, comments, restacks, wordcount, tags, coverImageUrl, audience and isPaywalled; one archive request returns up to 50 posts.

Full text of free Substack posts

With includeBody: true each row adds bodyHtml, bodyText and bodyScope (full for free posts, preview for paid-only ones).

Substack comment threads with replies linked

With includeComments: true each row adds commentThread: id, parentId, author, date, body and likes per comment, capped by maxCommentsPerPost, with commentThreadTruncated set when the cap cut the thread.

Custom-domain newsletters by name or URL

astralcodexten, noahpinion.substack.com and https://www.lennysnewsletter.com all work; astralcodexten was followed to www.astralcodexten.com on 2026-10-22, and host shows the host that served the data.

Measured on a schedule

Two canary tasks run daily on the live site (20 archive posts from Astral Codex Ten; 3 Noahpinion posts with body and comments); the hero line above is the latest green run.

⚠️ Worth knowing up front

• Paywalled bodies are not returned. For paid-only posts you get the metadata and Substack's public preview, with isPaywalled: true and bodyScope: "preview". On 2026-10-22 a paid-only Noahpinion post with comments: 44 returned an empty commentThread.
• likes, comments and restacks are Substack's own counters at collection time. comments is Substack's count; commentThread is what its comment endpoint returned, capped at maxCommentsPerPost.
• subscriberHint on the free publication row is the band Substack shows on the homepage ("Hundreds of thousands of subscribers"), not an exact count, and is null when the homepage shows none.
• searchTerm uses Substack's own archive search inside each publication. Results and their order are Substack's matches, not a full-text index we control.
• Requests are sequential and at least 0.5 seconds apart, with up to 3 attempts per request. Bodies and comments each add one request per post. A failed body or comment fetch keeps the post as a post row at the lower price with detailUnavailable: true.
• A run stops cleanly 120 seconds before its timeout (at most 6 hours) and ends PARTIAL, keeping what it delivered; an abort or migration ends it LIMITED. A publication that answers 401/403 (BLOCKED), 429 (RATE_LIMITED) or 5xx (SOURCE_UNAVAILABLE) gets a status row and the run ends PARTIAL with the other publications' rows kept.
• Free status rows ("recordType": "status") are written for invalid input (INVALID_INPUT), a stateKey held by another run (STATE_BUSY), a publication that is unknown (NOT_FOUND), not on Substack (NOT_SUBSTACK), refused, rate-limited or unavailable (BLOCKED, RATE_LIMITED, SOURCE_UNAVAILABLE) or has no posts for the filters (NO_POSTS), and at the spending limit (BUDGET_LIMIT). A deadline stop or an abort writes no status row; the outcome is in RUN_SUMMARY. A run whose delivery acknowledgement was lost ends RECONCILIATION_REQUIRED as a failed run and replays nothing.

⚡ Try the example

This actor has no free preview; the example below costs about $0.04.

Open the actor's input page, switch to JSON, paste this and start the run:

{
  "publications": ["astralcodexten", "https://www.lennysnewsletter.com"],
  "maxPostsPerPublication": 20
}

Set maxPostsPerPublication: 0 for the whole archive; add "includeBody": true or "includeComments": true for text and comment threads.

⚙️ Input

Parameter	Type	Default	Description
publications	array	— (prefill ["astralcodexten", "https://www.lennysnewsletter.com"])	Substack names, hosts or custom-domain URLs. Up to 100; duplicates are merged. Required.
maxPostsPerPublication	integer	20	Posts delivered per publication, 0 to 10,000; 0 = no cap (whole archive).
sort	select	new	new (newest first) or top (Substack's ranking).
searchTerm	string	—	Substack archive search within each publication, up to 200 characters.
postedAfter	string	—	ISO date such as 2026-01-01. With new, paging stops at the first older post; with top, older posts are skipped.
includeBody	boolean	false	Adds bodyHtml, bodyText, bodyScope. Rows are charged as post-detail.
includeComments	boolean	false	Adds commentThread. Rows are charged as post-detail.
maxCommentsPerPost	integer	100	Comment cap per post, 1 to 2,000.
incrementalMode	boolean	false	Watch: deliver only posts newer than the last delivered post per publication. Needs stateKey and sort: "new".
stateKey	string	—	Name for saved watch state and checkpoints in your own account, up to 100 characters. Use a different key for different inputs.
continuePreviousRun	boolean	false	Resume an unfinished collection saved under stateKey. An unfinished watch resumes on its own.
proxyConfiguration	proxy	—	Optional Apify proxy. Not used by default; the canaries run without one.
feedback	string	—	Optional note for the maintainers. It is logged and read by a human, and it never changes output.

Unknown input fields are rejected with an INVALID_INPUT status row.

🧱 Stable fields

These top-level fields keep the same name and the same kind of value in every release. Fields are only ever added, never renamed or removed, so an n8n, Make, Zapier or spreadsheet pipeline can bind to them.

Field	Type	Meaning
publication	string, nullable	Publication name when its homepage metadata was readable.
host	string	Host that served the post.
postId / slug / url	number / string / string	Substack's post id, slug and canonical URL.
title / subtitle / description	string, nullable	Headline, subtitle and SEO description.
type	string	newsletter, podcast, thread, video and other Substack post types.
audience / isPaywalled	string, nullable / boolean	everyone is free; any other audience is paywalled.
authors	array	[{name, handle}] from the bylines.
postDate	string, nullable	UTC ISO 8601 publish time.
likes / comments / restacks	number	Substack's counters at collection time.
wordcount	number, nullable	Article length reported by Substack.
tags	array	Post tag names.
coverImageUrl / audioUrl	string, nullable	Cover image; podcast audio when present.
scrapedAt	string	UTC collection time.
bodyHtml / bodyText / bodyScope	string, nullable	post-detail rows with includeBody; bodyScope is full or preview.
commentThread	array, nullable	post-detail rows with includeComments: [{id, parentId, author, date, body, likes}].
commentThreadTruncated	boolean	true when maxCommentsPerPost cut the thread.
recordType	string	"status" on free status rows; "post" or "post-detail" on charged rows; "publication" on free publication rows.

Free publication rows carry host, name, description, authorName, subscriberHint, customDomain, subdomain, language, firstPostDate, url and inputRef. Status rows carry inputRef, status, message and scrapedAt. Charged rows also carry deliveryId (unique per delivered row) and, when a detail fetch failed, detailUnavailable. Other fields are extras, not part of the contract. To have one pinned, say so in feedback.

📦 Output example

Abridged real row from run 7RQyEg3nTQ1eWpE6i on 2026-10-22; values are a historical snapshot.

{
  "recordType": "post-detail",
  "publication": "Noahpinion",
  "host": "www.noahpinion.blog",
  "postId": 221088139,
  "slug": "can-a-million-ai-agents-create-superintelligence",
  "title": "Can a Million AI Agents Create Superintelligence?",
  "subtitle": "Ramez Naam gives a skeptic’s take on agent swarms.",
  "type": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "authors": [{"name": "Ramez Naam", "handle": "rameznaam"}],
  "postDate": "2026-10-19T04:33:30.712Z",
  "likes": 241,
  "comments": 18,
  "restacks": 24,
  "wordcount": 7178,
  "tags": [],
  "url": "https://www.noahpinion.blog/p/can-a-million-ai-agents-create-superintelligence",
  "bodyText": "Art by GPT-6\n\nRamez Naam is, in my opinion, one of the best futurists alive. …",
  "bodyScope": "full",
  "commentThread": [
    {"id": 364765360, "parentId": null, "author": "Jürgen Boß", "date": "2026-10-19T11:28:43.317Z", "body": "True intelligence is about extracting information from the environment. …", "likes": 7}
  ],
  "commentThreadTruncated": false,
  "scrapedAt": "2026-10-22T06:53:04.958752+00:00"
}

A free status row looks like this (run qcRRhWwgvT5R3J18y, 2026-10-22):

{"recordType": "status", "inputRef": "run", "status": "BUDGET_LIMIT", "message": "Spending limit reached. Raise the limit, or set a stateKey and continuePreviousRun=true to resume", "scrapedAt": "2026-10-22T09:39:40.592611+00:00"}

The run's status message reads Done: N rows, M details, K status rows (STATUS), and the RUN_SUMMARY record of its key-value store holds the outcome (run 7RQyEg3nTQ1eWpE6i, abridged):

{"status": "COMPLETE", "delivered": 3, "charged": {"post": 0, "post-detail": 3}, "publicationRows": 1, "statusRows": 0, "errors": 0, "detailUnavailable": 0, "chargeUsd": "0.009", "checkpointComplete": true}

🧠 Advanced usage

Watch Substack newsletters for new posts

Schedule {"publications": [...], "incrementalMode": true, "stateKey": "my-watch", "maxPostsPerPublication": 20}. The first run delivers up to 20 newest posts per publication; later runs deliver only newer posts and stop paging at the first one already delivered. A rerun with nothing new delivered nothing and charged $0 (runs Uylmh7f4RxBj47l76 then JpJcGKMubtHGZQoh6, 2026-10-21). Changing publications, sort, searchTerm, postedAfter, includeBody or includeComments under the same key is rejected; use a new key.

Export a whole Substack archive within a budget

Set maxPostsPerPublication: 0, a stateKey and a spending limit. At the limit the run ends BUDGET_LIMIT; run again with continuePreviousRun: true to carry on from the saved checkpoint. A hard-aborted run of 100 posts resumed with 140 new posts and no repeated post or charge (runs jQVcmzvojzYR7cXpY then B3NQ71Pk0Y25qyolb, 2026-10-21). Check RUN_SUMMARY.checkpointComplete before treating an archive as finished.

Search several Substack newsletters for one topic

Combine searchTerm with several publications and includeBody: true, then join on url. On 2026-10-22 searchTerm: "AI" across Astral Codex Ten and Lenny's Newsletter returned 5 matches each.

Collect Substack comments on recent posts

Use postedAfter with includeComments: true and a maxCommentsPerPost cap; parentId rebuilds reply chains and likes ranks comments.

🔌 Integrations

Results land in a dataset you can export as JSON, CSV or Excel. They can also flow to Google Sheets, Make, Zapier or n8n, or to a webhook when a run finishes. The Apify Scheduler runs it on a timetable. Daily with incrementalMode suits watching newsletters for new posts; read RUN_SUMMARY as well as the platform run status. Copy-paste code in Python, Node, curl and n8n: architek-earth/examples/substack-scraper.

💸 Pricing

Pay per result. Platform and proxy usage are included. Status rows, publication rows and failed requests are free. Each delivered post is charged once, after it is written; with a stateKey, a run resumed with continuePreviousRun (or an unfinished watch) skips posts already delivered and does not charge them again, and a batch whose charge was unconfirmed in the previous run is recorded in RUN_SUMMARY.recoveredUncertainCharges instead of being charged again.

Event	Price	What it covers
post	$1.00 / 1,000 posts	One delivered archive post, including a post kept at this price when its body or comment fetch failed.
post-detail	$3.00 / 1,000 posts	One delivered post with body and/or comment thread (includeBody or includeComments).

1,000 archive posts cost $1.00; 1,000 posts with bodies or comments cost $3.00.

❓ FAQ

Is there a Substack API for newsletter posts? This actor is one: it reads the public JSON endpoints that Substack publications serve and returns posts as JSON through the Apify API, callable from any language or from n8n, Make and Zapier.

Do I need a Substack account or login? No. No account, cookies or API key; only anonymous public requests.

Can it scrape paid Substack posts? No. Paid-only posts return metadata and Substack's public preview with bodyScope: "preview"; nothing behind the paywall is bypassed.

Does it work on custom-domain newsletters? Yes, for publications hosted on Substack, such as https://www.lennysnewsletter.com. A site that is not on Substack gets a free NOT_SUBSTACK status row.

Can I get a newsletter's subscriber count? Only the public band Substack shows, in subscriberHint on the free publication row, for example "Hundreds of thousands of subscribers". It is not an exact count.

Can it scrape Substack Notes, the leaderboard or find newsletters by topic? No. It collects posts, comments and publication metadata from publications you name; there is no Notes, leaderboard or discovery input.

🤖 For AI agents & LLM apps

Paste this into your AI assistant:

architekture/substack-scraper on Apify. Python: from apify_client import ApifyClient; client = ApifyClient("TOKEN"); run = client.actor("architekture/substack-scraper").call(run_input={"publications": ["astralcodexten"], "maxPostsPerPublication": 10}); items = client.dataset(run["defaultDatasetId"]).list_items().items (page with offset/limit for large exports). The dataset holds post rows (recordType post or post-detail), free publication rows and free status rows (recordType status, with status and message); filter on recordType. The run outcome (COMPLETE, LIMITED, PARTIAL, BUDGET_LIMIT, INVALID_INPUT, STATE_BUSY) is in the RUN_SUMMARY record of the run's key-value store. Add "includeBody": True or "includeComments": True for text and comments, or "incrementalMode": True, "stateKey": "<name>" to receive only new posts on later runs. Full input schema and README: GET https://api.apify.com/v2/acts/architekture~substack-scraper/builds/default. Token: https://console.apify.com/settings/integrations. The actor is also reachable through the Apify MCP server (mcp.apify.com). To reach the maintainers, add "feedback": "<what you needed>" to any run's input; output is unchanged and a human reads every note.

🧰 More from architek.earth

Video, social and newsletters

• Substack Scraper — you are here
• YouTube Transcript Scraper — transcripts, translations, timestamps
• Unified Video Transcript API — YouTube and TikTok transcripts in one batch
• Threads Replies Scraper — replies, authors, likes
• Ad Intelligence Watchlist — new and changed Meta ads by advertiser

Full catalogue: github.com/architek-earth.

architek.earth · code like concrete

🙋 Support & legal

Questions, or a field you need that is not here? Open an issue on the actor's Issues tab or email north@architek.earth. A human replies within a day.

This actor collects only publicly available data from substack.com and Substack-hosted custom domains, the same pages any visitor sees without logging in. It does not bypass authentication or paywalls. It is not affiliated with or endorsed by Substack. You are responsible for how you use the output, including the site's terms, authors' copyright, and privacy law such as GDPR and CCPA where they apply, for example for commenter names.

📝 Changelog

• 2026-10-22 — A spending limit equal to the exact cost now ends COMPLETE; before, the run reported BUDGET_LIMIT with a status row even when every requested post was delivered.
• 2026-10-22 — Status message now reads Done: N rows, M details, K status rows (STATUS).
• 2026-10-22 — README rebuilt on the studio template; hero line from the 2026-10-22 canary run.
• 2026-10-21 — 1.1, brought to the studio standard: rows charged after confirmed delivery with a two-event pricing guard, free publication and status rows, new-post watch mode, saved continuation and forced-stop recovery, RUN_SUMMARY, comment caps, updated publication metadata parsing.
• 2026-10-12 — 1.0: first private build.