Substack Scraper

Substack Scraper

Substack Scraper extracts posts from any public Substack publication without a Substack account, API key or browser. It reads the same public JSON endpoints that power Substack's own archive pages (/api/v1/archive and /api/v1/posts/<slug>), walks back through the archive page by page, and returns one clean JSON record per post: title, subtitle, authors, publication date, post type, full text and HTML, paywall flag, word count, cover image, likes, restacks, comment count, tags and, optionally, the public comment thread.

Typical uses: monitoring newsletters in your niche, building datasets for research or NLP, feeding posts into an AI agent or n8n/Make automation, archiving a publication, tracking competitors, or syncing new posts into a database or CMS.

The scraper works with publications on *.substack.com and on custom domains (for example www.astralcodexten.com or www.noahpinion.blog); subdomains that moved to a custom domain are followed automatically. Paywalled posts return only the free preview that Substack shows to logged-out readers, flagged with isPaywalled: true. Private (invite-only) publications and unknown subdomains produce a status record (private, not_found) instead of failing the run.

How to scrape a Substack publication

1. Open the Actor in Apify Console and click Try for free.
2. In Publications, add one or more publications. Any of these forms work: noahpinion, noahpinion.substack.com, https://www.astralcodexten.com, https://www.lennysnewsletter.com/p/some-post.
3. Set Max posts per publication (default 10, newest first). Optionally set Since date to stop at a date, turn on Include full body to fetch each post's full text and HTML (off by default, so the run makes no per-post request), or turn on Include comments to get the public thread of every post.
4. Click Start. Results appear in the Output tab within seconds and can be downloaded as JSON, CSV, Excel or XML, or fetched through the API.

The archive is read 20 posts per page, so the default of 10 posts is one page load, and 50 posts means three page loads plus one request per post when the full body is on. Publications are processed in parallel with a short delay between requests to the same host.

How pagination works

Substack's archive endpoint lists posts newest first and accepts an offset. The Actor follows it until it has collected the requested number of posts, reaches the sinceDate cutoff, or receives an empty page (the beginning of the publication's history). Posts are de-duplicated by id, so a run that migrates between servers never produces the same post twice.

Custom domains

Many large publications serve their Substack from their own domain. You can pass either the Substack subdomain or the custom-domain URL. When a subdomain redirects to a custom domain the redirect is followed; when a custom domain is given without www. or is reachable only through its canonical URL, the Actor tries the www. variant and the homepage's canonical link before reporting not_found.

Substack Scraper pricing

This Actor uses Apify's pay-per-event model. There is no subscription and no per-result platform fee on top:

• Actor start: a flat fee of $0.00005 per run.
• Post scraped: $0.002 per post pushed to your dataset (that is $2 per 1,000 posts).
• Comment scraped: $0.0005 per comment returned inside a post's comments array (only when Include comments is on).

Status records for publications that do not exist, are private or are empty are free. You can cap spending with Maximum cost per run; when the cap is reached the Actor stops cleanly and keeps everything scraped so far. Apify's free plan credit is enough to scrape thousands of posts for evaluation.

Input example

{
    "publications": ["astralcodexten", "noahpinion", "https://www.lennysnewsletter.com"],
    "maxPostsPerPublication": 50,
    "sinceDate": "2026-01-01",
    "includeBody": true,
    "includeComments": false,
    "maxCommentsPerPost": 50,
    "proxyConfiguration": { "useApifyProxy": true }
}

Input fields

Field	Type	Default	Description
publications	array of strings	required	Subdomains, *.substack.com hosts or custom-domain URLs of public publications.
maxPostsPerPublication	integer	10	Newest N posts to collect per publication.
sinceDate	ISO date string	none	Stop once posts are older than this date.
includeBody	boolean	false	Fetch each post for its full text and bodyHtml.
includeComments	boolean	false	Fetch the public comment thread into comments.
maxCommentsPerPost	integer	50	Cap on comments (replies included) per post.
proxyConfiguration	object	Apify Proxy	Standard Apify proxy settings.

Output example

One dataset item per post:

{
    "publication": "https://www.noahpinion.blog",
    "publicationName": "Noahpinion",
    "publicationStatus": "ok",
    "postId": 219538380,
    "slug": "maybe-the-rapture-is-already-here",
    "url": "https://www.noahpinion.blog/p/maybe-the-rapture-is-already-here",
    "title": "Maybe the Rapture is already here",
    "subtitle": "In the future, we're all heavenbanned.",
    "authors": [{ "name": "Noah Smith", "handle": "noahpinion" }],
    "publishedAt": "2026-10-09T09:18:32.051Z",
    "type": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "wordCount": 1577,
    "text": "Back when I was a teenager, a lot of people were very into the Left Behind series...",
    "bodyHtml": "<div class=\"captioned-image-container\"><figure>...</figure></div><p>Back when I was a teenager...</p>",
    "coverImage": "https://substackcdn.com/image/fetch/...png",
    "likes": 458,
    "commentCount": 45,
    "restacks": 63,
    "tags": [],
    "scrapedAt": "2026-10-10T19:30:12.401Z"
}

text is a readable, markdown-ish rendering of the post (headings keep # markers, lists use - , footnote anchors become [n], subscribe buttons are removed). type is Substack's post type (newsletter, podcast, thread, video), audience is everyone or only_paid (founding tiers appear as served). When Include comments is on, comments holds { "id", "parentId", "author", "handle", "date", "body", "likes" } objects, flattened depth-first. Publications that could not be scraped produce a single record with publicationStatus set to not_found, private, empty or error and a human-readable message.

Integrations

• API: start runs and read results with the Apify API or the JavaScript and Python clients. Run the Actor synchronously with run-sync-get-dataset-items to get posts in one call.
• MCP / AI agents: the Actor is exposed through the Apify MCP server, so Claude, ChatGPT and other agents can call it as a tool with the same input.
• n8n and Make: use the Apify node or module, pick this Actor, pass the input JSON above and map the dataset items into your workflow.
• Schedules and webhooks: run it hourly or daily from Apify Schedules with a sinceDate of yesterday and let a webhook push new posts to Slack, Notion, Google Sheets, a database or a CRM.

Substack Scraper FAQ

Does it work for paywalled posts?

Partially. Substack's public API returns the free preview of paid posts (typically the first few paragraphs) to anyone who is not a paying subscriber, and that is what you get, flagged with isPaywalled: true. The Actor never logs in and cannot read the locked part.

Does it work for private or invite-only publications?

No. Those publications do not expose a public archive. The Actor returns a private status record so your workflow can skip them.

Can I scrape a publication on its own domain?

Yes. Pass the custom-domain URL (for example https://www.lennysnewsletter.com). Some publications have retired their original *.substack.com subdomain entirely; in that case pass the custom domain, because the subdomain now redirects to a profile page and the Actor reports not_found with an explanatory message.

Can I get comments?

Yes, turn on Include comments. Each post then costs one extra request and $0.0005 per comment returned, up to Max comments per post. Replies are included and flattened, with parentId pointing at the parent comment.

How fresh is the data?

Every run reads Substack's live API, so results are as current as the publication's own archive page, including likes, restacks and comment counts at the time of the run.

Can I scrape many publications at once?

Yes. Add as many publications as you like; they are processed in parallel with built-in rate limiting and automatic retries.

Related actors

Same author, same pay-per-event model and the same no-login approach. They combine well in one pipeline (for example calendar events plus prediction odds plus channel chatter around a release):

• Telegram Channel Scraper - posts, views, media and links from public Telegram channels, no login.
• Economic Calendar Scraper - Forex Factory and Investing.com macro events with UTC times, impact, actual, forecast and previous values.
• Prediction Market Odds Scraper - Polymarket and Kalshi markets with 0-1 prices, implied probabilities, bid/ask, volume and orderbooks.
• Product Hunt Scraper - daily, weekly and monthly leaderboards, topic pages, search and product launches with votes, makers and website URLs.

Legal and compliance

This Actor only reads publicly available data that Substack serves to anyone without logging in, and it does not bypass any access control or paywall. You are responsible for using the data in line with Substack's Terms of Use, the rights of the authors whose content you collect, applicable law and, where personal data is involved (for example comment authors), GDPR or other privacy regulations. Please read Apify's article on the legality of web scraping if you are unsure.

Support

Found a publication that does not parse correctly, or need an extra field? Open an issue on the Actor's Issues tab in Apify Console or contact the author through the Actor page. Bug reports that include the publication URL are usually fixed within days.