📰 Substack Scraper: Posts, Full Text, Reactions & Paywall

⚡ Pure HTTP. No login, no API key, no proxy, no browser.

💸 You are only charged for a post that actually lands in your dataset. Publications that fail to load are never billed.

Overview

Substack Scraper turns any Substack newsletter into structured rows. Give it a publication name, a subdomain, or a full URL, and it returns every post with title, subtitle, author, publish date, word count, reaction count, comment count, the canonical URL, and the full post text.

It reads each publication's own archive endpoint, so there is nothing to log into and no key to manage. Custom domains work exactly like *.substack.com addresses.

✅ Any publication, including custom domains · ✅ Free and paywalled posts identified · ✅ Engagement metrics per post · ✅ MCP-ready for AI agents

What you get that a plain RSS reader does not

Substack's public RSS feed carries roughly the last 20 posts and no engagement data at all. This actor reads the archive API instead, which means:

• The whole back catalogue, not the last 20 posts, paginated 50 at a time.
• Reactions and comment counts per post, so you can rank a newsletter's archive by what actually landed.
• Paywall status on every post, via audience (everyone or only_paid) and an is_paywalled boolean.
• Word count as Substack itself reports it.

About paywalled posts

Every post carries audience, so you always know whether it is free or paid. For paid posts Substack serves the publicly available portion, which for many publications is most of the article and for others is a short preview.

We report what we actually received rather than guessing: when the delivered text is materially shorter than the word count Substack declares, the row is flagged content_truncated: true.

We do not attempt to bypass the paywall. It is the publication's business model, breaking it would be a legal problem for you, and any such trick would break the moment Substack tightened it. If you need the full text of paid posts, subscribe to the publication.

Input

{
  "publications": ["bigtechnology", "platformer.news", "https://www.astralcodexten.com/"],
  "maxPostsPerPublication": 50,
  "includeBody": true,
  "audienceFilter": ""
}

Field	What it takes
publications	A plain name (bigtechnology), a subdomain, or any full URL. Custom domains are supported.
maxPostsPerPublication	Hard cap per publication. Your main cost and runtime control.
includeBody	Fetch the full post text. This costs one extra request per post, see the note below. Turn it off for a fast metadata-only pass.
audienceFilter	Limit to free posts, paywalled posts, or leave blank for both.

Why includeBody matters for cost. The archive endpoint returns 50 posts of metadata in a single request, but it does not include post bodies: it returns an empty body_html field at every page size. Real bodies come from the per-post endpoint, one request each. So a metadata-only run is one request per 50 posts, while a full-text run is one request per post. Both are supported; pick the one your use case needs.

Output

{
  "publication": "bigtechnology",
  "post_id": "209279848",
  "title": "When Artificial Intelligence Is Too Valuable To Sell",
  "subtitle": "The frontier labs are rethinking who gets access",
  "url": "https://www.bigtechnology.com/p/when-artificial-intelligence-is-too",
  "authors": ["Alex Kantrowitz"],
  "published_at": "2026-07-31T19:30:17.712Z",
  "post_type": "newsletter",
  "audience": "only_paid",
  "is_paywalled": true,
  "word_count": 1139,
  "reactions_total": 74,
  "reactions": { "❤": 74 },
  "comment_count": 0,
  "body_text": "It's time to delete the assumption that the frontier AI labs will always license...",
  "scraped_at": "2026-08-13T13:02:44.108Z"
}

Field	Description
📰 publication	Publication name
🔑 post_id	Substack post ID, stable for deduplication across runs
📄 title / subtitle	Post headline and standfirst
🔗 url	Canonical post URL
👤 authors	Bylined authors
📅 published_at	ISO publish timestamp
🏷️ post_type	newsletter, podcast, thread and so on
🔓 audience	everyone (free) or only_paid (paywalled)
🔒 is_paywalled	Boolean form of the above
📏 word_count	Word count Substack reports for the full post
❤️ reactions_total	All reactions summed into one sortable number
😀 reactions	The raw reaction map keyed by emoji
💬 comment_count	Number of comments
📃 body_text	Post body as clean plain text
✂️ content_truncated	True when only part of a paid post was publicly available
🕒 scraped_at	ISO timestamp of capture

Why reactions appear twice. Substack returns reactions as a map keyed by emoji ({"❤": 74}), which is faithful but impossible to sort on. We keep the raw map and add a summed integer so you can rank an archive by engagement without writing your own reducer.

Common use cases

Newsletter competitive research. Pull a competitor's full archive, sort by reactions_total, and see exactly which topics their audience responded to.

Content and topic research. Search across the body text of a set of newsletters in your space to find who has covered a subject and how.

Author and market intelligence. Track publishing cadence, word counts and the free-versus-paid mix across publications to understand how a newsletter is monetising.

AI and RAG pipelines. Newsletter archives are dense, well-edited prose, which makes them strong retrieval material. Pull once, index, and keep current on a schedule.

Media monitoring. Watch a set of publications for mentions of a company, person or product.

Pricing

Event	Price	You pay when
Post delivered	$0.0015	A post lands in your dataset

$1.50 per 1,000 posts. Publications that do not exist or return nothing cost nothing.

Run it on a schedule

1. Run once with the input you want repeated, then click Save as a task.
2. In the Apify Console go to Schedules → Create new.
3. Pick a frequency and attach the saved task.
4. Wire the dataset to Sheets, Slack or a webhook from the Integrations tab.

Deduplicate on post_id so a repeated schedule never reprocesses the same post.

FAQ

Do I need a Substack account or subscription? No. The actor reads publicly available archive data. A subscription is only relevant if you want the full text of paid posts, which this actor does not bypass.

Does it work with custom domains? Yes. Many established newsletters publish from their own domain rather than *.substack.com, and both are handled.

How far back does it go? As far as the publication's archive allows, paginated 50 posts at a time up to your cap.

Why is my run slower with includeBody on? Because each post body is a separate request. See the cost note above.

Why did a paid post come back shorter than its word count? Because only part of it is public. That row is flagged content_truncated: true.

Use from Claude, ChatGPT and any MCP agent

https://mcp.apify.com/?tools=themineworks/substack-scraper

Or call it programmatically:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('themineworks/substack-scraper').call({
  publications: ['bigtechnology', 'platformer.news'],
  maxPostsPerPublication: 50,
  includeBody: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Questions, or need a field we don't return yet? Reach out through the Apify profile.