Substack Publication & Posts Scraper

What is Substack Publication & Posts Scraper?

Substack Publication & Posts Scraper is an Apify Actor that reads any Substack publication's own public JSON archive API (<publication>.substack.com/api/v1/archive) — the same endpoint the publication's own archive page calls in your browser — with an RSS-feed fallback for the rare case the API answers something unexpected. Give it a subdomain handle (bariweiss), a full *.substack.com host, or any post/publication URL — including a custom domain a paid Substack has moved to — and it returns one clean row per post: title, author, publish date, paywall status, word count, reactions and comment count, with an option to also pull each post's full article text. No API key, no login, no proxies.

Why use Substack Publication & Posts Scraper?

• Media analysts and researchers track a newsletter's output — cadence, word count, paywall mix — across any number of publications in one run.
• Content and competitive intelligence teams pull a competitor's recent post titles, subtitles and engagement (likes, comments) without opening every post.
• RAG and archival pipelines turn a publication's back catalog into structured rows, with full article text available via includeBody.
• Newsletter growth teams measure their own publication's free-vs-paid post mix and engagement over time.

How to use Substack Publication & Posts Scraper

1. Fill Publications with one or more Substack identifiers — a bare handle (bariweiss), a full subdomain (bariweiss.substack.com), or any post/publication URL, including a custom domain a Substack has moved to.
2. Set Max posts per publication (default 50) — the Actor pages through the publication's own archive API in batches of 50 until it collects this many posts.
3. Turn on Include full article text to fetch each post's own page and extract its article body into bodyText (one extra request per post — leave it off for a faster, metadata-only run).
4. Turn on Free posts only to skip anything marked paywalled and keep only posts anyone can read.
5. Click Start and export the dataset as JSON, CSV, Excel or HTML from the Output tab.

Example input

{
    "publications": ["bariweiss"],
    "maxPosts": 50,
    "includeBody": false,
    "freeOnly": false
}

Example output

{
    "publication": "bariweiss",
    "publicationName": "The Free Press",
    "postId": 215282915,
    "title": "The Founder Building an Actual Unicorn. Plus . . .",
    "subtitle": "The man fixing San Francisco. Remembering 'Band of Brothers.' Suzy Weiss takes on Hunter Biden's meme coin. And more!",
    "slug": "the-founder-building-an-actual-unicorn",
    "url": "https://www.thefp.com/p/the-founder-building-an-actual-unicorn",
    "publishedAt": "2026-09-12T10:01:48.809Z",
    "author": "The Free Press",
    "isPaywalled": false,
    "type": "newsletter",
    "description": "In today's Weekend Press: the founder building a unicorn...",
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/6ba5f918.jpeg",
    "wordCount": 1050,
    "likeCount": 68,
    "commentCount": 12,
    "bodyText": null,
    "error": null,
    "scrapedAt": "2026-09-13T20:45:27.640Z"
}

Data table

Substack Publication & Posts Scraper extracts up to 19 fields per row:

Field	Type	Description
publication, publicationName	string	The identifier scraped and the publication's display name
postId, title, subtitle, slug	number, string	Internal post id, title, subtitle/dek and URL slug (postId/subtitle are null via the RSS fallback)
url, publishedAt, author	string	Canonical post URL, publish time (ISO 8601 UTC) and byline(s)
isPaywalled, type	boolean, string	Whether the post is subscriber-only, and its type (newsletter, podcast, thread)
description, coverImage	string	Short excerpt and the post's cover image, when it has one
wordCount, likeCount, commentCount	number	Approximate word count, total reactions and total comments (likes/comments are null via the RSS fallback)
bodyText	string	Full article text, only populated when includeBody is on
error, scrapedAt	string	Set when the publication could not be read (all other fields null), and the scrape timestamp

Input parameters

Parameter	Type	Default	Description
publications	array	["bariweiss"]	Subdomain handles, *.substack.com hosts or any post/publication URL
maxPosts	integer	50	Max recent posts to collect per publication (1-1000)
includeBody	boolean	false	Fetch each post's page and extract its full article text
freeOnly	boolean	false	Skip posts marked paywalled/subscriber-only
maxConcurrency	integer	3	Publications scraped in parallel (1-20)

Pricing

Substack Publication & Posts Scraper uses pay-per-event pricing: $0.001 per result row, plus a negligible actor-start fee. Reading the latest 50 posts from one publication costs about five cents; ten publications at 100 posts each costs about a dollar. Turning on includeBody adds one lightweight request per post but does not change the per-row price. Set Maximum cost per run and the Actor trims the work to what the budget covers instead of overspending.

Substack Publication & Posts Scraper vs. browsing the archive page

Substack's own archive page loads posts a page at a time in a UI built for reading, not exporting, and it has no built-in bulk-export button. This Actor reads the same public JSON the archive page itself calls, and turns any number of publications into one flat, typed row per post — ready to schedule, filter or feed into a content, analytics or RAG pipeline.

Using Substack Publication & Posts Scraper with AI agents and MCP

Substack Publication & Posts Scraper is pay-per-event with limited permissions — the two requirements for an Actor to be callable through the Apify MCP server at mcp.apify.com. An agent passes a publications list and gets structured post rows back, ready to summarise a newsletter's recent output, check whether a new post since a given date was free or paywalled, or pull full article text for a downstream summarisation step.

FAQ

Does this need a Substack account or API key? No. It reads <publication>/api/v1/archive, the same public JSON endpoint a browser loads when you open a publication's own archive page — no login, no key.

What if the publication moved to a custom domain? Give it the custom domain directly (https://www.thefp.com or any post URL on it) and the Actor queries that domain's own archive API. A bariweiss-style handle still works too — Substack's own redirect from bariweiss.substack.com resolves to the custom domain transparently.

When does the RSS fallback kick in, and what's different about it? Only when the archive API itself answers something other than valid JSON (rare). RSS-sourced rows have no numeric postId, subtitle, likeCount or commentCount (RSS does not carry them) and isPaywalled stays null, since RSS gives no reliable paywall signal. The Actor logs when this happens.

Why is bodyText null even though I turned on includeBody? A request to the post's own page failed, or the post is fully paywalled and Substack renders no readable preview text for it.

How accurate is isPaywalled? From the archive API it reflects Substack's own audience field (everyone vs. restricted) and is reliable. It is only null when the RSS fallback was used and the signal is unavailable.

Is this legal to run? Yes. <publication>/api/v1/archive and <publication>/feed are public, unauthenticated endpoints a Substack publication's own pages call client-side for anyone visiting them; no login wall is bypassed.

Related Actors

Part of the webdatatools web-intelligence suite — every Actor is pay-per-event, runs without proxies or a headless browser, and returns one clean row per entity:

Browse the whole suite at webdatatools, or call ten of these Actors straight from Claude, Cursor or Cline with the webdatatools MCP server.

Website & domain intelligence

• Website Contact & Social Extractor — e-mails, phones and social profiles per domain
• Website Tech Stack Detector — CMS, e-commerce, analytics, pixels and payments per domain
• Domain DNS & Email Security Checker — SPF, DKIM, DMARC, MX provider, registrar and domain age
• Domain Security Audit — TLS expiry, security headers, redirect chain, robots and llms.txt
• Subdomain Finder (Certificate Transparency) — every subdomain seen in CT logs, with a live DNS check
• Bulk Core Web Vitals & PageSpeed Audit — Lighthouse scores, LCP, CLS, INP and top fixes per URL
• On-Page SEO Audit — title, meta, headings, links, images and schema issues per page
• Sitemap URL Extractor & Change Monitor — every sitemap URL, or new and removed pages between runs
• Wayback Machine Snapshot & Page Change Tracker — how a page changed over time, or every archived snapshot
• Bulk Domain WHOIS & RDAP Lookup — registrar, dates, status and nameservers per domain

Content for AI, LLMs and RAG

• AI Web Search & Read — a query turned into clean Markdown from the top search results
• Website to Markdown Crawler for LLM & RAG — any site as clean Markdown per page, no browser
• Article & News Extractor — clean article text, author, date and Markdown per URL
• Structured Data & JSON-LD Extractor — Schema.org and Open Graph data from any page
• Google News Scraper — news results by keyword, topic or site
• Press Release Monitor — PR Newswire, Business Wire and GlobeNewswire releases

Search, video and social

• Google Search Results Scraper — organic SERP results per keyword and country
• YouTube Comments Scraper — comments and replies with likes, no API key
• YouTube Channel Latest Videos — the latest 15 videos of any channel from RSS
• YouTube Channel Videos Scraper — a channel's full video, shorts and stream list
• YouTube Search Results Scraper — videos, channels and playlists per query
• YouTube Video Details Scraper — views, likes, description, tags and chapters per video
• Apple Podcasts Lookup & Episodes Scraper — podcast metadata and episodes from iTunes and RSS
• Bluesky Scraper — posts, profiles, followers and threads from the AT Protocol API
• Telegram Channel Posts Scraper — posts, views and media flags from any public channel

Leads, jobs and company data

• Company 360 — one row per domain: contacts, tech, security, hiring and company facts
• Hiring Signals Scraper — open jobs and hiring velocity from 10 public ATS boards
• Y Combinator Companies & Founders Scraper — YC startups by batch, industry and hiring status
• Wikidata Entity & Company Enrichment — HQ, founders, employees, revenue and social IDs per company
• Bulk Email Validator — syntax, MX, disposable, role and free-provider checks
• OpenStreetMap POI Extractor — shops and amenities by radius, bbox or area
• Stock, Crypto & FX Quotes — one row per symbol from Yahoo, Binance and ECB rates
• Remote Jobs Aggregator (RemoteOK, WWR, Hacker News) — one clean row per remote job, de-duplicated across feeds

Developer, app and research data

• npm, PyPI & Crates.io Package Health Checker — releases, downloads, deprecation and a health score
• GitHub Repository Health & Activity Report — stars, commits, contributors and risk flags per repo
• VS Code Marketplace Extension Scraper — installs, ratings and versions per extension
• Chrome Web Store Extension Scraper — users, rating, version and developer per extension
• Google Play Store Scraper — apps, ratings, installs, developer contact and reviews
• App Store (iOS) App Metadata & Top Charts — ratings, price, version and charts per app
• CrossRef DOI & Citation Metadata Lookup — papers, authors, journals and citation counts
• FDA Recalls & Adverse Events Monitor — food, drug and device recalls from openFDA
• iCal / ICS Calendar Feed to Events Extractor — any public calendar feed as event rows
• Shopify Store Products Scraper — catalog, prices, variants and stock per store
• Hacker News Search & Front Page Scraper — stories, comments and points by query or front page
• GitHub Trending Repositories Scraper — trending repos and developers by language and period
• Stack Overflow & Stack Exchange Q&A Scraper — questions, answers and scores by query, tag or site

Support and feedback

Found a publication layout or edge case it should handle better? Open an issue on the Issues tab.