Substack Publication & Posts Scraper

What is Substack Publication & Posts Scraper?

Substack Publication & Posts Scraper is an Apify Actor that reads any Substack publication's own public JSON archive API (<publication>.substack.com/api/v1/archive) — the same endpoint the publication's own archive page calls in your browser — with an RSS-feed fallback for the rare case the API answers something unexpected. Give it a subdomain handle (bariweiss), a full *.substack.com host, or any post/publication URL — including a custom domain a paid Substack has moved to — and it returns one clean row per post: title, author, publish date, paywall status, word count, reactions and comment count, with an option to also pull each post's full article text. No API key, no login, no proxies.

Why use Substack Publication & Posts Scraper?

• Media analysts and researchers track a newsletter's output — cadence, word count, paywall mix — across any number of publications in one run.
• Content and competitive intelligence teams pull a competitor's recent post titles, subtitles and engagement (likes, comments) without opening every post.
• RAG and archival pipelines turn a publication's back catalog into structured rows, with full article text available via includeBody.
• Newsletter growth teams measure their own publication's free-vs-paid post mix and engagement over time.

How to use Substack Publication & Posts Scraper

1. Fill Publications with one or more Substack identifiers — a bare handle (bariweiss), a full subdomain (bariweiss.substack.com), or any post/publication URL, including a custom domain a Substack has moved to.
2. Set Max posts per publication (default 50) — the Actor pages through the publication's own archive API in batches of 50 until it collects this many posts.
3. Turn on Include full article text to fetch each post's own page and extract its article body into bodyText (one extra request per post — leave it off for a faster, metadata-only run).
4. Turn on Free posts only to skip anything marked paywalled and keep only posts anyone can read.
5. Click Start and export the dataset as JSON, CSV, Excel or HTML from the Output tab.

Example input

{
    "publications": ["bariweiss"],
    "maxPosts": 50,
    "includeBody": false,
    "freeOnly": false
}

Example output

{
    "publication": "bariweiss",
    "publicationName": "The Free Press",
    "postId": 215282915,
    "title": "The Founder Building an Actual Unicorn. Plus . . .",
    "subtitle": "The man fixing San Francisco. Remembering 'Band of Brothers.' Suzy Weiss takes on Hunter Biden's meme coin. And more!",
    "slug": "the-founder-building-an-actual-unicorn",
    "url": "https://www.thefp.com/p/the-founder-building-an-actual-unicorn",
    "publishedAt": "2026-09-12T10:01:48.809Z",
    "author": "The Free Press",
    "isPaywalled": false,
    "type": "newsletter",
    "description": "In today's Weekend Press: the founder building a unicorn...",
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/6ba5f918.jpeg",
    "wordCount": 1050,
    "likeCount": 68,
    "commentCount": 12,
    "bodyText": null,
    "error": null,
    "scrapedAt": "2026-09-13T20:45:27.640Z"
}

Data table

Substack Publication & Posts Scraper extracts up to 19 fields per row:

Field	Type	Description
publication, publicationName	string	The identifier scraped and the publication's display name
postId, title, subtitle, slug	number, string	Internal post id, title, subtitle/dek and URL slug (postId/subtitle are null via the RSS fallback)
url, publishedAt, author	string	Canonical post URL, publish time (ISO 8601 UTC) and byline(s)
isPaywalled, type	boolean, string	Whether the post is subscriber-only, and its type (newsletter, podcast, thread)
description, coverImage	string	Short excerpt and the post's cover image, when it has one
wordCount, likeCount, commentCount	number	Approximate word count, total reactions and total comments (likes/comments are null via the RSS fallback)
bodyText	string	Full article text, only populated when includeBody is on
error, scrapedAt	string	Set when the publication could not be read (all other fields null), and the scrape timestamp

Input parameters

Parameter	Type	Default	Description
publications	array	["bariweiss"]	Subdomain handles, *.substack.com hosts or any post/publication URL
maxPosts	integer	50	Max recent posts to collect per publication (1-1000)
includeBody	boolean	false	Fetch each post's page and extract its full article text
freeOnly	boolean	false	Skip posts marked paywalled/subscriber-only
maxConcurrency	integer	3	Publications scraped in parallel (1-20)

Pricing

Substack Publication & Posts Scraper uses pay-per-event pricing: $0.001 per result row, plus a negligible actor-start fee. Reading the latest 50 posts from one publication costs about five cents; ten publications at 100 posts each costs about a dollar. Turning on includeBody adds one lightweight request per post but does not change the per-row price. Set Maximum cost per run and the Actor trims the work to what the budget covers instead of overspending.

Substack Publication & Posts Scraper vs. browsing the archive page

Substack's own archive page loads posts a page at a time in a UI built for reading, not exporting, and it has no built-in bulk-export button. This Actor reads the same public JSON the archive page itself calls, and turns any number of publications into one flat, typed row per post — ready to schedule, filter or feed into a content, analytics or RAG pipeline.

Using Substack Publication & Posts Scraper with AI agents and MCP

Substack Publication & Posts Scraper is pay-per-event with limited permissions — the two requirements for an Actor to be callable through the Apify MCP server at mcp.apify.com. An agent passes a publications list and gets structured post rows back, ready to summarise a newsletter's recent output, check whether a new post since a given date was free or paywalled, or pull full article text for a downstream summarisation step.

FAQ

Does this need a Substack account or API key? No. It reads <publication>/api/v1/archive, the same public JSON endpoint a browser loads when you open a publication's own archive page — no login, no key.

What if the publication moved to a custom domain? Give it the custom domain directly (https://www.thefp.com or any post URL on it) and the Actor queries that domain's own archive API. A bariweiss-style handle still works too — Substack's own redirect from bariweiss.substack.com resolves to the custom domain transparently.

When does the RSS fallback kick in, and what's different about it? Only when the archive API itself answers something other than valid JSON (rare). RSS-sourced rows have no numeric postId, subtitle, likeCount or commentCount (RSS does not carry them) and isPaywalled stays null, since RSS gives no reliable paywall signal. The Actor logs when this happens.

Why is bodyText null even though I turned on includeBody? A request to the post's own page failed, or the post is fully paywalled and Substack renders no readable preview text for it.

How accurate is isPaywalled? From the archive API it reflects Substack's own audience field (everyone vs. restricted) and is reliable. It is only null when the RSS fallback was used and the signal is unavailable.

Is this legal to run? Yes. <publication>/api/v1/archive and <publication>/feed are public, unauthenticated endpoints a Substack publication's own pages call client-side for anyone visiting them; no login wall is bypassed.

Related Actors

Part of the webdatatools web-intelligence suite — every Actor is pay-per-event, reads public data without a login, and returns one clean row per entity:

Browse the whole suite at webdatatools, or call ten of these Actors straight from Claude, Cursor or Cline with the webdatatools MCP server.

Website & domain intelligence

• Email Extractor — Website Contact & Social Finder — e-mails, phones and social profiles per domain
• Tech Stack Detector — Wappalyzer & BuiltWith Alternative — CMS, e-commerce, analytics, pixels and payments per domain
• Domain DNS & Email Security Checker — SPF, DKIM, DMARC, MX provider, registrar and domain age
• Domain Security Audit (TLS, HTTP headers, redirects, robots) — TLS expiry, security headers, redirect chain, robots and llms.txt
• Subdomain Finder (Certificate Transparency) — every subdomain seen in CT logs, with a live DNS check
• Bulk Core Web Vitals & PageSpeed Audit — Lighthouse scores, LCP, CLS, INP and top fixes per URL
• On-Page SEO Audit — title, meta, headings, links, images and schema issues per page
• Sitemap URL Extractor & Change Monitor — every sitemap URL, or new and removed pages between runs
• Wayback Machine Snapshot & Page Change Tracker — how a page changed over time, or every archived snapshot
• Bulk Domain WHOIS & RDAP Lookup — registrar, dates, status and nameservers per domain
• Web Scraper — CSS Selector & Data Extractor — pull any CSS selector off any page, one row per URL
• Website Screenshot Generator — full-page or viewport PNG/JPEG screenshots of any URL

Content for AI, LLMs and RAG

• AI Web Search & Read: Google results as clean Markdown — a query turned into clean Markdown from the top search results
• Website to Markdown — Content Crawler for LLM & RAG — any site as clean Markdown per page, no browser
• Article & News Extractor (clean text, author, date, markdown) — clean article text, author, date and Markdown per URL
• Structured Data & JSON-LD Extractor (Schema.org, Open Graph) — Schema.org and Open Graph data from any page
• Google News Scraper (RSS search by keyword, topic, site) — news results by keyword, topic or site
• Press Release Monitor: PR Newswire, BusinessWire, GlobeNewswire — PR Newswire, Business Wire and GlobeNewswire releases

Search, video and social

• YouTube Shorts Scraper — Shorts from channels, hashtags and searches with view counts
• Pinterest Pins Scraper — latest pins of public Pinterest profiles and boards
• YouTube Transcript Scraper — captions and subtitles as text + timed segments, per video or channel
• Google Search Results Scraper — SERP API — organic SERP results per keyword and country
• YouTube Comments Scraper — Comments & Replies — comments and replies with likes, no API key
• YouTube Channel Latest Videos (RSS, no API key) — the latest 15 videos of any channel from RSS
• YouTube Channel Scraper (videos, shorts, live) — a channel's full video, shorts and stream list
• YouTube Search Results Scraper (videos, channels, no API key) — videos, channels and playlists per query
• YouTube Video Details Scraper (views, likes, description, tags) — views, likes, description, tags and chapters per video
• Apple Podcasts Lookup & Episodes Scraper — podcast metadata and episodes from iTunes and RSS
• Bluesky Post, Search & Profile Scraper — posts, profiles, followers and threads from the AT Protocol API
• Telegram Channel Posts Scraper — posts, views and media flags from any public channel
• Google Play Reviews Scraper — reviews, ratings, replies and app versions per app
• App Store Reviews Scraper — iOS reviews and ratings per app and country
• Google Trends Scraper — interest over time, by region, and related queries per keyword
• Google Ads Transparency Scraper — ads any advertiser runs on Google, with format and dates
• Keyword Suggestions Scraper (Google, YouTube, Amazon, Bing) — autocomplete keyword ideas from four search engines
• Bilibili Scraper (Videos, Search, Popular) — Chinese video platform: views, likes, coins, danmaku, uploader
• Mastodon Scraper (Hashtags, Accounts, Trending) — public fediverse posts by hashtag, account or trending
• Meetup Events Scraper (Search by Keyword & City) — upcoming events with RSVPs, fees, venues and groups
• Eventbrite Scraper (Events by Keyword & City) — events by keyword and city with venue, dates and organizer

Leads, jobs and company data

• Career Site Jobs API (Greenhouse, Lever, Ashby, Workday +1) — company domains in, their open jobs out, ATS detected automatically
• Workday Jobs Scraper — jobs with full descriptions from any Workday career site
• Google Maps Scraper — businesses with phone, website, address, rating and coordinates per search
• LinkedIn Jobs Scraper — job titles, companies, locations and full descriptions from LinkedIn job search
• Company 360: full company profile from a domain — one row per domain: contacts, tech, security, hiring and company facts
• Hiring Signals Scraper (Greenhouse, Lever, Ashby, Workable) — open jobs and hiring velocity from 10 public ATS boards
• Y Combinator Companies & Founders Scraper — YC startups by batch, industry and hiring status
• Wikidata Entity & Company Enrichment (facts, IDs, links) — HQ, founders, employees, revenue and social IDs per company
• Email Validator & Verifier — Bulk Email Check — syntax, MX, disposable, role and free-provider checks
• OpenStreetMap POI Extractor (Overpass API: shops, amenities) — shops and amenities by radius, bbox or area
• Stock, Crypto & FX Quotes — one row per symbol from Yahoo, Binance and ECB rates
• Remote Jobs Aggregator (RemoteOK, WWR, Hacker News) — one clean row per remote job, de-duplicated across feeds
• Greenhouse Jobs Scraper — jobs with descriptions from any Greenhouse job board
• Lever Jobs Scraper — jobs with descriptions from any Lever careers page
• Ashby Jobs Scraper — jobs, salaries and descriptions from any Ashby job board
• SmartRecruiters Jobs Scraper — jobs with descriptions from any SmartRecruiters company
• Seek Jobs Scraper (Australia & New Zealand) — Seek job ads with salary, work type and location
• Dice Jobs Scraper — US tech jobs from Dice with salary and remote flag
• AutoScout24 Scraper — European car listings with price, mileage and seller
• Rightmove Scraper — UK property for sale or rent with price and agent
• Wellfound Jobs Scraper (AngelList Startup Jobs) — startup jobs with salary and equity ranges, company size and stage
• Yandex Maps Scraper (Places, Ratings, Phones) — businesses in Russia, Türkiye and the CIS with phones, ratings, hours
• Craigslist Scraper (Listings, Prices, Locations) — listings in any area and category with price, date and coordinates
• JobStreet Scraper (Malaysia, Singapore, PH, ID + JobsDB) — JobStreet and JobsDB jobs in 6 Asian countries with parsed salaries
• InfoJobs Scraper (Spain Jobs, Salaries, Companies) — Spanish jobs with salary range, contract type and full description
• Redfin Scraper (Homes for Sale, Prices, Details) — US homes for sale or sold from any Redfin search, with price and details
• Kleinanzeigen Scraper (Ads, Prices, Locations) — German classifieds with price, VB flag, ZIP, city and seller type

Developer, app and research data

• npm, PyPI & Crates.io Package Health Checker — releases, downloads, deprecation and a health score
• GitHub Repository Health & Activity Report — stars, commits, contributors and risk flags per repo
• VS Code Marketplace Extension Scraper (installs, ratings) — installs, ratings and versions per extension
• Chrome Web Store Extension Scraper (installs, ratings) — users, rating, version and developer per extension
• Google Play Scraper — apps, ratings, installs, developer contact and reviews
• App Store (iOS) App Metadata, Ratings & Top Charts Lookup — ratings, price, version and charts per app
• CrossRef DOI & Citation Metadata Lookup — papers, authors, journals and citation counts
• FDA Recalls & Adverse Events Monitor (openFDA) — food, drug and device recalls from openFDA
• iCal / ICS Calendar Feed to Events Extractor — any public calendar feed as event rows
• Shopify Store Products Scraper — catalog, prices, variants and stock per store
• Hacker News Search & Front Page Scraper — stories, comments and points by query or front page
• GitHub Trending Repositories Scraper — trending repos and developers by language and period
• Stack Overflow & Stack Exchange Q&A Scraper — questions, answers and scores by query, tag or site
• Bulk Image Downloader — download image URLs to storage with size, dimensions and a ZIP
• Google Flights Scraper (Prices, Airlines, Stops) — flight prices, airlines, times, stops and CO2 by route and date
• Google Hotels Scraper (Prices, Ratings, Reviews) — hotel prices per night, stars, rating and reviews by city and dates
• AliExpress Scraper (Search Products & Prices) — AliExpress search results with USD price, discount and rank
• Lazada Scraper (Products, Prices, Sold, Ratings) — Lazada products in 6 countries with price, rating, units sold and seller

Support and feedback

Found a publication layout or edge case it should handle better? Open an issue on the Issues tab.