🔥 Substack Publication Scraper

Pull posts from any public Substack publication — no API key, no login. The Actor uses Substack's own open JSON feed (/api/v1/posts), so you get clean, structured data straight from the source. Point it at a newsletter and get back 35+ fields per post, plus publication metadata.

What you get

For every post:

• Content: title, subtitle, social_title, description, truncated_body_text, slug, url, cover_image
• Timing: post_date, updated_at, language
• Engagement: reaction_count, reactions (full map), comment_count, child_comment_count, restacks
• Access / type: type, audience, is_paid, is_free, is_preview
• Reading: wordcount, reading_time_minutes
• Taxonomy: tags, section_id, publication_id, id
• Authors: authors (names) and authors_detailed (name, handle, bio, photo, Twitter)
• Audio / podcast: has_audio, audio_url, has_podcast, podcast_url, podcast_duration
• Full body (optional): body_html and plain-text body_text — enable includeBody
• Publication: a publicationMeta object on each row — name, description, author, logo_url, language, url

What it's good for

• Tracking a competitor's or your own publishing cadence and engagement (likes, comments, restacks).
• Building a content dataset (titles, dates, tags, wordcount, full text) for analysis or LLM pipelines.
• Monitoring several newsletters at once and exporting to CSV / Excel / JSON / API.

Input

Field	Type	Description
publications	array of strings	Required. One or more publications. Accepts a bare subdomain (platformer), a full URL (https://newsletter.pragmaticengineer.com) or a custom domain (www.astralcodexten.com). Normalized for you.
maxPosts	integer	Max posts to fetch per publication. Default 50 (1–5000). Paginates until reached or the archive ends.
includePublicationMeta	boolean	Attach publicationMeta to every post (one extra request per publication). Default true.
includeBody	boolean	Also fetch the full article body (body_html + body_text). One extra request per post, so runs take longer. Default false.
maxConcurrency	integer	How many publications to scrape in parallel. Default 3 (1–10).
timeoutSecs	integer	Timeout per request in seconds. Default 30 (5–120).
maxRetries	integer	Retries per failed request (network error, timeout, 429, 5xx) with exponential backoff. Default 3 (0–8).

Example input

{
  "publications": [
    "https://newsletter.pragmaticengineer.com",
    "platformer",
    "www.astralcodexten.com"
  ],
  "maxPosts": 25,
  "includeBody": false,
  "maxConcurrency": 3
}

Output

One dataset item per post:

{
  "publication": "newsletter.pragmaticengineer.com",
  "title": "The Pulse: Firebase’s global outage & poor response",
  "subtitle": "Also: OpenAI’s platform play…",
  "slug": "the-pulse-firebases-global-outage",
  "url": "https://newsletter.pragmaticengineer.com/p/the-pulse-firebases-global-outage",
  "post_date": "2026-10-01T16:43:19.985Z",
  "type": "newsletter",
  "audience": "only_paid",
  "reaction_count": 53,
  "comment_count": 3,
  "authors": ["Gergely Orosz"],
  "description": "…",
  "cover_image": "https://substackcdn.com/image/fetch/…png",

  "id": 218367831,
  "publication_id": 458709,
  "section_id": 24961,
  "social_title": "The Pulse: Firebase’s global outage & poor response",
  "restacks": 1,
  "child_comment_count": 2,
  "reactions": { "❤": 53 },
  "is_paid": true,
  "is_free": false,
  "is_preview": true,
  "wordcount": 2889,
  "reading_time_minutes": 14,
  "language": "en",
  "updated_at": "2026-10-03T20:34:07.897Z",
  "tags": ["Google", "OpenAI"],
  "authors_detailed": [
    { "name": "Gergely Orosz", "handle": "pragmaticengineer", "bio": "Big Tech and startups…", "photo_url": "https://…", "twitter_screen_name": "GergelyOrosz" }
  ],
  "has_audio": true,
  "audio_url": null,
  "has_podcast": false,
  "body_html": null,
  "body_text": null,
  "publicationMeta": {
    "name": "The Pragmatic Engineer",
    "description": "Big Tech and startups, from the inside…",
    "author": "Gergely Orosz",
    "logo_url": "https://substackcdn.com/image/…300x300.png",
    "language": "en-us",
    "url": "https://newsletter.pragmaticengineer.com"
  }
}

If a publication can't be read (private, invite-only, wrong host, network error), the Actor pushes a single row { "publication": "<host>", "error": "<reason>" } and moves on to the next one — it never crashes the whole run.

Notes

• Some publications are private or invite-only and will return an error row instead of posts.
• audience values from Substack include everyone (free), only_paid and only_free. is_paid / is_free are convenience flags derived from it.
• The feed is ordered newest-first, so maxPosts gives you the most recent N posts.
• includeBody returns the full text of free/public posts; for paid posts it returns the public preview Substack serves without login.
• Robust by default: failed requests are retried with exponential backoff before an error row is emitted.

Pricing

Pay-per-result: you are charged one dataset-item event per post returned. Error rows are not charged as posts, and publication metadata / full-body fetching do not add charges.