Substack Scraper — Newsletter Posts, Engagement & Analytics Data

Extract structured post and engagement data from any public Substack newsletter — posts, authors, publish dates, likes, comments, restacks, subscriber estimates, and archive URLs — using Substack's own public archive JSON API. No browser, no login, no proxy needed.

Use This Actor For

• Newsletter sponsorship prospecting — rank publications by engagement metrics (likes, comments, restacks) and filter free vs paid audiences before pitching
• Creator-economy lead generation — build author lists with handles, bios, bylines, and post history for outreach and PR campaigns
• Competitive newsletter monitoring — track competitor publications for new posts, publishing cadence, and engagement trends
• Audience research — analyze topics, posting frequency, and which posts resonate in any Substack niche
• Newsletter market intelligence — compile structured datasets of Substack publications for market sizing and category analysis
• Content strategy research — identify high-performing post formats, topics, and engagement patterns across newsletters

How It Works

1. Point the actor at one or more Substack publication URLs (both xxx.substack.com subdomains and custom domains work).
2. It fetches the homepage to extract publication metadata, then pages through the internal archive endpoint to collect posts.
3. Each post is returned as a structured JSON record enriched with engagement metrics, author bylines, and publication context.
4. Optionally enable full body text extraction for free posts (adds one request per post).

Runs entirely on HTTP requests — no headless browser, no Cloudflare bypass, no DOM scraping. Uses the same JSON endpoints Substack's own web client calls.

Extract Real Substack Newsletter Data for Sponsorship Research

Field	Type	Required	Default	Notes
publicationUrls	string[]	Yes	—	Substack publication URLs. Works with xxx.substack.com subdomains and custom domains (e.g. https://www.lennysnewsletter.com).
maxPostsPerPublication	integer	No	50	Maximum number of posts to extract per publication.
maxTotalPosts	integer	No	200	Maximum total posts across all publications. Free users are capped at 10 posts per run.
sortOrder	string	No	"new"	Archive order: new (newest posts first) or top (most engaged posts first).
searchQuery	string	No	""	Optional keyword filter applied server-side to the publication archive (e.g. "AI"). Leave empty for all posts.
audienceFilter	string	No	"all"	Filter by paywall status: all, free (non-paywalled), or paid (paywalled posts).
includePostBodies	boolean	No	false	Fetch full body text for free (non-paywalled) posts. Costs one extra request per post. Paywalled bodies are never fetched.
maxRequestsPerCrawl	integer	No	300	Safety cap on total HTTP requests for the whole run.
proxy	object	No	{"useApifyProxy":true}	Proxy settings. Apify proxy is ON by default so your IP is never exposed.

Build Structured Substack Databases Without Official API Access

Each dataset item represents one post with the following fields:

Field	Description
publicationName	Newsletter name (from embedded page state).
publicationUrl	Canonical publication URL.
publicationTagline	Publication tagline/description.
publicationLogo	Publication logo URL.
publicationLanguage	Publication language code.
subscriberEstimate	Raw "X subscribers / readers" hero copy where the publication shows it; null otherwise.
archiveUrl	Public archive URL of the publication.
postId	Substack internal post ID.
title	Post title.
subtitle	Post subtitle.
slug	URL slug.
author	Primary author name.
authorHandle	Author Substack handle.
authorBio	Author biography.
authorPhoto	Author profile photo URL.
guestAuthors	Array of guest author names (if multi-author post).
publishDate	ISO 8601 publish timestamp.
audience	Paywall status (everyone = free, only_paid).
isPaid	Boolean: true if post is behind a paywall.
likes	reaction_count from Substack.
reactions	Per-emoji reaction breakdown where present.
comments	comment_count + child_comment_count.
restacks	Restack (repost) count.
wordCount	Post word count.
section	Post section/category.
description	Post description/summary.
coverImage	Cover image URL.
url	Canonical post URL.
type	Post type (newsletter, podcast, thread, etc.).
scrapedAt	ISO 8601 timestamp of when the data was scraped.

Example Input

{
  "publicationUrls": [
    "https://astralcodexten.substack.com",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 50,
  "maxTotalPosts": 200,
  "sortOrder": "new",
  "audienceFilter": "all",
  "includePostBodies": false
}

Example Output

{
  "publicationName": "Astral Codex Ten",
  "publicationUrl": "https://www.astralcodexten.com",
  "author": "Scott Alexander",
  "authorHandle": "astralcodexten",
  "title": "Your Book Review: The Tale Of Genji",
  "publishDate": "2026-09-04T19:34:24.551Z",
  "audience": "everyone",
  "isPaid": false,
  "likes": 165,
  "comments": 75,
  "restacks": 6,
  "reactions": { "\u2764\uFE0F": 142, "\uD83D\uDE02": 23 },
  "subscriberEstimate": null,
  "archiveUrl": "https://www.astralcodexten.com/archive",
  "url": "https://www.astralcodexten.com/p/your-book-review-the-tale-of-genji",
  "wordCount": 3200,
  "type": "newsletter",
  "scrapedAt": "2026-09-06T18:30:00.000Z"
}

Schedule Runs for Ongoing Newsletter Monitoring

• API — call the actor via the Apify REST API for programmatic data pipelines.
• Webhooks — connect to Zapier, Make, or n8n to trigger downstream workflows on new data.
• Scheduling — set up daily or weekly scheduled runs with a small maxPostsPerPublication to detect new posts cheaply.

Recurring usage improves Apify Store recommendation signals and keeps your newsletter monitoring data fresh.

Related Actors

• Substack Newsletter Scraper — more comprehensive scraping with comments, Notes, and facepile data (50+ fields per post).
• Substack Scraper — includes monitor/incremental mode and single-post detail mode.
• Substack Lead Gen Scraper — B2B lead generation focused on author profiles, subscriber counts, and sponsorship signals.
• Substack Scraper by samwise.agency — lightweight scraper with engagement metrics and custom domain support.
• Substack Newsletter & Creator API — creator-focused with social link discovery and recent post metrics.

FAQ

Why use this actor instead of the Substack API?

Substack does not publish an official public content API for third-party developers. This actor uses the same internal JSON endpoints (/api/v1/archive, embedded homepage JSON) that Substack's own web client calls — no browser rendering, no authentication, no Cloudflare bypass required. It handles pagination, rate limiting, data normalization, and custom domain redirects automatically.

What are alternatives to scraping Substack data?

Alternatives include: (1) Substack's built-in export feature for your own publication's subscriber and post data, (2) third-party analytics tools like StackStats or Metrisub that read your Substack CSV exports, (3) building a custom scraper using Substack's hidden JSON endpoints directly, or (4) using the Python scrape-substack package. This actor automates the extraction of other publications' public data at scale, which none of these alternatives do.

Can this actor scrape paywalled post content?

No. The actor only collects publicly visible metadata for paywalled posts (title, author, publish date, audience flag). Full body text is extracted only for free (non-paywalled) posts when includePostBodies is enabled. Substack's archive API returns only the free preview for paid posts.

Does this actor work with custom-domain Substack publications?

Yes. Both xxx.substack.com subdomain URLs and custom domain URLs (e.g. https://www.lennysnewsletter.com) are supported. The actor follows redirects from subdomain to custom domain automatically and builds API URLs from the canonical host.

How much does it cost to scrape Substack?

The actor is HTTP-only (no browser) and runs at 256 MB memory. A typical run uses ~1 request for the homepage plus ~1 request per 12 posts (Substack's page size), so 100 posts costs approximately 10 requests. A few thousand posts cost a fraction of a dollar in Apify compute. Free-plan users get a capped 10-post preview per run.

How does subscriberEstimate work?

Substack does not expose subscriber counts in any public JSON endpoint. The actor best-effort matches "X subscribers / readers" hero copy from the publication homepage HTML and returns null when the publication hides this information. Most publications do not show subscriber counts publicly.

Can I discover Substack publications by keyword or topic?

No. Substack's global search API requires a logged-in session. This actor covers public data only — you provide the publication URLs you want to analyze. For discovery, try the Substack Lead Gen Scraper which supports category-based discovery.

Can I filter posts by topic or keyword?

Yes. Use the searchQuery input field to apply a server-side keyword filter to a publication's archive. For example, set searchQuery: "AI" to only return posts matching that topic. This reduces request volume compared to fetching all posts and filtering client-side.

Can I sort by most-engaged posts?

Yes. Set sortOrder: "top" to retrieve posts sorted by engagement (most likes/comments first) instead of chronological order. This is useful for identifying top-performing content for sponsorship analysis or content strategy research.

For AI Agents & LLM Apps

Purpose: Extract structured post-level data and engagement metrics from public Substack newsletters via Substack's own archive JSON API. Returns one JSON record per post with publication metadata, author bylines, engagement counts, and paywall status.

Minimal working input:

{
  "publicationUrls": ["https://astralcodexten.substack.com"]
}

Variant — topic-filtered extraction:

{
  "publicationUrls": ["https://astralcodexten.substack.com"],
  "searchQuery": "AI",
  "audienceFilter": "free",
  "sortOrder": "top"
}

Output fields: publicationName, publicationUrl, author, authorHandle, authorBio, title, subtitle, publishDate, audience, isPaid, likes, comments, restacks, reactions, subscriberEstimate, archiveUrl, url, wordCount, type, scrapedAt

Behaviors an agent should know:

• publicationUrls is the only required field — all other fields have sensible defaults.
• Free users are hard-capped at 10 posts per run by Substack, regardless of maxPostsPerPublication setting.
• includePostBodies: true roughly doubles request volume and cost — only enable when body text is needed.
• subscriberEstimate is null for most publications (Substack hides counts in public JSON).
• sortOrder: "top" may return fewer posts than sortOrder: "new" for some publications.
• maxTotalPosts is a hard ceiling across all publications; maxPostsPerPublication is per-publication.
• Custom domain publications must be called on their canonical host — the actor follows the redirect automatically.

Billing model: Pay-per-event. $0.005 per actor start + $0.0025 per result (per post returned).

Legal & Compliance Disclaimer

This actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Substack, Inc. It accesses only publicly available pages and uses Substack's own unauthenticated archive JSON API endpoints — no login bypass, no paywall circumvention, no CAPTCHA solving. Users are responsible for their own compliance with Substack's Terms of Service and applicable data-protection regulations (including GDPR and CCPA where relevant) when processing the extracted data. The engagement data returned (likes, comments, restacks) reflects publicly visible metrics at the time of scraping and should not be treated as private analytics.

SEO Keywords

substack scraper, substack data extraction, substack newsletter data, substack api alternative, scrape substack newsletters, substack post analytics, substack engagement metrics, substack newsletter research, substack sponsorship prospecting, substack creator lead generation, newsletter data scraping, substack archive scraper, substack publication data, substack newsletter analytics, substack scraper apify, scrape substack api, substack newsletter database, substack content analysis, newsletter competitive intelligence, substack post archive, substack newsletter scraping tool, substack custom domain scraper, newsletter data extraction api, substack restacks comments likes