Substack Scraper — Free Posts & Publications avatar

Substack Scraper — Free Posts & Publications

Pricing

$1.80 / 1,000 free post scrapeds

Go to Apify Store
Substack Scraper — Free Posts & Publications

Substack Scraper — Free Posts & Publications

Extract free Substack posts from publication URLs, custom domains or publication search. Get clean text, HTML, bylines, dates and engagement. No paywall bypass.

Pricing

$1.80 / 1,000 free post scrapeds

Rating

0.0

(0)

Developer

abhinav subedi

abhinav subedi

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

4 hours ago

Last modified

Features

Limited permissions

Share

Extract publicly available free Substack posts from newsletter subdomains and custom domains. Each row contains publication identity, post metadata, author bylines, engagement and optional full HTML/plain text. Export from Apify as JSON, CSV, Excel or through the dataset API.

Quick start

{
"publicationUrls": ["astralcodexten.substack.com", "https://www.lennysnewsletter.com", "noahpinion.substack.com"],
"maxPostsPerPublication": 10,
"publishedAfter": "2026-01-01",
"includeBody": true
}

Provide at least one publication URL or subdomain. Custom domains must actually host a Substack publication; not every newsletter does.

Inputs

FieldDefaultMeaning
publicationUrlsemptyArray of bare Substack subdomains, HTTPS publication URLs or custom domains
maxPostsPerPublication10Maximum accepted free posts per publication (1–1000)
publishedAfter, publishedBeforenoneInclusive ISO dates/timestamps; date alone means midnight UTC
includeBodytrueInclude bodyText and bodyHtml; false omits both fields
requestDelayMillis1500Pause before each request, at least 1000ms
maxArchivePages100Safety cap per publication, up to 1000 pages of 12

Sample output

{
"publicationName": "Astral Codex Ten",
"publicationUrl": "https://www.astralcodexten.com",
"publicationSubdomain": "astralcodexten",
"title": "The AI Forecasting Agenda",
"subtitle": "...",
"url": "https://www.astralcodexten.com/p/the-ai-forecasting-agenda",
"slug": "the-ai-forecasting-agenda",
"publishedAt": "2026-11-19T20:29:45.205Z",
"audience": "everyone",
"bylines": [{"name":"Scott Alexander","handle":"astralcodexten","profileUrl":"https://substack.com/@astralcodexten"}],
"reactions": {"❤":141},
"likeCount": 141,
"commentCount": 139,
"wordCount": 3638,
"coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/4125aad4-afa1-465e-ac49-4f4926beff76_1536x1024.png",
"bodyText": "Full freely accessible article text…",
"bodyHtml": "<p>Full freely accessible article HTML…</p>"
}

Counts change over time. Missing publication metadata falls back to the input host; the subdomain is null for unidentified custom domains. likeCount uses Substack's reaction count, with the heart count as fallback. Byline objects contain only name, handle and profile URL: no author email, biography, subscriber profiles or comments are collected. Body content remains publisher-authored public content and may itself mention people.

Pricing

Target price: $0.0018 per result ($1.80 per 1,000 posts), one result charge event per saved post, with or without bodies. No charge events for skipped paid posts, errors or empty results. This is below the popular competitor's $0.002–$0.0023 full-content price checked at launch. Apify's live pricing panel is authoritative; monetization requires the owner's payout/billing setup and may remain unavailable until completed. Platform usage charges may apply as shown by Apify.

Limits, reliability and responsible use

  • No credentials, subscriber cookies, paywall bypass, or paid previews. Only audience: everyone detail responses with a nonempty body and no truncation/paywall flags are accepted. This check runs even when bodies are omitted. Archive excerpts (truncated_body_text) are never used as full content.
  • One request at a time. Requests pause between calls; HTTP 429 and server errors retry up to three times with exponential backoff and respect Retry-After. Access-denied responses are not bypassed.
  • Archives are requested in pages of 12 with sort=new, an empty search parameter and an offset. If the public archive rejects an offset with HTTP 400, the scraper continues through public previous_post_slug links. Both paths obey the same free-content checks, rate limit and scan bound. Substack's undocumented endpoints can change; failures yield partial results with explicit errors. A bounded scan can return fewer posts than requested. Repeated pages and links are detected.
  • Dataset URLs are deduplicated within a run. No stored history across runs. Metadata-only or empty-body posts are intentionally skipped.
  • Inspect the SUMMARY key-value record for result/skipped counts and per-publication errors. A run with errors and no results fails; a legitimately empty archive succeeds with zero results.
  • Public access does not grant republication rights. Respect publisher terms, copyright, privacy and applicable law. Use for permitted research/analysis; do not redistribute full articles without permission. Render untrusted HTML safely.

Development

Node.js 22+. npm ci, npm test, then apify run --purge --input-file example-input.json. Deploy only this directory using apify push. The folder-local publish.mjs configures only this Actor, never other Actors in the account.