Substack Scraper — Posts, Comments, Search & Publication Intel for AI Agents & Data Pipelines

The all-in-one Substack scraper for AI agents and data pipelines: scrape Substack posts, full article content, comment threads, global search results, and publication metadata from any Substack newsletter — including custom-domain publications. Every run returns typed JSON rows validated against a strict output schema (documented below), read straight from Substack's public JSON endpoints: fast, stable, no login, no cookies, no browser. Export as JSON, CSV, or Excel, or call it as an MCP tool.

Honest limitation: paywalled post bodies return only the free preview — Substack does not expose subscriber-only content publicly, and this actor does not log in.

Why AI agents use this Substack scraper

• Deterministic typed output — every row is validated against the dataset schema below; fields are stable across runs, safe to parse programmatically.
• Cost-predictable per-event pricing — tiny $0.005 start fee (competitors charge ~$0.09 per start) plus a small charge per row. Easy to budget for autonomous agents. Uses Apify datacenter proxy by default (included in Apify plans), so no residential proxy costs.
• One actor, five jobs — posts, full post content, comments, global search, and publication intel. No juggling four separate actor rentals.
• No auth needed — public JSON endpoints only; no credentials, cookies, or CAPTCHAs.
• Clear error semantics — bad input fails fast with a non-zero exit and a message; transient HTTP errors are retried with backoff; failures are never silent (see Error semantics).

Input schema

Field	Type	Required	Default	Description
mode	string (enum)	no	"posts"	What to scrape: posts (post metadata, optionally full content, from publication archives), comments (flattened comment threads of recent posts), search (global Substack post search), publication (publication metadata / intel).
publicationUrls	array of strings	for posts/comments/publication	—	Substack publication homepages, e.g. https://astralcodexten.substack.com or custom domains like https://www.astralcodexten.com.
queries	array of strings	for search	—	Search terms for the global Substack post search.
includeContent	boolean	no	false	Posts mode only: additionally fetch each post's full body (HTML + plain text). Slower; charged per post-content event. Paywalled posts return only the free preview.
maxItems	integer	no	50	Maximum total dataset rows per run, across all publications/queries.
proxyConfiguration	object	no	Apify datacenter proxy	Proxy for requests. Default datacenter proxy is included in Apify plans (no residential costs).

Output schema

Field presence by mode (all fields are nullable unless noted; the dataset schema enforces types on every row):

Field	Type	Modes	Description
id	integer	posts, search, publication	Post ID (posts/search) or publication ID (publication).
title	string	posts, search	Post title.
subtitle	string	posts, search	Post subtitle.
slug	string	posts, search	Post URL slug.
postDate	string (ISO 8601)	posts, search	Publish date.
type	string	posts, search	Post type (newsletter, podcast, thread, ...).
likes	integer	posts, search, comments	Reaction count (post or comment). Defaults to 0.
commentCount	integer	posts, search	Comment count on the post.
audience	string	posts, search	everyone for free posts; other values mean paywalled.
paywalled	boolean	posts, search	True when the full body is subscriber-only.
wordCount	integer	posts, search	Post word count.
description	string	posts, search, publication	Post preview description, or publication hero text.
canonicalUrl	string	posts, search	Canonical post URL.
coverImage	string	posts, search	Post cover image URL.
publicationId	integer	posts, search	Publication ID the post belongs to.
publicationUrl	string	all (null in search)	Normalized publication base URL scraped from.
authorName	string	all	Author display name (byline / commenter / publication author).
authorHandle	string	all	Author Substack handle.
authorId	integer	posts, search, comments	Author user ID.
bodyHtml	string	posts (includeContent: true)	Full post body HTML (free preview only for paywalled posts).
bodyText	string	posts (includeContent: true)	Full post body as plain text.
query	string	search	The search term that produced this row.
postId	integer	comments	ID of the post the comment belongs to.
postTitle	string	comments	Title of that post.
postSlug	string	comments	Slug of that post.
postUrl	string	comments	Canonical URL of that post.
commentId	integer	comments	Unique comment ID.
parentCommentId	integer	comments	Parent comment ID; null for top-level comments.
body	string	comments	Comment text (null if deleted).
bodyDeleted	boolean	comments	True when the comment was deleted.
date	string (ISO 8601)	comments	Comment date.
childCount	integer	comments	Number of direct replies (replies are also emitted as rows).
name	string	publication	Publication display name.
subdomain	string	publication	*.substack.com subdomain.
customDomain	string	publication	Custom domain, if any.
hostname	string	publication	Serving hostname.
authorBio	string	publication	Author bio.
logoUrl	string	publication	Logo image URL.
coverPhotoUrl	string	publication	Cover photo URL.
language	string	publication	Language code.
createdAt	string (ISO 8601)	publication	Publication creation date.
firstPostDate	string (ISO 8601)	publication	Date of first post.
freeSubscriberCount	string/integer	publication	Free subscriber count hint (often rounded).
subscriberCountOrderOfMagnitude	number	publication	Order-of-magnitude subscriber hint.
rankingDetail	string/number	publication	Substack leaderboard/ranking detail.
paymentsEnabled	string/boolean	publication	Payments state (paid subscriptions enabled).
podcastEnabled	boolean	publication	True when the publication has a podcast.
twitter	string	publication	Twitter/X screen name.
copyright	string	publication	Copyright line.

Example item — posts mode

{
    "id": 210672789,
    "title": "Your Book Review: The Escape Artist",
    "subtitle": "Finalist #5 in the Book Review Contest",
    "slug": "your-book-review-the-escape-artist",
    "postDate": "2026-08-14T22:21:13.104Z",
    "type": "newsletter",
    "likes": 61,
    "commentCount": 14,
    "audience": "everyone",
    "paywalled": false,
    "wordCount": 6807,
    "description": "Finalist #5 in the Book Review Contest",
    "canonicalUrl": "https://www.astralcodexten.com/p/your-book-review-the-escape-artist",
    "coverImage": "https://substackcdn.com/image/fetch/w_1200/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fexample.png",
    "publicationId": 89120,
    "publicationUrl": "https://astralcodexten.substack.com",
    "authorName": "Scott Alexander",
    "authorHandle": "astralcodexten",
    "authorId": 12345678
}

With includeContent: true each row additionally contains bodyHtml and bodyText.

Example item — comments mode

{
    "postId": 210672789,
    "postTitle": "Your Book Review: The Escape Artist",
    "postSlug": "your-book-review-the-escape-artist",
    "postUrl": "https://www.astralcodexten.com/p/your-book-review-the-escape-artist",
    "publicationUrl": "https://astralcodexten.substack.com",
    "commentId": 98765432,
    "parentCommentId": null,
    "body": "Great review — the framing of the escape as a moral act is what stuck with me.",
    "bodyDeleted": false,
    "authorName": "Jane Reader",
    "authorHandle": "janereader",
    "authorId": 2345678,
    "date": "2026-08-15T03:12:44.000Z",
    "likes": 12,
    "childCount": 3
}

Example item — search mode

{
    "query": "AI agents",
    "id": 198765432,
    "title": "The Agentic Web Is Coming",
    "subtitle": "What happens when most web traffic is bots with budgets",
    "slug": "the-agentic-web-is-coming",
    "postDate": "2026-07-02T15:00:00.000Z",
    "type": "newsletter",
    "likes": 214,
    "commentCount": 45,
    "audience": "everyone",
    "paywalled": false,
    "wordCount": 2140,
    "description": "What happens when most web traffic is bots with budgets",
    "canonicalUrl": "https://www.example-newsletter.com/p/the-agentic-web-is-coming",
    "coverImage": null,
    "publicationId": 55501,
    "publicationUrl": null,
    "authorName": "Alex Writer",
    "authorHandle": "alexwriter",
    "authorId": 7654321
}

Example item — publication mode

{
    "publicationUrl": "https://astralcodexten.substack.com",
    "id": 89120,
    "name": "Astral Codex Ten",
    "subdomain": "astralcodexten",
    "customDomain": "www.astralcodexten.com",
    "hostname": "www.astralcodexten.com",
    "authorName": "Scott Alexander",
    "authorHandle": "astralcodexten",
    "authorBio": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
    "description": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
    "logoUrl": "https://substackcdn.com/image/fetch/w_256/https%3A%2F%2Fexample-logo.png",
    "coverPhotoUrl": null,
    "language": "en",
    "createdAt": "2021-01-21T00:00:00.000Z",
    "firstPostDate": "2021-01-21T08:00:00.000Z",
    "freeSubscriberCount": "100000",
    "subscriberCountOrderOfMagnitude": 100000,
    "rankingDetail": "#2 in Science",
    "paymentsEnabled": "enabled",
    "podcastEnabled": false,
    "twitter": "slatestarcodex",
    "copyright": "Scott Alexander"
}

Error semantics

• Bad input fails fast: unknown mode, search without queries, or other modes without publicationUrls throw immediately — the run exits non-zero with a clear status message. Nothing is charged beyond the start fee.
• Transient HTTP errors (429, 5xx, network) are retried 3 times with increasing backoff (30 s request timeout). If all retries fail, the run fails with a non-zero exit — never a silently truncated dataset.
• Non-retryable 4xx on a publication endpoint (e.g. a wrong URL returning 404) yields zero rows for that source; other sources in the same run still proceed.
• Zero results (empty archive, no search hits) is a successful run with an empty dataset — check the item count, not the exit code, to distinguish "nothing found" from "failed".
• Publication mode is best-effort: if homepage parsing fails, the actor falls back to archive data and still pushes a row (with nulls for unavailable fields); it never throws mid-run.

Use from AI agents (MCP)

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=apricot_blackberry/substack-all-in-one",
      "headers": { "Authorization": "Bearer <YOUR_APIFY_TOKEN>" }
    }
  }
}

Works in Claude, Cursor, ChatGPT deep research connectors, and any MCP client; the input schema above is the tool's parameter schema.

Use from code

curl (synchronous run, returns dataset items directly):

curl -X POST "https://api.apify.com/v2/acts/apricot_blackberry~substack-all-in-one/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"posts","publicationUrls":["https://astralcodexten.substack.com"],"maxItems":50}'

JavaScript (apify-client):

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('apricot_blackberry/substack-all-in-one').call({
    mode: 'posts',
    publicationUrls: ['https://astralcodexten.substack.com'],
    maxItems: 50,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python (apify-client):

from apify_client import ApifyClient

client = ApifyClient(token=os.environ["APIFY_TOKEN"])
run = client.actor("apricot_blackberry/substack-all-in-one").call(run_input={
    "mode": "posts",
    "publicationUrls": ["https://astralcodexten.substack.com"],
    "maxItems": 50,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)

Use from automation platforms

• n8n / Make / Zapier — use the native Apify integration and pick this actor by name (apricot_blackberry/substack-all-in-one).
• LangChain / LlamaIndex — use the Apify actor tool wrappers (ApifyActorsTool / ApifyActor reader) with the same input JSON.
• Webhooks — Apify webhooks can fire on run completion (e.g. ACTOR.RUN.SUCCEEDED) to push new posts/comments into your own pipeline automatically, no polling required.

Pricing

Pay per event — no monthly rental, just a tiny $0.005 start fee (vs ~$0.09 typical for competitors):

Event	Charged for
actor-start	Once per run
post	Each post metadata row
post-content	Each post row including the full body
comment	Each comment row
search-result	Each search result row
publication	Each publication intel row

Runs use the Apify datacenter proxy by default (included in Apify subscription plans), so there are no residential proxy surcharges.

FAQ

Can I scrape Substack posts without the API or login? Yes — this actor reads Substack's public JSON endpoints (the same ones the website uses), so no credentials, cookies, or browser automation are needed.

Does it work with newsletters on custom domains? Yes. Pass the custom domain (e.g. https://www.lennysnewsletter.com) or the original *.substack.com URL — redirects are followed automatically.

Can I get the full text of paywalled posts? No. Paywalled posts return only the free preview. Free posts return the complete article body.

How do I export Substack data to CSV or Excel? Every run stores results in an Apify dataset; download as JSON, CSV, Excel, or XML from the console or via API — ideal for newsletter research and content analysis.

Can my AI agent use this? Yes — see the MCP section above. The actor is a first-class MCP tool with a typed input and output schema.

Changelog

• 1.1 — Added strict dataset output schema (typed, validated rows) and agent-first documentation: MCP config, code snippets, error semantics.
• 1.0 — Initial release: posts, comments, search, publication modes; full-content option; pay-per-event pricing.