Substack Scraper - Markdown Posts, Search & Comments avatar

Substack Scraper - Markdown Posts, Search & Comments

Pricing

from $1.20 / 1,000 post delivereds

Go to Apify Store
Substack Scraper - Markdown Posts, Search & Comments

Substack Scraper - Markdown Posts, Search & Comments

Scrape public Substack posts into clean Markdown for LLMs and RAG, with metadata and optional comments. Find publications by search; scrape by name, URL or custom domain. Filter by date or collect only new posts across scheduled runs. Paywalled posts: public previews only.

Pricing

from $1.20 / 1,000 post delivereds

Rating

0.0

(0)

Developer

Alejandro Rodriguez

Alejandro Rodriguez

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

11 minutes ago

Last modified

Categories

Features

Limited permissions

Share

Scrape public Substack publications through their JSON API and get every post as clean Markdown, ready to drop into an LLM, a RAG pipeline or a vector database.

When to use it: choose this Actor to find Substack publications or retrieve their public posts as clean Markdown with metadata and optional comment threads for LLM/RAG ingestion or monitoring, including date filtering and incremental collection across scheduled runs.

Full paywalled content is unavailable; only public previews can be retrieved. Not affiliated with Substack.

  • Fast and cheap - plain HTTP requests. No headless browser, no residential proxies.
  • LLM-ready output - subscribe buttons, share widgets and embeds markup are stripped; headings, lists, code blocks, images, captions and footnotes are kept as standard Markdown.
  • Search built in - give it a topic and it finds matching publications for you. Discovery is free.
  • Comments on demand - optionally attach the comment thread to each post.
  • Incremental runs - schedule it and receive only posts you have not seen yet.
  • Fair billing - you pay only for posts actually delivered. Errors, duplicates and skipped paywalled posts are never charged.

Quick start

Run the Actor with the prefilled input. It fetches the 10 latest posts of one publication in a few seconds and costs about $0.01:

{
"publications": ["astralcodexten"],
"maxPosts": 10
}

Open the Output tab of the run to see the posts, or download them as JSON, CSV or Excel. Then replace the publication with your own and raise maxPosts.

What you can scrape

DataFields
Postid, url, slug, title, subtitle, type, publishedAt, updatedAt, description, coverImage
Contentmarkdown, markdownLength, wordCount, isPaywalled, contentComplete
People & topicsauthors[] (id, name, handle), tags[]
EngagementreactionCount, commentCount
Comments (optional)comments[] with id, parentId, depth, author, authorHandle, date, body, reactionCount
PublicationpublicationId, publicationUrl

Input

FieldDescription
publicationsNames (astralcodexten), Substack URLs or custom domains.
searchQueryOptional topic. Matching publications are scraped as well.
maxSearchPublicationsHow many publications from the search to scrape (default 5, max 25).
maxPostsTotal posts to deliver in the run (default 100).
maxPostsPerPublicationOptional cap per publication (0 = no cap).
sinceDateOnly posts published on or after this date. YYYY-MM-DD or relative, e.g. 7 days.
incrementalDeliver only posts newer than the previous run (see below).
paywalledPostsskip (default, not charged) or preview (public preview, flagged).
includeComments / maxCommentsPerPostAttach up to N comments and replies per post, best first.
proxyConfigurationOff by default. Enable Apify datacenter proxy only if a run reports a block.

Provide publications, searchQuery, or both. Posts are delivered newest first.

{
"publications": ["astralcodexten", "https://www.lennysnewsletter.com"],
"maxPosts": 200,
"sinceDate": "30 days"
}

Output

{
"id": 224550256,
"publicationId": 89120,
"publicationUrl": "https://astralcodexten.substack.com",
"url": "https://www.astralcodexten.com/p/ai-company-employees-should-unionize",
"title": "AI Company Employees Should Unionize",
"type": "newsletter",
"audience": "everyone",
"isPaywalled": false,
"contentComplete": true,
"publishedAt": "2026-11-12T23:09:15.486Z",
"authors": [{ "id": 12009663, "name": "Scott Alexander", "handle": "astralcodexten" }],
"tags": [],
"wordCount": 1273,
"reactionCount": 306,
"commentCount": 211,
"markdown": "# AI Company Employees Should Unionize\n\n**I.**\n\nShould safety-conscious employees ...",
"markdownLength": 11328,
"commentsStatus": "not_requested",
"comments": null,
"scrapedAt": "2026-11-15T21:50:40.000Z"
}

The markdown field starts with the title as an H1 (and the subtitle, when there is one) so each document is self-contained when chunked.

The key-value store of every run also contains:

  • RUN_SUMMARY - why the run stopped (completed, max_posts, budget, blocked), per-publication status, and how many posts were skipped and why.
  • DISCOVERED_PUBLICATIONS - the publications found by searchQuery.

Pricing

Pay per event. You are charged only when a post is stored in your dataset.

EventPriceWhen
Post delivered$0.0012 ($1.20 per 1,000 posts)A valid post with content is stored in the dataset.
Post comments$0.0005 ($0.50 per 1,000 posts)Only with includeComments, once per post that has at least one comment delivered. It is per post, not per comment.
Actor start$0.00005Once per run.

What typical runs cost (plus $0.00005 for the start):

RunCost
10 latest posts$0.012
500 posts$0.60
A 10,000-post archive$12.00
1,000 posts, all with comments$1.70

Never charged: failed requests, duplicates, posts with no content, paywalled posts you chose to skip, and search discovery. A run that delivers nothing only costs the start event.

Set a maximum cost per run in the run options: the Actor stops cleanly when the limit is reached and never goes over it.

Incremental mode

With incremental: true the Actor remembers, per publication, the newest post it delivered.

  • The first run works normally and sets the baseline.
  • Later runs deliver only posts published after that baseline, oldest first, so a run interrupted by maxPosts or your cost limit simply continues next time. Nothing is lost and nothing is delivered twice.
  • State is kept in a named key-value store, substack-scraper-state, in your account. Use a different stateNamespace for each independent job.

Incremental mode tracks new posts only. A post edited after it was delivered is not delivered again.

To keep an index or a feed up to date:

  1. Set your publications and incremental: true, and save that configuration as a task.
  2. Create a schedule for the task, for example once a day.
  3. Each run's dataset then contains only the posts published since the previous run. Forward them to your pipeline with a webhook or any Apify integration.

Run it from code

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("gunpark/substack-scraper").call(
run_input={"publications": ["astralcodexten"], "maxPosts": 20}
)
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
print(post["title"], len(post["markdown"]))

Use it from AI agents (MCP)

The Actor works as a tool for any MCP client through the hosted Apify MCP server. Preload it with:

{
"mcpServers": {
"apify": {
"url": "https://mcp.apify.com?tools=gunpark/substack-scraper"
}
}
}

The agent can then call it with a publication name or a search query and receive Markdown it can read directly.

Limits and notes

  • Paywalled posts expose only a public preview without a subscription. With paywalledPosts: "preview" they are delivered with isPaywalled: true and contentComplete: false; some have no public preview at all and are skipped for free.
  • Podcast and video posts are delivered when they have written content; the media itself is not downloaded.
  • Blocks and rate limits - requests are retried with backoff. On a hard block the run stops, keeps what it already delivered and reports blocked in RUN_SUMMARY. If nothing was delivered the run is marked as failed.
  • Personal data - comments include public author names and handles. Make sure your use of that data complies with the laws that apply to you.
  • This Actor reads only publicly available data and is not affiliated with Substack.