Substack Scraper - Markdown Posts, Search & Comments
Pricing
from $1.20 / 1,000 post delivereds
Substack Scraper - Markdown Posts, Search & Comments
Scrape public Substack posts into clean Markdown for LLMs and RAG, with metadata and optional comments. Find publications by search; scrape by name, URL or custom domain. Filter by date or collect only new posts across scheduled runs. Paywalled posts: public previews only.
Pricing
from $1.20 / 1,000 post delivereds
Rating
0.0
(0)
Developer
Alejandro Rodriguez
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
11 minutes ago
Last modified
Categories
Features
Share
Scrape public Substack publications through their JSON API and get every post as clean Markdown, ready to drop into an LLM, a RAG pipeline or a vector database.
When to use it: choose this Actor to find Substack publications or retrieve their public posts as clean Markdown with metadata and optional comment threads for LLM/RAG ingestion or monitoring, including date filtering and incremental collection across scheduled runs.
Full paywalled content is unavailable; only public previews can be retrieved. Not affiliated with Substack.
- Fast and cheap - plain HTTP requests. No headless browser, no residential proxies.
- LLM-ready output - subscribe buttons, share widgets and embeds markup are stripped; headings, lists, code blocks, images, captions and footnotes are kept as standard Markdown.
- Search built in - give it a topic and it finds matching publications for you. Discovery is free.
- Comments on demand - optionally attach the comment thread to each post.
- Incremental runs - schedule it and receive only posts you have not seen yet.
- Fair billing - you pay only for posts actually delivered. Errors, duplicates and skipped paywalled posts are never charged.
Quick start
Run the Actor with the prefilled input. It fetches the 10 latest posts of one publication in a few seconds and costs about $0.01:
{"publications": ["astralcodexten"],"maxPosts": 10}
Open the Output tab of the run to see the posts, or download them as JSON, CSV or Excel. Then replace the publication with your own and raise maxPosts.
What you can scrape
| Data | Fields |
|---|---|
| Post | id, url, slug, title, subtitle, type, publishedAt, updatedAt, description, coverImage |
| Content | markdown, markdownLength, wordCount, isPaywalled, contentComplete |
| People & topics | authors[] (id, name, handle), tags[] |
| Engagement | reactionCount, commentCount |
| Comments (optional) | comments[] with id, parentId, depth, author, authorHandle, date, body, reactionCount |
| Publication | publicationId, publicationUrl |
Input
| Field | Description |
|---|---|
publications | Names (astralcodexten), Substack URLs or custom domains. |
searchQuery | Optional topic. Matching publications are scraped as well. |
maxSearchPublications | How many publications from the search to scrape (default 5, max 25). |
maxPosts | Total posts to deliver in the run (default 100). |
maxPostsPerPublication | Optional cap per publication (0 = no cap). |
sinceDate | Only posts published on or after this date. YYYY-MM-DD or relative, e.g. 7 days. |
incremental | Deliver only posts newer than the previous run (see below). |
paywalledPosts | skip (default, not charged) or preview (public preview, flagged). |
includeComments / maxCommentsPerPost | Attach up to N comments and replies per post, best first. |
proxyConfiguration | Off by default. Enable Apify datacenter proxy only if a run reports a block. |
Provide publications, searchQuery, or both. Posts are delivered newest first.
{"publications": ["astralcodexten", "https://www.lennysnewsletter.com"],"maxPosts": 200,"sinceDate": "30 days"}
Output
{"id": 224550256,"publicationId": 89120,"publicationUrl": "https://astralcodexten.substack.com","url": "https://www.astralcodexten.com/p/ai-company-employees-should-unionize","title": "AI Company Employees Should Unionize","type": "newsletter","audience": "everyone","isPaywalled": false,"contentComplete": true,"publishedAt": "2026-11-12T23:09:15.486Z","authors": [{ "id": 12009663, "name": "Scott Alexander", "handle": "astralcodexten" }],"tags": [],"wordCount": 1273,"reactionCount": 306,"commentCount": 211,"markdown": "# AI Company Employees Should Unionize\n\n**I.**\n\nShould safety-conscious employees ...","markdownLength": 11328,"commentsStatus": "not_requested","comments": null,"scrapedAt": "2026-11-15T21:50:40.000Z"}
The markdown field starts with the title as an H1 (and the subtitle, when there is one) so each document is self-contained when chunked.
The key-value store of every run also contains:
RUN_SUMMARY- why the run stopped (completed,max_posts,budget,blocked), per-publication status, and how many posts were skipped and why.DISCOVERED_PUBLICATIONS- the publications found bysearchQuery.
Pricing
Pay per event. You are charged only when a post is stored in your dataset.
| Event | Price | When |
|---|---|---|
| Post delivered | $0.0012 ($1.20 per 1,000 posts) | A valid post with content is stored in the dataset. |
| Post comments | $0.0005 ($0.50 per 1,000 posts) | Only with includeComments, once per post that has at least one comment delivered. It is per post, not per comment. |
| Actor start | $0.00005 | Once per run. |
What typical runs cost (plus $0.00005 for the start):
| Run | Cost |
|---|---|
| 10 latest posts | $0.012 |
| 500 posts | $0.60 |
| A 10,000-post archive | $12.00 |
| 1,000 posts, all with comments | $1.70 |
Never charged: failed requests, duplicates, posts with no content, paywalled posts you chose to skip, and search discovery. A run that delivers nothing only costs the start event.
Set a maximum cost per run in the run options: the Actor stops cleanly when the limit is reached and never goes over it.
Incremental mode
With incremental: true the Actor remembers, per publication, the newest post it delivered.
- The first run works normally and sets the baseline.
- Later runs deliver only posts published after that baseline, oldest first, so a run interrupted by
maxPostsor your cost limit simply continues next time. Nothing is lost and nothing is delivered twice. - State is kept in a named key-value store,
substack-scraper-state, in your account. Use a differentstateNamespacefor each independent job.
Incremental mode tracks new posts only. A post edited after it was delivered is not delivered again.
To keep an index or a feed up to date:
- Set your publications and
incremental: true, and save that configuration as a task. - Create a schedule for the task, for example once a day.
- Each run's dataset then contains only the posts published since the previous run. Forward them to your pipeline with a webhook or any Apify integration.
Run it from code
from apify_client import ApifyClientclient = ApifyClient("YOUR_APIFY_TOKEN")run = client.actor("gunpark/substack-scraper").call(run_input={"publications": ["astralcodexten"], "maxPosts": 20})for post in client.dataset(run["defaultDatasetId"]).iterate_items():print(post["title"], len(post["markdown"]))
Use it from AI agents (MCP)
The Actor works as a tool for any MCP client through the hosted Apify MCP server. Preload it with:
{"mcpServers": {"apify": {"url": "https://mcp.apify.com?tools=gunpark/substack-scraper"}}}
The agent can then call it with a publication name or a search query and receive Markdown it can read directly.
Limits and notes
- Paywalled posts expose only a public preview without a subscription. With
paywalledPosts: "preview"they are delivered withisPaywalled: trueandcontentComplete: false; some have no public preview at all and are skipped for free. - Podcast and video posts are delivered when they have written content; the media itself is not downloaded.
- Blocks and rate limits - requests are retried with backoff. On a hard block the run stops, keeps what it already delivered and reports
blockedinRUN_SUMMARY. If nothing was delivered the run is marked as failed. - Personal data - comments include public author names and handles. Make sure your use of that data complies with the laws that apply to you.
- This Actor reads only publicly available data and is not affiliated with Substack.