Newsletter to Markdown: Substack, Ghost, beehiiv Archive Export

Export a Substack, Ghost or beehiiv newsletter to Markdown: the whole archive, not just the last 20 posts in the RSS feed. Give it a newsletter URL (custom domains work). It returns one record per post with the body as clean Markdown and a ready-to-save .md file with YAML front matter (title, authors, date, tags, paywall flag). Set a State key and schedule it: each run returns only posts it has not exported before, and you pay only for those.

Input	Newsletter home page URLs: *.substack.com, *.ghost.io, *.beehiiv.com or a custom domain on one of them
Output	Title, subtitle, authors, date, tags, free/paid, post body as Markdown, .md file with front matter, word count, likes, comments
Price	$0.003 per full-text post, $0.001 per metadata-only or paywalled post, $0.005 per run. A 500-post archive ≈ $1.51
Speed	15 posts from three newsletters (one per platform) in about 10 seconds on 256 MB. No browser

{
    "platform": "substack",
    "publication_name": "Simon Willison’s Newsletter",
    "url": "https://simonw.substack.com/p/conceptual-integrity-and-counting",
    "title": "Conceptual integrity and counting lines of code",
    "subtitle": "Plus Bun 1.4 and the latest on Anthropic’s revenue",
    "authors": ["Simon Willison"],
    "published": "2026-08-23T20:57:32.675Z",
    "tags": ["anthropic", "llms", "ai"],
    "audience": "free",
    "paywalled": false,
    "word_count": 3116,
    "content_markdown": "In this newsletter:\n\n- Conceptual integrity and counting lines of code\n...",
    "markdown_file": "---\ntitle: \"Conceptual integrity and counting lines of code\"\n...\n---\n\n# Conceptual integrity ...",
    "full_text": true
}

Jump to the task you have:

1. Download a Substack (or Ghost, beehiiv) archive as Markdown files
2. Get only new posts every week
3. Load newsletters into a RAG index, vector database or LLM wiki
4. Get a list of posts and dates without the text
5. Let an AI agent read a newsletter
6. What this Actor cannot do

1. Download a Substack (or Ghost, beehiiv) archive as Markdown files

Paste the newsletter's home page and set Max posts per newsletter to the archive size. A post URL also works: only its origin is used.

{ "publications": ["https://simonw.substack.com"], "maxPostsPerPublication": 500 }

How the posts are listed, newest first:

• Substack: the publication's archive API (the list behind the Archive page), then each post's body from the post API. No HTML scraping.
• Ghost and beehiiv: the site's sitemap.xml (the posts sitemap only, so tag, author and page URLs are left out), then each post page.
• Anything else: the sitemap, falling back to the RSS / Atom feed.

The platform is detected from the home page, so custom domains (www.platformer.news) work. If detection is wrong, set platform.

To save files, use the Markdown files view of the dataset, or write each item's markdown_file to <slug>.md. It is a complete document: YAML front matter (title, subtitle, publication, URL, authors, dates, tags, audience, paywall flag, content hash), the title as # heading and the body. It drops straight into an Obsidian vault or a Claude Code / LLM wiki raw/ folder.

2. Get only new posts every week

Set State key to any name, for example newsletters-ai-weekly, and create an Apify Schedule with that input. The Actor remembers every post it has exported in a named key-value store; the next run skips them and returns only posts not exported before. Skipped posts are not charged.

{
    "publications": ["https://simonw.substack.com", "https://www.platformer.news", "https://www.superhuman.ai"],
    "stateKey": "newsletters-ai-weekly",
    "sinceDays": 14,
    "maxPostsPerPublication": 50
}

• Set sinceDays a little longer than the schedule interval (14 for a weekly run). Without it, each run also back-fills older posts it has not exported yet, up to maxPostsPerPublication per newsletter, until the whole archive is done. With it, only posts inside the window are considered.
• The state is also saved when the platform migrates or aborts the run, so a restarted run does not export and charge the same posts twice.
• Use a different State key per project. Deleting the key-value store resets it.
• SUMMARY.json in the run's key-value store shows, per newsletter, the detected platform, how posts were listed (substack-api, sitemap, feed), and how many were listed, kept, skipped as already exported and skipped as too old, plus any error.

3. Load newsletters into a RAG index, vector database or LLM wiki

Each record is one document. content_markdown is the text to chunk and embed; title, url, authors, published, tags and publication_name are the metadata to store with it.

• url_hash (SHA-256 of the normalised URL) is a stable document ID. Use it as the primary key so re-runs never create duplicates.
• content_hash tells you whether the text you stored differs from what you have now.
• Filter on full_text: true to index only complete posts; paywalled previews have paywalled: true.

Read the dataset with the Apify API (/v2/datasets/{id}/items?format=json) or the Apify integrations for Pinecone, Qdrant, Weaviate and others. In n8n, Make or Zapier, use the Apify app's "run finished" trigger with a scheduled run (section 2): every item is a new post.

4. Get a list of posts and dates without the text

Turn off Fetch full post content. Each post is stored with title, URL, date, authors, tags (Substack also gives likes and comments) at the metadata price, $0.001. Useful for tracking how often a competitor publishes, or picking which posts to export in full.

{ "publications": ["https://www.platformer.news"], "fetchContent": false, "maxPostsPerPublication": 1000 }

5. Let an AI agent read a newsletter (MCP)

With the Apify MCP server, an agent (Claude, Cursor, ChatGPT and others) can call this Actor by name with a newsletter URL and read the Markdown from the dataset. The dataset schema describes every field. Keep maxPostsPerPublication low (3 to 10) for interactive use.

What this Actor cannot do

Read this before your first run. Nothing here fails the whole run: a broken newsletter or post is reported and the rest continues. The run fails only if no newsletter produced a post.

Limit	What happens	What to do
Paid posts	No login, no cookies, no paywall bypass. You get the public preview the publisher shows to everyone, paywalled: true, full_text: false, charged at the $0.001 rate. If there is no preview the body is empty	None. Export only free posts, or use the publisher's own export if you are the author
Other platforms (Medium, Buttondown, Kit, Mailchimp archives, WordPress)	Treated as other: posts come from the sitemap or RSS feed and the body from the generic article extractor. Works on many sites, but every sitemap URL that is not an obvious tag / author / legal page is treated as a post	Set sinceDays and maxPostsPerPublication, or use RSS to Markdown with the feed URL
Very large Ghost / beehiiv archives	Up to 10 posts sitemaps are read per newsletter. Substack: up to 200 archive pages (about 10,000 posts)	Open an issue if you need more
Sites that block bots (Cloudflare challenge, 403)	The post is stored with metadata only and extraction_error: "page fetch failed: ..." after 2 retries	Turn on Apify Proxy in proxyConfiguration
robots.txt disallows a path	That path is not fetched (on by default). A disallowed post is stored with metadata only	Set respectRobotsTxt: false only if you have the right to fetch those pages
JavaScript-only pages	The page HTML has no post text: metadata only, extraction_error: "no article text found on page"	None for that site
Dates on Ghost / beehiiv	Posts are listed from the sitemap, whose date is the last modification, so sinceDays keeps an old post that was edited recently. published in the output is read from the post page	Filter on published downstream if you need publish dates exactly
A post is edited after export	Dedupe is by URL, so an edited post is not exported again	Compare content_hash yourself, or run without a State key
Comments, podcasts, video	Only counts (likes, comments, Substack). No comment text, no audio transcription. Embedded media stays as links	—
Rate limits	Substack API: 2 requests in parallel with a 250–300 ms pause. Pages: 3 in parallel by default, 45 s timeout. 429 / 5xx are retried with back-off, Retry-After is honoured	Lower maxConcurrency for small sites that throttle
Very large state	The State key remembers the 100,000 most recent posts; older ones are forgotten	Split very large projects across several State keys
Budget cap	With Max total charge per run set, the run stops cleanly before going over it. Posts not exported are not saved to the state, so the next run picks them up	Raise the cap or lower maxPostsPerPublication
Copyright	Posts belong to their authors	Use the output for research, analysis and indexing; check the publisher's terms before republishing

Input

Field	Type	Default	Notes
publications	string[]	required	Newsletter home page URLs. Custom domains and bare hosts are accepted.
maxPostsPerPublication	integer	50	Newest posts kept per newsletter after filtering and dedupe. Caps cost.
fetchContent	boolean	true	Convert post bodies to Markdown. Off = metadata only (cheaper event).
sinceDays	integer	0	0 = no date filter.
platform	string	auto	Force substack, ghost, beehiiv or other if auto-detection is wrong.
stateKey	string	""	Named key-value store that remembers exported posts between runs.
respectRobotsTxt	boolean	true	Skip URLs disallowed by the newsletter's robots.txt.
maxConcurrency	integer	3	Parallel page requests (Substack API calls are limited to 2 in parallel).
proxyConfiguration	object	no proxy	Enable Apify Proxy only if a site blocks you.

Output fields

Field	Description
platform	substack, ghost, beehiiv or other
publication_name, publication_url	Newsletter name and origin
url, slug, title, subtitle	Post identity
authors, published, updated, tags	Metadata (Substack: bylines and post tags; Ghost / beehiiv: page meta tags)
audience, paywalled	free / paid / unknown, and whether only the preview was available
content_markdown, word_count	Post body as GitHub-flavoured Markdown (headings, lists, links, images, code, tables)
markdown_file	Front matter + title + body, ready to save as <slug>.md
content_hash, url_hash	SHA-256 for change detection and dedupe
likes, comments	Substack reaction and comment counts
full_text	true when the full public body was exported. This decides which billing event applies
extraction_error	Why the body could not be exported, if anything went wrong

Pricing

Pay-per-event. Every stored post is charged exactly once, at one of two rates. Platform usage (compute, storage) is included.

Event	Price	When
post-full-text	$0.003 per post ($3 per 1,000)	A free post exported with its full body as Markdown (full_text: true)
post-metadata	$0.001 per post ($1 per 1,000)	Metadata only (fetchContent off), a paywalled preview, or a failed page
Actor start	$0.005 per run	Apify's standard start event

Examples: 50 recent posts from one newsletter ≈ $0.155; a weekly run over 10 newsletters that finds 15 new posts ≈ $0.05. Posts skipped by the date filter or the State key are never charged.

Verified

Tested on 2026-09-23 with the example input: 15 posts (Substack 5, Ghost 5, beehiiv 5), 0 failures, about 10 seconds. Paywalled Ghost posts were flagged and billed as metadata. npm test runs unit tests on real fixtures (platform detection, Substack archive mapping, Ghost sitemap index, post URL filter, beehiiv body, Ghost paywall, front matter). Platform run on 2026-10-05 (build 0.1.3): 15 posts, 13 full text, 2 paywalled Platformer posts billed as metadata, 0 failures, 7.3 s. A local run with a State key returned 15 posts, and a second run with the same key skipped all 15. The platform runs the prefill input daily as an automated check.

Changelog

• 0.1.3 (2026-10-05): the dedupe state is also saved on migration / abort (no double export after a platform restart). README rewritten by task, with a limits table; prices in the README corrected to the live prices.
• 0.1 (2026-09-23): first release: Substack archive API, Ghost and beehiiv sitemaps, RSS fallback, paywall flag, front matter, dedupe across runs.

Support

Found a newsletter that does not export correctly? Open an issue on the Issues tab with the URL; issues are answered within one business day.

Made by Vellum Labs.