Substack Scraper & Medium Scraper: Articles to Markdown

Price: $8 per 1,000 articles ($0.008 per article). Pay per result: no start fee, no monthly rental. Errors, blocked sources, empty results, duplicates and already-seen articles are free.

Turn any Substack newsletter or Medium author, publication or tag into clean, LLM-ready Markdown, with the publication date, author(s), likes, comments, tags, word count and paywall status. One Actor, both platforms, one consistent output format (JSON, CSV, Excel or Markdown files).

• Substack: reads the newsletter's full public archive (not just the last 20 posts of the RSS feed) through Substack's public JSON API. Free posts come back in full; for subscriber-only posts you get the public preview, clearly flagged.
• Medium: reads the public RSS feeds of authors, publications (including custom domains) and tags.
• Clean Markdown: subscribe boxes, share buttons, tracking pixels and the "originally published in…" footer are removed. Headings, lists, links, quotes, code blocks and image captions are kept.
• RAG-ready: optional chunks array (Markdown split on paragraph boundaries) for embeddings and vector databases.
• Scheduled monitoring: onlyNew delivers (and bills) only the articles published since your last run.
• No API key, no login, no browser, no proxy needed: fast and cheap. Only public data is read.

How much does it cost?

Event	Price
article: one article delivered to the dataset	$0.008

Examples: 100 articles = $0.80 · 1,000 articles = $8 · 10,000 articles = $80. A daily schedule on 10 newsletters with onlyNew (≈ 1 new post each per day) costs about $2.40 per month.

Set Max articles in total or a maximum run cost in Apify Console to stay within your budget: the Actor stops cleanly at the limit and never delivers unpaid "preview" items.

What you get for each article

Field	Description
platform	substack or medium
title, subtitle, url, slug, id	Article identity
author, authorHandle, authors[]	Author name(s), handle, profile URL (Substack co-authors and guest authors included)
publication, publicationUrl	Newsletter or Medium publication
publishedAt, updatedAt	ISO 8601 dates (UTC)
likes	Substack hearts (reactions). null on Medium: claps are not exposed publicly
comments, restacks	Substack comment and restack counts
tags	Substack post tags / Medium categories
wordCount, readingTimeMinutes	Length
isPaywalled, audience, contentStatus	contentStatus = full or preview: you always know whether the text is complete
markdown	Self-contained Markdown document: title, byline line, then the body
chunks	Markdown split into RAG chunks (optional)
html	Original HTML (optional)
markdownFileUrl	Link to a .md file in the key-value store (optional)

Input example

{
  "substackNewsletters": ["simonw", "www.noahpinion.blog", "@lenny"],
  "mediumSources": ["@karpathy", "netflixtechblog.com"],
  "mediumTags": ["llm"],
  "maxArticlesPerSource": 20,
  "publishedAfter": "2026-01-01",
  "onlyNew": false,
  "outputChunks": true,
  "chunkSize": 2000,
  "keepImages": true
}

Accepted formats:

• Substack: simonw, https://simonw.substack.com, www.noahpinion.blog (custom domain), @lenny or https://substack.com/@lenny (author, resolved to their main newsletter).
• Medium: @karpathy, https://medium.com/@karpathy, karpathy.medium.com, https://medium.com/some-publication, netflixtechblog.com, or tags in the Medium tags field (llm, product-management).

Output example

{
  "platform": "substack",
  "sourceInput": "simonw",
  "id": "171234567",
  "url": "https://simonw.substack.com/p/2026-in-llms-so-far",
  "slug": "2026-in-llms-so-far",
  "title": "2026 in LLMs (so far)",
  "subtitle": null,
  "author": "Simon Willison",
  "authorHandle": "simonw",
  "authors": [{ "name": "Simon Willison", "handle": "simonw", "profileUrl": "https://substack.com/@simonw" }],
  "publication": "Simon Willison’s Newsletter",
  "publicationSlug": "simonw",
  "publicationUrl": "https://simonw.substack.com",
  "publishedAt": "2026-09-28T16:42:27.944000Z",
  "updatedAt": null,
  "postType": "newsletter",
  "tags": [],
  "likes": 93,
  "comments": 2,
  "restacks": 11,
  "wordCount": 5120,
  "readingTimeMinutes": 22,
  "isPaywalled": false,
  "audience": "everyone",
  "contentStatus": "full",
  "language": "en",
  "coverImage": "https://substackcdn.com/image/fetch/...",
  "markdown": "# 2026 in LLMs (so far)\n\nBy Simon Willison · Simon Willison’s Newsletter · 2026-09-28 · https://simonw.substack.com/p/2026-in-llms-so-far\n\n---\n\nIn this newsletter:\n\n- ...",
  "chunks": ["# 2026 in LLMs (so far)\n\nBy Simon Willison ...", "..."],
  "scrapedAt": "2026-10-15T09:12:44.120000Z"
}

A Medium item has the same shape, with likes, comments and restacks set to null.

Use with AI agents (MCP)

This Actor is MCP-ready: add it to Claude, Cursor, ChatGPT or any MCP client through the Apify MCP server (https://mcp.apify.com), then ask for example "Get the last 10 posts of simonw on Substack as Markdown and summarise them". The input is a small JSON object, and every item carries a self-contained markdown field (plus optional chunks), so an agent can read the result directly without any HTML cleaning.

Integrations

• Apify API: run synchronously and get the items in one call (run-sync-get-dataset-items), in JSON, CSV, Excel or XML.
• Make, Zapier, n8n: use the Apify app/node to start a run and loop over the dataset items.
• Google Sheets: export the dataset or use the Apify Google Sheets integration for a live sheet of new articles.
• Schedules + webhooks: schedule a daily run with onlyNew: true and send new articles to Slack, email or your vector database.

Use cases

• RAG and AI agents: load a newsletter's whole archive into a vector database with the ready-made chunks.
• Newsletter monitoring: schedule the Actor daily with onlyNew to collect only new posts from a list of newsletters and authors.
• Content research and competitive analysis: compare publishing frequency, length and engagement (likes, comments, restacks) across Substack newsletters.
• Summaries and digests: pipe the Markdown into an LLM to produce a weekly digest of your favourite writers.
• Archiving: save each article as a .md file in the key-value store.

How it works (and why it is reliable)

• Substack: /api/v1/archive (paged, newest first; the body is often included) and /api/v1/posts/{slug} only when the body is missing: structured JSON, no fragile HTML parsing.
• Medium: official RSS feeds (medium.com/feed/@author, /feed/publication, /feed/tag/...).
• Automatic retries with exponential back-off on network errors, HTTP 429 and 5xx; polite per-site delay and at most 2 parallel requests per site.
• One failing article or source never stops the run; each problem is logged in plain English and summarised in the run status.
• If a site refuses the direct connection, the Actor switches that site to the Apify datacenter proxy automatically. Residential proxy is never used unless you pick it.
• Duplicate URLs (same article from two inputs) are delivered and billed once.

FAQ

Can it get the full text of paid (subscriber-only) Substack posts? No. It never logs in. You get the public preview, flagged with isPaywalled: true and contentStatus: "preview". Turn on Skip paywalled to drop them (they are then not billed).

How does onlyNew work? The IDs of delivered articles are remembered in the named key-value store substack-medium-feeds-state, one record per source list. The next run with the same sources skips them, without billing. Delete that store to start over.

How many Medium stories can I get per author? Medium's public feeds hold roughly the 10 latest stories per author, publication or tag. There is no public feed for an author's full back catalogue.

Why is likes null for Medium? Medium does not publish clap counts in its public feeds, and this Actor does not scrape protected pages. A null is better than a made-up number.

Do Medium tag feeds return the full text? No, only an excerpt (contentStatus: "preview"). Author and publication feeds return the full text of non-member-only stories.

Does it need a proxy? No. Both platforms serve these public endpoints to normal HTTP clients.

What if a newsletter or author does not exist? It is logged as "not found", reported in the run status, and nothing is billed. The other sources continue.

Limitations

• Medium: about 10 latest stories per feed; no claps, comments or member-only full text.
• Substack: paid posts return a preview only; podcast/video posts return their text notes only.
• Substack @handle inputs resolve to the author's primary newsletter.
• Very large archives (thousands of posts) take a few minutes because the Actor deliberately paces its requests.