# Substack & Medium Scraper: Articles, Markdown, Likes (`actose/substack-medium-feeds`) Actor

Extract Substack & Medium articles as clean LLM-ready Markdown: title, date, author, likes, comments, tags, full text, RAG chunks. Full Substack archive, Medium authors, publications & tags. No API key, no login. MCP-ready for AI agents. Errors and empty results are free. $8 per 1,000 articles.

- **URL**: https://apify.com/actose/substack-medium-feeds.md
- **Developed by:** [ABM](https://apify.com/actose) (community)
- **Categories:** News, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $6.80 / 1,000 articles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper & Medium Scraper: Articles to Markdown

**Price: $8 per 1,000 articles ($0.008 per article).** Pay per result: no start fee, no monthly rental. Errors, blocked sources, empty results, duplicates and already-seen articles are free.

Turn any **Substack newsletter** or **Medium author, publication or tag** into clean, LLM-ready Markdown, with the publication date, author(s), likes, comments, tags, word count and paywall status. One Actor, both platforms, one consistent output format (JSON, CSV, Excel or Markdown files).

- **Substack**: reads the newsletter's full public archive (not just the last 20 posts of the RSS feed) through Substack's public JSON API. Free posts come back in full; for subscriber-only posts you get the public preview, clearly flagged.
- **Medium**: reads the public RSS feeds of authors, publications (including custom domains) and tags.
- **Clean Markdown**: subscribe boxes, share buttons, tracking pixels and the "originally published in…" footer are removed. Headings, lists, links, quotes, code blocks and image captions are kept.
- **RAG-ready**: optional `chunks` array (Markdown split on paragraph boundaries) for embeddings and vector databases.
- **Scheduled monitoring**: `onlyNew` delivers (and bills) only the articles published since your last run.
- **No API key, no login, no browser, no proxy needed**: fast and cheap. Only public data is read.

### How much does it cost?

| Event | Price |
|---|---|
| `article`: one article delivered to the dataset | $0.008 |

Examples: 100 articles = **$0.80** · 1,000 articles = **$8** · 10,000 articles = **$80**.
A daily schedule on 10 newsletters with `onlyNew` (≈ 1 new post each per day) costs about $2.40 per month.

Set *Max articles in total* or a maximum run cost in Apify Console to stay within your budget: the Actor stops cleanly at the limit and never delivers unpaid "preview" items.

### What you get for each article

| Field | Description |
|---|---|
| `platform` | `substack` or `medium` |
| `title`, `subtitle`, `url`, `slug`, `id` | Article identity |
| `author`, `authorHandle`, `authors[]` | Author name(s), handle, profile URL (Substack co-authors and guest authors included) |
| `publication`, `publicationUrl` | Newsletter or Medium publication |
| `publishedAt`, `updatedAt` | ISO 8601 dates (UTC) |
| `likes` | Substack hearts (reactions). `null` on Medium: claps are not exposed publicly |
| `comments`, `restacks` | Substack comment and restack counts |
| `tags` | Substack post tags / Medium categories |
| `wordCount`, `readingTimeMinutes` | Length |
| `isPaywalled`, `audience`, `contentStatus` | `contentStatus` = `full` or `preview`: you always know whether the text is complete |
| `markdown` | Self-contained Markdown document: title, byline line, then the body |
| `chunks` | Markdown split into RAG chunks (optional) |
| `html` | Original HTML (optional) |
| `markdownFileUrl` | Link to a `.md` file in the key-value store (optional) |

### Input example

```json
{
  "substackNewsletters": ["simonw", "www.noahpinion.blog", "@lenny"],
  "mediumSources": ["@karpathy", "netflixtechblog.com"],
  "mediumTags": ["llm"],
  "maxArticlesPerSource": 20,
  "publishedAfter": "2026-01-01",
  "onlyNew": false,
  "outputChunks": true,
  "chunkSize": 2000,
  "keepImages": true
}
```

Accepted formats:

- **Substack**: `simonw`, `https://simonw.substack.com`, `www.noahpinion.blog` (custom domain), `@lenny` or `https://substack.com/@lenny` (author, resolved to their main newsletter).
- **Medium**: `@karpathy`, `https://medium.com/@karpathy`, `karpathy.medium.com`, `https://medium.com/some-publication`, `netflixtechblog.com`, or tags in the *Medium tags* field (`llm`, `product-management`).

### Output example

```json
{
  "platform": "substack",
  "sourceInput": "simonw",
  "id": "171234567",
  "url": "https://simonw.substack.com/p/2026-in-llms-so-far",
  "slug": "2026-in-llms-so-far",
  "title": "2026 in LLMs (so far)",
  "subtitle": null,
  "author": "Simon Willison",
  "authorHandle": "simonw",
  "authors": [{ "name": "Simon Willison", "handle": "simonw", "profileUrl": "https://substack.com/@simonw" }],
  "publication": "Simon Willison’s Newsletter",
  "publicationSlug": "simonw",
  "publicationUrl": "https://simonw.substack.com",
  "publishedAt": "2026-09-28T16:42:27.944000Z",
  "updatedAt": null,
  "postType": "newsletter",
  "tags": [],
  "likes": 93,
  "comments": 2,
  "restacks": 11,
  "wordCount": 5120,
  "readingTimeMinutes": 22,
  "isPaywalled": false,
  "audience": "everyone",
  "contentStatus": "full",
  "language": "en",
  "coverImage": "https://substackcdn.com/image/fetch/...",
  "markdown": "# 2026 in LLMs (so far)\n\nBy Simon Willison · Simon Willison’s Newsletter · 2026-09-28 · https://simonw.substack.com/p/2026-in-llms-so-far\n\n---\n\nIn this newsletter:\n\n- ...",
  "chunks": ["# 2026 in LLMs (so far)\n\nBy Simon Willison ...", "..."],
  "scrapedAt": "2026-10-15T09:12:44.120000Z"
}
```

A Medium item has the same shape, with `likes`, `comments` and `restacks` set to `null`.

### Use with AI agents (MCP)

This Actor is MCP-ready: add it to Claude, Cursor, ChatGPT or any MCP client through the Apify MCP server (`https://mcp.apify.com`), then ask for example *"Get the last 10 posts of simonw on Substack as Markdown and summarise them"*. The input is a small JSON object, and every item carries a self-contained `markdown` field (plus optional `chunks`), so an agent can read the result directly without any HTML cleaning.

### Integrations

- **Apify API**: run synchronously and get the items in one call (`run-sync-get-dataset-items`), in JSON, CSV, Excel or XML.
- **Make, Zapier, n8n**: use the Apify app/node to start a run and loop over the dataset items.
- **Google Sheets**: export the dataset or use the Apify Google Sheets integration for a live sheet of new articles.
- **Schedules + webhooks**: schedule a daily run with `onlyNew: true` and send new articles to Slack, email or your vector database.

### Use cases

- **RAG and AI agents**: load a newsletter's whole archive into a vector database with the ready-made `chunks`.
- **Newsletter monitoring**: schedule the Actor daily with `onlyNew` to collect only new posts from a list of newsletters and authors.
- **Content research and competitive analysis**: compare publishing frequency, length and engagement (likes, comments, restacks) across Substack newsletters.
- **Summaries and digests**: pipe the Markdown into an LLM to produce a weekly digest of your favourite writers.
- **Archiving**: save each article as a `.md` file in the key-value store.

### How it works (and why it is reliable)

- Substack: `/api/v1/archive` (paged, newest first; the body is often included) and `/api/v1/posts/{slug}` only when the body is missing: structured JSON, no fragile HTML parsing.
- Medium: official RSS feeds (`medium.com/feed/@author`, `/feed/publication`, `/feed/tag/...`).
- Automatic retries with exponential back-off on network errors, HTTP 429 and 5xx; polite per-site delay and at most 2 parallel requests per site.
- One failing article or source never stops the run; each problem is logged in plain English and summarised in the run status.
- If a site refuses the direct connection, the Actor switches that site to the Apify datacenter proxy automatically. Residential proxy is never used unless you pick it.
- Duplicate URLs (same article from two inputs) are delivered and billed once.

### FAQ

**Can it get the full text of paid (subscriber-only) Substack posts?**
No. It never logs in. You get the public preview, flagged with `isPaywalled: true` and `contentStatus: "preview"`. Turn on *Skip paywalled* to drop them (they are then not billed).

**How does `onlyNew` work?**
The IDs of delivered articles are remembered in the named key-value store `substack-medium-feeds-state`, one record per source list. The next run with the same sources skips them, without billing. Delete that store to start over.

**How many Medium stories can I get per author?**
Medium's public feeds hold roughly the 10 latest stories per author, publication or tag. There is no public feed for an author's full back catalogue.

**Why is `likes` null for Medium?**
Medium does not publish clap counts in its public feeds, and this Actor does not scrape protected pages. A `null` is better than a made-up number.

**Do Medium tag feeds return the full text?**
No, only an excerpt (`contentStatus: "preview"`). Author and publication feeds return the full text of non-member-only stories.

**Does it need a proxy?**
No. Both platforms serve these public endpoints to normal HTTP clients.

**What if a newsletter or author does not exist?**
It is logged as "not found", reported in the run status, and nothing is billed. The other sources continue.

### Limitations

- Medium: about 10 latest stories per feed; no claps, comments or member-only full text.
- Substack: paid posts return a preview only; podcast/video posts return their text notes only.
- Substack `@handle` inputs resolve to the author's primary newsletter.
- Very large archives (thousands of posts) take a few minutes because the Actor deliberately paces its requests.

# Actor input Schema

## `substackNewsletters` (type: `array`):

One per line. Accepted: a subdomain (<code>simonw</code>), a URL (<code>https://simonw.substack.com</code>), a custom domain (<code>www.noahpinion.blog</code>) or an author handle (<code>@lenny</code>, resolved to the author's main newsletter). The full public archive is read through Substack's JSON API.

## `mediumSources` (type: `array`):

One per line. Accepted: an author (<code>@karpathy</code>, <code>https://medium.com/@karpathy</code>, <code>karpathy.medium.com</code>), a publication (<code>https://medium.com/better-programming</code>) or a custom-domain publication (<code>netflixtechblog.com</code>). Medium only exposes the ~10 latest stories of each feed.

## `mediumTags` (type: `array`):

Medium tags such as <code>llm</code> or <code>product-management</code>. Medium's tag feeds only give a short excerpt of the latest ~10 stories (contentStatus = preview).

## `maxArticlesPerSource` (type: `integer`):

Maximum number of articles delivered for each newsletter, author, publication or tag. Substack: newest first, the full archive can be read. Medium: the feed holds about 10 stories.

## `maxTotalArticles` (type: `integer`):

Hard cap on the number of articles delivered (and billed) for the whole run. Leave empty for no cap.

## `publishedAfter` (type: `string`):

Only keep articles published on or after this date (YYYY-MM-DD). Substack paging stops as soon as older posts are reached.

## `publishedBefore` (type: `string`):

Only keep articles published before this date (YYYY-MM-DD).

## `tagFilter` (type: `array`):

Optional. Keeps an article only if one of its tags matches (case-insensitive, e.g. <code>ai</code>, <code>machine-learning</code>). Works on Substack post tags and Medium categories.

## `includeFullText` (type: `boolean`):

On: one extra request per Substack post to get the complete body. Off: faster, only the archive excerpt is returned (contentStatus = preview).

## `skipPaywalled` (type: `boolean`):

On: subscriber-only posts (where only a preview is public) are not delivered and not billed.

## `keepImages` (type: `boolean`):

Keep image links (![caption](url)) in the Markdown. Turn off for text-only LLM input.

## `onlyNew` (type: `boolean`):

On: articles already delivered by an earlier run with the same source list are skipped (and not billed). The memory is kept in the named key-value store <code>substack-medium-feeds-state</code>. Ideal for a daily or weekly schedule.

## `outputChunks` (type: `boolean`):

On: each article also gets a <code>chunks</code> array (Markdown split on paragraph boundaries), ready for embeddings and vector databases.

## `chunkSize` (type: `integer`):

Target size of each RAG chunk when 'Add RAG chunks' is on (about 4 characters per token).

## `includeHtml` (type: `boolean`):

Also return the original article HTML in an <code>html</code> field.

## `saveMarkdownFiles` (type: `boolean`):

Store one Markdown file per article in the run's key-value store; the link is in <code>markdownFileUrl</code>.

## `requestDelayMs` (type: `integer`):

Polite pause before each request to the same site (randomised ±40%). Raise it if you see HTTP 429 messages.

## `proxyConfiguration` (type: `object`):

Not needed in normal use. If a site refuses the direct connection, the Actor falls back to the Apify datacenter proxy automatically. Residential proxy is never used unless you select it here (billed per GB).

## Actor input object example

```json
{
  "substackNewsletters": [
    "simonw",
    "www.noahpinion.blog",
    "www.astralcodexten.com"
  ],
  "mediumSources": [
    "@karpathy",
    "@netflixtechblog",
    "@addyosmani"
  ],
  "mediumTags": [],
  "maxArticlesPerSource": 5,
  "tagFilter": [],
  "includeFullText": true,
  "skipPaywalled": false,
  "keepImages": true,
  "onlyNew": false,
  "outputChunks": false,
  "chunkSize": 2000,
  "includeHtml": false,
  "saveMarkdownFiles": false,
  "requestDelayMs": 500,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `articles` (type: `string`):

No description

## `articlesJson` (type: `string`):

No description

## `markdownFiles` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "substackNewsletters": [
        "simonw",
        "www.noahpinion.blog",
        "www.astralcodexten.com"
    ],
    "mediumSources": [
        "@karpathy",
        "@netflixtechblog",
        "@addyosmani"
    ],
    "mediumTags": [],
    "maxArticlesPerSource": 5,
    "tagFilter": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("actose/substack-medium-feeds").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "substackNewsletters": [
        "simonw",
        "www.noahpinion.blog",
        "www.astralcodexten.com",
    ],
    "mediumSources": [
        "@karpathy",
        "@netflixtechblog",
        "@addyosmani",
    ],
    "mediumTags": [],
    "maxArticlesPerSource": 5,
    "tagFilter": [],
}

# Run the Actor and wait for it to finish
run = client.actor("actose/substack-medium-feeds").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "substackNewsletters": [
    "simonw",
    "www.noahpinion.blog",
    "www.astralcodexten.com"
  ],
  "mediumSources": [
    "@karpathy",
    "@netflixtechblog",
    "@addyosmani"
  ],
  "mediumTags": [],
  "maxArticlesPerSource": 5,
  "tagFilter": []
}' |
apify call actose/substack-medium-feeds --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,actose/substack-medium-feeds"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mlHZEZWPdQk25KiSH/builds/bJx1OQh1Lg5uL3ZKc/openapi.json