# Substack Newsletter to Markdown Scraper (`zucchini_gopher_m2v/substack-scraper`) Actor

Convert Substack newsletter posts to clean Markdown -- a single post by URL, or a whole publication's archive -- for AI ingestion (RAG, embeddings, LLM context). No account or API key needed.

- **URL**: https://apify.com/zucchini\_gopher\_m2v/substack-scraper.md
- **Developed by:** [Faisal Ahdan naufal](https://apify.com/zucchini_gopher_m2v) (community)
- **Categories:** Social media, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.90 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Newsletter to Markdown Scraper

Convert Substack newsletter posts into clean Markdown — ready to feed into
an AI pipeline (RAG, embeddings, LLM context) — either one post at a time or
a publication's entire archive. No account or login needed.

### Why use this actor

- Turns a newsletter post straight into clean, ready-to-use Markdown —
  title, subtitle, author, and date up top, then the body text, with the
  platform's own "Subscribe"/"Share" buttons stripped out
- Also returns the full original data for every post (title, word count,
  publish date, cover image, author info, and more) alongside the Markdown
- Crawl a single post by URL, or point it at a whole publication and get
  every post in the archive automatically, newest-first or top-first
- No account, login, or API key required
- Honest about paywalled posts — free posts return the full text; posts
  restricted to paying subscribers return the preview text the publication
  itself makes public, clearly flagged so you know which is which (see
  `content_truncated` below)
- Automatic retries on temporary network hiccups

### How it works

Pick a `mode`:

1. **`post`** — give it one or more newsletter post URLs; get back the
   Markdown and full details for each.
2. **`archive`** — give it a publication (its own domain, or just its
   Substack subdomain); get back that publication's profile plus every post
   in its archive, up to the limit you set.

Results stream into your dataset, ready to download as JSON, CSV, or open
directly for your AI pipeline.

### Input

**Single post:**

```json
{
  "mode": "post",
  "url": "https://noahpinion.substack.com/p/the-death-of-market-street"
}
```

**Whole publication archive:**

```json
{
  "mode": "archive",
  "url": "noahpinion.substack.com",
  "sort": "new",
  "maxItems": 50
}
```

| Field | Type | Description |
|---|---|---|
| `mode` | string | `"post"` (default) — convert specific post URL(s). `"archive"` — crawl a whole publication. |
| `url` | string | `post` mode: a single post URL. `archive` mode: a publication URL, custom domain, or bare subdomain (e.g. `"noahpinion"`, `"noahpinion.substack.com"`, or `"https://www.noahpinion.blog"`). |
| `urls` | array | Multiple post URLs (`post` mode) or multiple publications (`archive` mode) in one run. |
| `sort` | string | `archive` mode only — `"new"` (default, newest first) or `"top"` (most popular first). |
| `maxItems` | integer | `archive` mode only — max posts per publication. Default 50, max 10,000. Pagination is followed automatically until this cap or the end of the archive. |
| `maxConcurrency` | integer | How many URLs/publications to process in parallel (and, in `archive` mode, that publication's own per-post fetch concurrency). Default 4. |
| `proxyConfiguration` | object | Apify Proxy settings. Residential is on by default (kept for production robustness; see `CRAWLING_METHOD.md`). |

### Output

**`post` mode / `archive` mode's per-post records** — one record per post
(truncated here for readability; the full original post data is included
on every record alongside `markdown`):

```json
{
  "_input": "https://noahpinion.substack.com/p/the-death-of-market-street",
  "_source": "S1-api",
  "_scrapedAt": "2026-08-28T20:24:29Z",
  "title": "The death of Market Street",
  "subtitle": "How San Francisco turned its most iconic street into a giant empty scar.",
  "slug": "the-death-of-market-street",
  "post_date": "2026-08-26T09:12:35.276Z",
  "canonical_url": "https://www.noahpinion.blog/p/the-death-of-market-street",
  "audience": "everyone",
  "content_truncated": false,
  "wordcount": 2165,
  "publishedBylines": [{ "name": "Noah Smith", "handle": "noahpinion", "...": "5 more fields" }],
  "markdown": "# The death of Market Street\n\nHow San Francisco turned its most iconic street into a giant empty scar.\n\n*Noah Smith · 2026-08-26*\n\nhttps://www.noahpinion.blog/p/the-death-of-market-street\n\n---\n\n[![](https://substackcdn.com/image/fetch/...)](https://substackcdn.com/...)\n\nI still remember the first time I set foot on Market Street. ... <19,560 chars total>",
  "body_html": "<61,018 chars — the raw HTML markdown was converted from>",
  "...": "40+ more original fields (comments, reactions, cover image, etc.)"
}
```

**`archive` mode** — one `PUBLICATION_PROFILE` record per publication, then
one `POST` record per post found (same shape as above, plus `recordType`):

```json
{
  "_input": "noahpinion.substack.com",
  "_source": "S2-html",
  "recordType": "PUBLICATION_PROFILE",
  "name": "Noahpinion",
  "subdomain": "noahpinion",
  "custom_domain": "www.noahpinion.blog",
  "hero_text": "Economics and other interesting stuff",
  "...": "60+ more original fields"
}
```

| Field | Type | Description |
|---|---|---|
| `markdown` | string | The post converted to clean Markdown, with a title/subtitle/author/date header and the platform's own Subscribe/Share buttons stripped out. |
| `content_truncated` | boolean | `true` for posts restricted to paying subscribers — `markdown`/`body_html` is then the publication's own public preview, not the full post. |
| `title` / `subtitle` / `slug` / `post_date` / `canonical_url` | string | Post metadata, passed through as the publication provides it. |
| `audience` | string | `"everyone"` (free) or `"only_paid"` — see `content_truncated`. |
| `wordcount` | integer | The publication's own word count for the full post. |
| `publishedBylines` | array | Author(s) of the post. |
| `body_html` | string | The original HTML `markdown` was generated from. |
| `recordType` | string | `archive` mode only — `"PUBLICATION_PROFILE"` or `"POST"`. |
| `name` / `subdomain` / `custom_domain` / `hero_text` | string | `PUBLICATION_PROFILE` record — the newsletter's own name, domain, and tagline. |

A record with `_error` instead of the fields above means that input could
not be processed (for example, `"_error": "not_found"` for a post that
doesn't exist, or `"_error": "invalid_post_url"` if a URL isn't a Substack
post link).

### Known limits

- **Posts restricted to paying subscribers return a preview, not the full
  text.** This is the publication's own choice, enforced by Substack for
  every anonymous visitor — not something this actor can or should bypass.
  `content_truncated: true` flags exactly which records this applies to.
- **Reader comments are included verbatim when a post has them**, since
  they're part of the same data the publication returns for a post. If your
  pipeline only wants the article text, use the `markdown` field, which
  never includes comments.

### Store positioning (market research, 2026-08-29)

This is a saturated niche — 20+ distinct Substack-scraper sellers were found
on the Store during research for this actor, from 1 to 463 users and up to
\~95,000 total runs on the largest listing (`digispruce/substack-scraper`).
Two direct competitors already sell the same "Markdown for AI/RAG" angle
this actor leads with: `brilliant_gum/substack-insights-scraper` ($0.015 per
item, flat — only 242 runs to date) and `benthepythondev/newsletter-scraper`
(multi-platform, $0.001 per item at the FREE tier, marketed "LLM-Ready").
This actor's own differentiation is scope, not novelty: most competitors
bundle in Notes feeds, subscriber-email lead-gen, or nested comment
threads by default — this one stays to post↔Markdown conversion plus an
explicit `content_truncated` paywall flag (a transparency point only one
other listing, `hata1234/substack-scraper`, makes as plainly).

**Category:** Social Media (primary, matching 12 of the 20 comparable
listings) + AI (secondary — shared with the two direct Markdown/RAG
competitors above; not claimed by the lead-gen/email-harvest majority).

**Pricing:** Pay-per-event, single chargeable event
(`apify-default-dataset-item`, covering both `PUBLICATION_PROFILE` and
`POST` records) — no separate Actor-start fee, since this actor has no
warmup/auth cost to recoup (see `CRAWLING_METHOD.md`: no WAF, no login).
Comparable per-full-content-item prices found: $0.001 (`hata1234`,
`benthepythondev` FREE tier), $0.0023 (`automation-lab` "post-with-content"
FREE tier), $0.015 (`brilliant_gum`, an outlier with the lowest traction of
the direct-comparable set). Recommended: **$0.0015 per result** at the FREE
tier, tiering down to $0.0006 at DIAMOND (mirroring `benthepythondev`'s
tier spacing) — inside the sustainable $0.001–$0.0023 cluster, reflecting
that every record here always includes full Markdown content (not a
cheaper metadata-only tier the way `automation-lab`/`digispruce` split
theirs).

# Actor input Schema

## `mode` (type: `string`):

"post" converts specific post URL(s) to Markdown. "archive" crawls a whole publication's archive (paginated) and converts every post found to Markdown, plus one publication-profile record.

## `url` (type: `string`):

"post" mode: a single post URL (e.g. "https://noahpinion.substack.com/p/some-slug"). "archive" mode: a publication URL, custom domain, or bare subdomain (e.g. "noahpinion", "noahpinion.substack.com", or "https://www.noahpinion.blog").

## `urls` (type: `array`):

Multiple post URLs ("post" mode) or multiple publications ("archive" mode) to process in one run.

## `sort` (type: `string`):

"archive" mode only -- crawl newest posts first or top/most-popular first.

## `maxItems` (type: `integer`):

"archive" mode only -- max posts to fetch and convert per publication. Pagination is followed automatically until this cap or the end of the archive, whichever comes first.

## `maxConcurrency` (type: `integer`):

How many URLs/publications to process in parallel. In "archive" mode this is also used for that publication's own per-post fetch concurrency.

## `proxyConfiguration` (type: `object`):

Apify Proxy configuration. Residential is on by default. Not required for this target (no WAF was found -- see CRAWLING\_METHOD.md), but kept on for production robustness.

## Actor input object example

```json
{
  "mode": "post",
  "url": "https://noahpinion.substack.com/p/the-death-of-market-street",
  "sort": "new",
  "maxItems": 50,
  "maxConcurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped post / publication-profile records produced by this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "post",
    "url": "https://noahpinion.substack.com/p/the-death-of-market-street",
    "sort": "new",
    "maxItems": 50,
    "maxConcurrency": 4,
    "proxyConfiguration": {
        "useApifyProxy": true,
        "apifyProxyGroups": [
            "RESIDENTIAL"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("zucchini_gopher_m2v/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "post",
    "url": "https://noahpinion.substack.com/p/the-death-of-market-street",
    "sort": "new",
    "maxItems": 50,
    "maxConcurrency": 4,
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["RESIDENTIAL"],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("zucchini_gopher_m2v/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "post",
  "url": "https://noahpinion.substack.com/p/the-death-of-market-street",
  "sort": "new",
  "maxItems": 50,
  "maxConcurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}' |
apify call zucchini_gopher_m2v/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zucchini_gopher_m2v/substack-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YVV33LYr0a0nWcD3G/builds/93N15le54acGtWiyG/openapi.json