# Substack Public RSS Posts Scraper (`w3crawler/substack-scraper`) Actor

Extract structured posts, publication metadata, categories, and optional public body content from newsletter RSS feeds.

- **URL**: https://apify.com/w3crawler/substack-scraper.md
- **Developed by:** [w3crawler](https://apify.com/w3crawler) (community)
- **Features:** Limited permissions
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 posts

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Public RSS Posts Scraper

**Substack Scraper** extracts public newsletter posts through Substack-compatible RSS feeds. Supply Substack names, subdomains, custom publication domains, or complete publication URLs and receive structured post titles, URLs, stable identities, authors, publication dates, descriptions, optional body HTML and text, categories, enclosures, and publication metadata. No account or API key is required.

### Get started

1. Open the [Actor Store page](https://apify.com/w3crawler/substack-scraper) and select **Try for free** or open the **Input** tab.
2. Enter one or more publications in `publications`, such as `noahpinion.substack.com`, `https://www.noahpinion.blog/`, or a full feed URL.
3. Optionally narrow the result with `categoryAnyOf`, `publishedAfter`, or `containsKeyword`, and set `maxItems`.
4. Start the Actor, open the dataset, and export the saved records as JSON, CSV, or Excel. Open **Run summary** (`OUTPUT`) to check the outcome and counts.

### Input

`publications` is required; the other fields are optional. Unsupported field names and out-of-range values are rejected before requests start.

| Field | Type | Required? | Default | Meaning |
| --- | --- | --- | --- | --- |
| `publications` | string array | Required | `["noahpinion.substack.com"]` (schema prefill) | One to 50 publication URLs, domains, Substack subdomains, or short Substack names. Replace the sample with your own targets. |
| `categoryAnyOf` | string array | Optional | `[]` | Keep posts matching at least one RSS category, case-insensitively. Up to 50 values, each up to 200 characters. |
| `publishedAfter` | string | Optional | Not supplied | Inclusive lower date boundary in `YYYY-MM-DD` form; a date without a time zone. |
| `containsKeyword` | string | Optional | Not supplied | Keep posts whose title or description contains this text, case-insensitively; up to 200 characters. |
| `includeBody` | boolean | Optional | `true` | Include `content:encoded` HTML when the public feed exposes it. |
| `maxItems` | integer | Optional | `50` | Maximum unique posts saved across all feeds, from 1 to 1,000. |
| `minDelayMs` | integer | Optional | `500` | Minimum delay between publication feeds, from 250 to 10,000 ms. |
| `requestTimeoutSecs` | integer | Optional | `30` | Per-feed request timeout, from 10 to 90 seconds. |

Default input:

```json
{
  "publications": [
    "noahpinion.substack.com"
  ],
  "categoryAnyOf": [],
  "includeBody": true,
  "maxItems": 50,
  "minDelayMs": 500,
  "requestTimeoutSecs": 30
}
```

The Actor reads public data from Substack over anonymous HTTPS RSS requests. It does not use a browser, account, cookie, API key, or proxy.

### Output

The dataset contains one row per unique post accepted by the filters. Optional information appears only when the public feed publishes it; missing values are omitted rather than inferred. No diagnostic or placeholder rows are written.

| Field | Type | Availability | Meaning |
| --- | --- | --- | --- |
| `recordId` | string | Always | Deterministic row ID built from the normalized post URL and a short hash. |
| `recordType` | string enum | Always | Row discriminator; always `substackPost`. |
| `postId` | string | Always | Stable source identity from the RSS `guid`, falling back to the canonical post URL. The saved rows use the post URL, not a numeric Substack API ID. |
| `title` | string | Always | Public RSS post title. |
| `url` | string | Always | Canonical public post URL from the feed item. |
| `author` | string | When published | RSS author or byline name when the feed supplies one; present in the archived rows. |
| `publishedAt` | string | When published | RSS publication time in ISO UTC. |
| `description` | string | When published | Feed description or summary text; present in the archived rows. |
| `bodyHtml` | string | When `includeBody` is enabled and the feed exposes it | `content:encoded` HTML from the public feed. |
| `bodyText` | string | When the feed exposes body HTML or a description | Plain text derived from public body HTML, or from the feed description when no body HTML is exposed; capped in length. |
| `categories` | string array | When the feed exposes RSS categories | RSS category labels. No archived row in this storage contains them, so the example omits the field. |
| `enclosure` | object | When the feed exposes an enclosure | RSS enclosure with `url`, `type`, and declared `length`. It can describe audio or an image/media attachment; the saved example is an image enclosure with `length: 0`, not a podcast episode. |
| `publication` | object | When the feed exposes channel metadata | Publication object with `title`, `url`, `feedUrl`, `description`, `imageUrl`, and `generator`. |
| `source` | string | Always | Source label; `public_substack_rss`. |
| `provenance` | string | Always | Extraction lineage label; `public_newsletter_rss_feed`. |
| `sourceUrl` | string | Always | Final public feed URL used for the row. |
| `extractionMethod` | string | Always | Extraction label; `rss_xml`. |
| `scrapedAt` | string | Always | Time the row was written, in ISO UTC. |

`bodyHtml` and `bodyText` are only present when the feed carries `content:encoded`. A feed can expose a description, enclosure, or categories without exposing the full body. `postId` repeats the normalized post URL because that is how this RSS Actor identifies posts; it is not a numeric ID from Substack's API.

The example below is a real saved row from the 2026-10-25 run. Long `bodyHtml` and `bodyText` strings are shortened with an ellipsis; all other values are complete.

```json
{
  "recordId": "substack_post_https-www-noahpinion-blog-p-the-internet-breaks_78fa83ccab0f",
  "recordType": "substackPost",
  "postId": "https://www.noahpinion.blog/p/the-internet-breaks",
  "title": "The internet breaks",
  "url": "https://www.noahpinion.blog/p/the-internet-breaks",
  "author": "Noah Smith",
  "publishedAt": "2026-10-25T07:19:36.000Z",
  "description": "What will be next to break?",
  "bodyHtml": "<div class=\"captioned-image-container\"><figure><a class=\"image-link image2 is-viewable-img\" target=\"_blank\" href=\"https://substackcdn.com/image/fetch/$s_!o244!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7188959d-9dd1-4495-b047-a37f0ee21d86_1075x717.jpeg\" data-component-name=\"Image2ToDOM\"><div class=\"image2-inset\"><picture><source …",
  "bodyText": "Art by GPT-6“No man could bind him, no, not with chains…neither could any man tame him” — Mark 5:3—4When I was 14 or 15 — I don’t quite remember — I first got access to the World Wide Web. I remember clicking on the little blue square of the Netscape Navigator icon — that little ship wheel with a starry sky behind it — and feeling like an entire new magical world had opened up. My life changed tha…",
  "enclosure": {
    "url": "https://substackcdn.com/image/fetch/$s_!o244!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7188959d-9dd1-4495-b047-a37f0ee21d86_1075x717.jpeg",
    "type": "image/jpeg",
    "length": 0
  },
  "publication": {
    "title": "Noahpinion",
    "url": "https://www.noahpinion.blog/",
    "feedUrl": "https://www.noahpinion.blog/feed",
    "description": "Economics and other interesting stuff",
    "imageUrl": "https://substackcdn.com/image/fetch/$s_!l14h!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F04281755-2cd6-42e5-a496-e69153abebb2_281x281.png",
    "generator": "Substack"
  },
  "source": "public_substack_rss",
  "provenance": "public_newsletter_rss_feed",
  "sourceUrl": "https://www.noahpinion.blog/feed",
  "extractionMethod": "rss_xml",
  "scrapedAt": "2026-10-25T21:09:12.601Z"
}
```

### Run summary

**Run summary** is stored separately from the dataset in the run's key-value store under `OUTPUT`.

| Field | Meaning |
| --- | --- |
| `recordType` | Summary discriminator; `substack_run_summary`. |
| `status` | Run outcome; the saved example records `SUCCEEDED`. |
| `runStatus` | Canonical run status; matches `status`. |
| `dataStatus` | Dataset completeness: `COMPLETE`, `PARTIAL`, `NO_RESULTS`, or `FAILED`. |
| `dataAvailable` | Whether at least one source-backed post row was saved. |
| `sourceUrl` | Requested feed URL, before any custom-domain redirect. |
| `requestedPublicationCount` | Number of validated publications requested. |
| `requestedMaxItems` | Global post limit requested. |
| `itemsEmitted` | Unique post rows written. |
| `itemCount` | Compatibility count equal to `itemsEmitted`. |
| `diagnosticCount` | Operational issues recorded for the run. |
| `duplicateCount` | Duplicate post identities skipped across feeds. |
| `failureCount` | Feed or summary-write failures recorded. |
| `outputWriteFailureCount` | Failed key-value summary writes. |
| `filters` | Applied `categoryAnyOf`, `containsKeyword`, and `includeBody` values. |
| `source` | Source label; `public_substack_rss`. |
| `provenance` | Extraction lineage; `public_newsletter_rss_feed`. |
| `completedAt` | Time the summary was written, in ISO UTC. |

| Status | Data status | Meaning |
| --- | --- | --- |
| `SUCCEEDED` | `COMPLETE` | Eligible posts were saved without an extraction problem. |
| `SUCCEEDED` | `PARTIAL` | Posts were saved, but a later feed or parse problem prevented a complete run. Saved rows remain available. |
| `SUCCEEDED` | `NO_RESULTS` | The feeds completed cleanly and returned no posts matching the filters. |
| `FAILED` | `FAILED` | A fatal error occurred before any usable post was saved. |

Example `OUTPUT` from the saved run:

```json
{
  "recordType": "substack_run_summary",
  "status": "SUCCEEDED",
  "runStatus": "SUCCEEDED",
  "dataStatus": "COMPLETE",
  "dataAvailable": true,
  "sourceUrl": "https://noahpinion.substack.com/feed",
  "requestedPublicationCount": 1,
  "requestedMaxItems": 3,
  "itemsEmitted": 3,
  "itemCount": 3,
  "diagnosticCount": 0,
  "duplicateCount": 0,
  "failureCount": 0,
  "outputWriteFailureCount": 0,
  "filters": {
    "categoryAnyOf": [],
    "containsKeyword": "",
    "includeBody": true
  },
  "source": "public_substack_rss",
  "provenance": "public_newsletter_rss_feed",
  "completedAt": "2026-10-25T21:09:12.972Z"
}
```

The summary's `sourceUrl` is the requested Substack feed, while rows can record a custom-domain feed such as `https://www.noahpinion.blog/feed` after the publication redirects.

### Coverage and limitations

The Actor reads public RSS feeds only. It does not sign in, call a private API, bypass paywalls, or fetch a separate post page to invent missing body content. `bodyHtml` appears only when the feed includes `content:encoded` and `includeBody` is enabled. `bodyText` is derived from that body HTML or, when no body HTML is present, from the feed description; a description-only row is not full text.

`categoryAnyOf`, `publishedAfter`, and `containsKeyword` filter posts after the feed is read, so they reduce saved rows but do not reduce feed requests. `maxItems` is a global cap, and one row is written per unique post identity. Optional fields are omitted when the feed does not publish them; values are not inferred. Source availability can change, and blocked, empty, or partial feed responses are counted in `OUTPUT` rather than written as data rows.

### Usage and support

Cost depends on Apify compute and one HTTPS feed request per publication plus bounded retries. There is no browser or proxy step, so runs are lightweight; increasing `publications` increases request count, while filters mainly reduce output size. See the Actor's Pricing tab for current account rates.

Use the [Actor Issues page](https://apify.com/w3crawler/substack-scraper/issues) to report a problem. Include the run ID, the input used, and the `OUTPUT` summary; do not include tokens, cookies, or private data. This Actor is not affiliated with Substack. Use it only for public data and in accordance with the source's terms, robots guidance, and applicable law. Public posts can contain personal information; do not use the output for unauthorized profiling or contact.

# Actor input Schema

## `publications` (type: `array`):

Publication URLs, domains, Substack subdomains, or short Substack names.

## `categoryAnyOf` (type: `array`):

Keep posts matching at least one RSS category, case-insensitively.

## `publishedAfter` (type: `string`):

Optional inclusive lower date boundary in YYYY-MM-DD form.

## `containsKeyword` (type: `string`):

Keep posts whose title or description contains this text.

## `includeBody` (type: `boolean`):

Include content:encoded HTML when the public feed exposes it.

## `maxItems` (type: `integer`):

Maximum unique posts across all feeds.

## `minDelayMs` (type: `integer`):

Minimum delay between publication feeds.

## `requestTimeoutSecs` (type: `integer`):

Maximum seconds for a feed request.

## Actor input object example

```json
{
  "publications": [
    "noahpinion.substack.com"
  ],
  "categoryAnyOf": [],
  "includeBody": true,
  "maxItems": 5,
  "minDelayMs": 500,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset API URL containing public post and diagnostic records.

## `runSummary` (type: `string`):

Key-value record with run status, filters, counts, and failures.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "noahpinion.substack.com"
    ],
    "maxItems": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("w3crawler/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["noahpinion.substack.com"],
    "maxItems": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("w3crawler/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "noahpinion.substack.com"
  ],
  "maxItems": 5
}' |
apify call w3crawler/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,w3crawler/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wQGBm3ZWhRGf6g1dg/builds/Nl7xMe4Z5rXIIgyhc/openapi.json
