# Substack Publication Scraper (`zinin/substack-publication-scraper`) Actor

Read recent posts from any Substack publication's own domain — title, date, full HTML body, comment count and paywall status (audience) — via the official but undocumented JSON API. No login, no API key.

- **URL**: https://apify.com/zinin/substack-publication-scraper.md
- **Developed by:** [Tim Zinin](https://apify.com/zinin) (community)
- **Categories:** Social media, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.85 / 1,000 publication post delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Publication Scraper: Read Recent Posts From Any Substack Publication, Custom Domain Included, With Honest Paywall Status

![How this Actor turns your input into a decision](https://raw.githubusercontent.com/TimmyZinin/apify-actor-assets/e1ce54c6c199c6d374dde18443e244118994a66b/hub10/substack-publication-scraper/readme-hero.webp)
This Actor takes a Substack publication you already know — a bare domain like `astralcodexten.com`
or a full URL — and returns its most recent posts through the same JSON endpoint the publication's
own site already calls: `<publication-domain>/api/v1/posts?limit=N`. No login, no API key, no
browser. One HTTP GET per publication, parsed, returned as flat JSON: title, publish date, full HTML
body, comment count, canonical URL, and the source's own paywall flag.

This is deliberately narrow and deliberately honest about one thing before anything else: **a paid
publication's posts come back with an empty body.** Substack's own API answers a paywalled post with
`HTTP 200` and `audience: "only_paid"` — but `body_html` is a zero-length string. That is the
source's documented behavior, not a bug in this Actor, and it is not something this Actor tries to
work around. Every row where the body is empty is marked `status: "partial"` and is never billed as
delivered text. The next section, and Evidence and boundaries further down, show the actual bytes
from a paid and a free publication side by side.

The billable outcome is a **delivered post**: one row, read from a publication you supplied, with
`status: "ok"` — meaning the source actually handed over a non-empty post body. Every other outcome
is free: a paywalled post (`status: "partial"`, empty body, still delivered as a row but not sold as
text), a publication whose target domain's own `robots.txt` disallows this path
(`status: "blocked_by_robots"`, no request to the content path is made at all), and a domain that
isn't running a reachable Substack publication (`status: "error"`). This page shows real, unedited
JSON for all four outcomes, captured from actual local runs of this exact build.

### What you get

- **Full post HTML body**, title, publish date, comment count and canonical URL for the most recent
  posts of each publication you list — up to 50 per publication per run.
- **Honest paywall disclosure, proven with real numbers, not a promise.** A run against
  `bigtechnology.substack.com` returned three posts, all `audience: "only_paid"`, all with
  `bodyHtmlLength: 0` at `HTTP 200`. A same-day run against `astralcodexten.com` returned three
  posts, all `audience: "everyone"`, with real body lengths of 1,310 / 82,935 / 21,603 bytes — see
  Real happy, partial and failure output for the full rows. The `audience` field itself is handed to
  you exactly as the source sends it — this Actor does not interpret, hide, or normalize it away.
- **Works on a publication's own custom domain, not only `*.substack.com`.** Plenty of publications
  run on their own domain behind Substack's infrastructure. A live run against
  `bigtechnology.substack.com` was answered with an `HTTP 301` redirect to `www.bigtechnology.com`,
  and this Actor followed it — re-checking `robots.txt` on the new host before reading anything there,
  because a redirect target is a different domain with its own, potentially different, publisher
  policy.
- **A runtime `robots.txt` check on the domain actually being requested, on every run, not a
  build-time assumption.** This is the reason this Actor exists as its own build rather than a shared
  fallback: Substack publications live on domains their own publishers control, and this Actor cannot
  promise in advance what any one of those domains' `robots.txt` files will say six months from now —
  see Real happy, partial and failure output for a live `old.reddit.com` block, quoted `error` and
  all.
- **A discovered apex-vs-`www` quirk, handled automatically.** While verifying this build,
  `astralcodexten.com` (no `www`) answered `/api/v1/posts` with a direct `HTTP 404` — not a redirect —
  while `www.astralcodexten.com` answered `HTTP 200` on the identical path, a behavior not documented
  by Substack. This Actor retries once, through the same `robots.txt`-guarded path, with a `www.`
  prefix, whenever a bare non-`www.`, non-numeric host's first request fails — see Evidence and
  boundaries for the exact trigger condition.
- **No login, no API key, no paywall bypass, by design and by code.** This Actor sends no credentials
  of any kind. It reads exactly the same public, unauthenticated JSON response any anonymous visitor's
  browser would receive.
- **A hardened SSRF and host-confusion guard**, confirmed live against a real loopback address
  (`127.0.0.1` — see the SSRF error example below) and re-applied on every redirect hop, not only
  on the input value.
- **A billing guard that never double-charges under concurrency.** The charge and the dataset write
  happen inside one mutex-protected step, reused unmodified from the same billing-guard pattern
  already running on the other Actors in this hub — see Pricing for why that mattered enough to be
  worth documenting explicitly.
- Runs on Apify: schedule it, call it from the API or an MCP client, export results to JSON, CSV or
  Excel, or push rows straight into your own pipeline.

### Who uses it

- **Content analysts and agencies monitoring a specific set of newsletters** — competitive content
  research, publication-cadence tracking, or building a digest from a known list of Substack
  publications, rather than discovering new ones.
- **Competitive intelligence teams** who already know which publications matter to them and want the
  actual text of recent posts, not just a feed summary, to analyze themes, framing, or topic shifts
  over time.
- **Newsletter curators and digest builders** pulling the last few posts from several publications
  into one structured dataset for a roundup, without opening each site by hand.
- **Developers and agent/workflow builders** who want one flat JSON row per post — `status`,
  `audience`, `bodyHtmlLength` — that a script, an n8n node, or an LLM agent can branch on directly,
  without parsing Substack's HTML or guessing at an undocumented response shape themselves.
- **Not** a paywall bypass tool, under any framing — see Limits and Sources and rights. This is a
  boundary, not a missing feature this Actor intends to fill later.
- **Not** for comment mining. Substack's own `robots.txt` disallows `/p/*/comment/*`; this Actor
  never requests it. `commentCount` is metadata only — never a fetch of the thread.
- **Not** for a complete historical archive. This Actor reads the most recent `limit` posts per
  publication per run (up to 50), not the full back catalog, and does not paginate deeper into a
  publication's history. See Limits.

### How to run

1. Populate **Publications** with one or more values — a bare domain (`astralcodexten.com`) or a full
   URL (`https://www.astralcodexten.com`) both work. Up to 20 publications per run; duplicates are
   silently deduplicated before any request is made.
2. Leave **Posts per publication** at its default of `10`, or lower it toward the `3` used in this
   page's own evidence (1–50 range) — see Operating guide for what is and is not verified above `3`.
3. Leave **Max concurrency** at its default of `5` unless you have a reason to raise it (up to 15) or
   lower it (down to 1).
4. Press **Start**. The prefilled input (`["astralcodexten.com"]`) is a real, live, free publication —
   a default run with no changes produces a small, real, non-empty dataset at close to zero cost, so
   you can see the actual shape of the data before pointing this Actor at a larger list.
5. Read the Dataset: one row per post for a publication that returned content, or one summary row for
   a publication that was blocked by `robots.txt` or was not a reachable Substack publication. Every
   row shares the same 14-field shape, and a pre-built "Post overview" table view is also available
   in the Console — see Field dictionary for both.
6. Check `status` before trusting `bodyHtml`. `"ok"` is the only status where `bodyHtml` is guaranteed
   non-empty. `"partial"` means the row was delivered but the body is empty by the source's own
   design — see Pricing for why.
7. A run-level summary is written to the `OUTPUT` record of this run's key-value store: how many
   publications were attempted, how many rows were delivered, how many of those were charged and how
   many were free, how many failed, how many came back incomplete, whether re-running the same input
   is safe, and whether the run ended in a fatal error. Read it to reconcile your Apify invoice
   against what you actually received. The Dataset itself stays the authoritative record of the
   content; the summary tells you what happened to the run as a whole.

### Pricing

**$0.005 per Actor start, plus $0.001 per delivered post.** The primary pricing noun on this page, in
the Dataset, and in this Actor's platform pricing record is the same throughout: the **delivered
post** — one row, read from a publication you supplied, where the source actually handed over a
non-empty post body (`status: "ok"`). 100 delivered posts cost about **$0.105** in total — metered on
Apify's invoice as `apify-actor-start` + `result-found`.

A **paywalled post is never billed as delivered text.** A row with `status: "partial"` — the source
answered `HTTP 200` with `audience` set to something other than `"everyone"` and `body_html` empty —
is written to the Dataset for free, so you still see that the post exists, its title, its date, and
its paywall status, without paying for text you were never sent. A publication blocked by its own
`robots.txt` (`status: "blocked_by_robots"`) and a publication that could not be read at all
(`status: "error"`) are also always free.

This is enforced in code, not left as a documentation promise. Charging and writing a row happen
together, inside one mutex-protected step (`lib/billing.js`), and only a row already carrying
`status: "ok"` at that point is ever passed to the paid path; every other status is pushed through a
separate, unconditionally free path. `lib/billing.js`'s own header comment documents *why* this
matters here specifically: an earlier version of this pattern checked the remaining budget and then
pushed the row as two separate steps, which is safe only when a single caller runs at a time — this
Actor's default concurrency is 5 (up to 15), so two workers could otherwise both pass the budget check
before either had actually charged, double-billing the buyer at the budget boundary. The guard used
here is copied, not re-derived, from the same proven pattern already running on the other Actors in
this factory's hub, specifically to avoid reintroducing a bug that had already been found and fixed
once elsewhere.

This Actor's own external data cost is close to zero — one GET request per publication, roughly 68 KB
per post on the publications measured for this page, no proxy, no LLM, no paid third-party API — so
the per-post price reflects extraction, the paywall-honesty guarantee, the per-run robots guard and
delivery verification described on this page, not a resold data bill.

### Input contract

One field is required.

- **`publications`** — array of strings, up to 20 items, required. Each item is either a bare domain
  (`astralcodexten.com`) or a full URL (`https://www.astralcodexten.com`); this Actor normalizes both
  forms down to a lowercase hostname before making any request. A value is rejected before any network
  call (empty `input` handling, no request made) if it fails to parse as a URL once `https://` is
  prepended, if it carries embedded userinfo (`user@host` — a classic host-confusion vector where the
  readable "host" is not where a request would actually go), or if the parsed hostname contains `..`
  or a non-printable character. Duplicate hosts across the list are silently deduplicated.
- **`postsPerPublication`** — integer, 1 to 50, default `10`. How many of the most recent posts to
  request per publication (`limit=N` in the source URL). This page's own live evidence was gathered at
  `3`, matching this Actor's own acceptance-test goldens; see Operating guide for what is honestly
  unverified above that value.
- **`maxConcurrency`** — integer, 1 to 15, default `5`. How many publications this run reads at once.

Two runnable input examples, matching the values used to produce the live JSON in the next section —
these are input illustrations, not output; Real happy, partial and failure output below shows the
actual Dataset rows these exact inputs produced on a real run of this build.

```json
{
  "publications": ["astralcodexten.com"],
  "postsPerPublication": 3,
  "maxConcurrency": 1
}
```

*(A free, live publication. This is the Actor's own prefilled domain, run at the same
`postsPerPublication` value used throughout this page's evidence so the JSON below is exactly
reproducible.)*

```json
{
  "publications": [
    "bigtechnology.substack.com",
    "old.reddit.com",
    "example.com"
  ],
  "postsPerPublication": 3,
  "maxConcurrency": 1
}
```

*(Mixes three different outcomes in one batch — a paid publication, a `robots.txt`-blocked domain, and
a non-Substack domain — to show a buyer doesn't need to pre-filter their list. Each produces a
different, correctly labeled `status`, shown individually below.)*

### Real happy, partial and failure output

Every row in the Dataset shares one flat, 14-field shape regardless of outcome — `input`,
`publicationDomain`, `found`, `postId`, `title`, `postDate`, `audience`, `bodyHtmlLength`, `bodyHtml`,
`commentCount`, `canonicalUrl`, `status`, `error`, `checkedAt`. All five examples below are copied
verbatim from real local runs of this exact `main.js` build (`apify run --purge` against
`APIFY_LOCAL_STORAGE_DIR`), executed while this page was being written, against the live domains named
in each block. Nothing here is invented or reconstructed from the schema — every field value below is
what the Dataset actually contained after each run finished.

#### Happy path — a free publication, full text delivered

`astralcodexten.com` is a real, publicly accessible, free Substack publication. This exact row is the
full, untruncated Dataset entry for its most recent post at the time of this run:

```json
{
  "input": "astralcodexten.com",
  "publicationDomain": "www.astralcodexten.com",
  "found": true,
  "postId": 211610979,
  "title": "Open Thread 447",
  "postDate": "2026-08-17T19:52:17.873Z",
  "audience": "everyone",
  "bodyHtmlLength": 1310,
  "bodyHtml": "<p>This is the weekly visible open thread. Post about anything you want, ask random questions, whatever. ACX has an unofficial <a href=\"https://www.reddit.com/r/slatestarcodex/\">subreddit</a>, <a href=\"https://discord.gg/RTKtdut\">Discord</a>, and <a href=\"https://www.datasecretslox.com/index.php\">bulletin board</a>, and <a href=\"https://www.lesswrong.com/community?filters%5B0%5D=SSC\">in-person meetups around the world</a>. Most content is free, some is subscriber only; you can subscribe <strong><a href=\"https://astralcodexten.substack.com/subscribe?\">here</a></strong>. Also:</p><div><hr></div><p><strong>1: </strong>AI Security Bootcamp is running another (very timely) cohort, and has asked me to signal-boost them again.</p>",
  "commentCount": 9,
  "canonicalUrl": "https://www.astralcodexten.com/p/open-thread-447",
  "status": "ok",
  "error": null,
  "checkedAt": "2026-08-17T20:31:15.241Z"
}
```

Notice `publicationDomain` is `www.astralcodexten.com`, not the bare `astralcodexten.com` that was
supplied as input — the apex-fallback quirk described in What you get. The same run also returned two
larger posts from the same publication, confirming this is not a small-body coincidence:
"Your Book Review: The Escape Artist" at `bodyHtmlLength: 82935`, and "The Foothills Of Bay Area House
Party" at `bodyHtmlLength: 21603` — both also `audience: "everyone"`, `status: "ok"`.

#### Partial path — a real paid publication, proven not to leak paid text

`bigtechnology.substack.com` is a real, live, paid Substack publication. This is the full,
untruncated Dataset entry for its most recent post — note `bodyHtml` is a genuinely empty string, not
a truncated one:

```json
{
  "input": "bigtechnology.substack.com",
  "publicationDomain": "www.bigtechnology.com",
  "found": true,
  "postId": 209279848,
  "title": "When Artificial Intelligence Is Too Valuable To Sell",
  "postDate": "2026-07-31T19:30:17.712Z",
  "audience": "only_paid",
  "bodyHtmlLength": 0,
  "bodyHtml": "",
  "commentCount": 0,
  "canonicalUrl": "https://www.bigtechnology.com/p/when-artificial-intelligence-is-too",
  "status": "partial",
  "error": null,
  "checkedAt": "2026-08-17T20:31:30.461Z"
}
```

All three posts requested from this publication in the same run came back the same way:
`audience: "only_paid"`, `bodyHtmlLength: 0`, `status: "partial"` — "When Artificial Intelligence Is
Too Valuable To Sell", "Everything You Need to Know About Kimi K3, the Latest Model From China to
Shake Up AI", and "AI Model Prices Are Falling At The Worst Moment For The U.S. Frontier Labs". Note
also `publicationDomain: "www.bigtechnology.com"` — this publication's own domain, reached via the
`HTTP 301` redirect this Actor followed from `bigtechnology.substack.com`, with `robots.txt` on the
new host checked before the redirect was followed (see Evidence and boundaries). This run's log line
read exactly: `Done: 1 publication(s) with data, 0 failed of 1 (0 post rows billed)`.

#### Blocked path — a target domain whose own `robots.txt` disallows this path entirely

`old.reddit.com` is not a Substack publication, but it is a real, live domain whose own `robots.txt`
reads `User-Agent: * / Disallow: /` — everything is disallowed. This Actor checked that file before
attempting any content request and stopped:

```json
{
  "input": "old.reddit.com",
  "publicationDomain": "old.reddit.com",
  "found": false,
  "postId": null,
  "title": null,
  "postDate": null,
  "audience": null,
  "bodyHtmlLength": null,
  "bodyHtml": null,
  "commentCount": null,
  "canonicalUrl": null,
  "status": "blocked_by_robots",
  "error": "robots.txt on old.reddit.com disallows /api/v1/posts for User-agent: *",
  "checkedAt": "2026-08-17T20:31:38.449Z"
}
```

No request was made to `/api/v1/posts` on this host at all — this run's log line confirms the block
happened before the content fetch, not after an unauthorized read. This row is free.

#### Error path — a domain that is not a reachable Substack publication

`example.com` is a real, live, ordinary domain with no Substack publication behind it. Its own
`robots.txt` does not block anything relevant, so this Actor proceeded to the content request, which
the domain answered honestly with `HTTP 404`:

```json
{
  "input": "example.com",
  "publicationDomain": "example.com",
  "found": false,
  "postId": null,
  "title": null,
  "postDate": null,
  "audience": null,
  "bodyHtmlLength": null,
  "bodyHtml": null,
  "commentCount": null,
  "canonicalUrl": null,
  "status": "error",
  "error": "http 404",
  "checkedAt": "2026-08-17T20:31:45.133Z"
}
```

This row is also free — `status: "error"` is never billed, same as `"blocked_by_robots"` and
`"partial"`. The distinction that matters to you as a buyer: `"blocked_by_robots"` means the target
domain itself refused to be crawled and this Actor never attempted the content read; `"error"` means
the content read was attempted and the source answered that nothing was there.

#### A second error example — the SSRF/host guard, not a content-level error

A distinct kind of `"error"` row, worth showing separately because it never reaches the network at
all. `127.0.0.1` was supplied directly as a "publication domain" — a loopback address, the kind of
value a redirect-based SSRF attack would try to smuggle in:

```json
{
  "input": "127.0.0.1",
  "publicationDomain": "127.0.0.1",
  "found": false,
  "postId": null,
  "title": null,
  "postDate": null,
  "audience": null,
  "bodyHtmlLength": null,
  "bodyHtml": null,
  "commentCount": null,
  "canonicalUrl": null,
  "status": "error",
  "error": "blocked host (private/loopback/metadata)",
  "checkedAt": "2026-08-17T20:31:51.340Z"
}
```

This row is also free. The same guard code runs identically on every redirect hop this Actor follows,
not only on the initial host — see Evidence and boundaries for why this page can show the direct
loopback case live but not a combined redirect-to-private-IP case.

### Field dictionary

| Field | Meaning | Important boundary |
|---|---|---|
| `input` | Exactly the publication value you supplied, after normalization to a lowercase host | Echoed even when the request was rejected before any network call — lets you match every input back to its outcome |
| `publicationDomain` | The domain the post was actually read from, after following any redirect or apex-to-`www` fallback | Can differ from `input` — see the `bigtechnology.substack.com` → `www.bigtechnology.com` and `astralcodexten.com` → `www.astralcodexten.com` examples above |
| `found` | `true` once a post record was actually read from the source, `false` on every summary row | `true` even when `status` is `"partial"` — the post was found, its body just wasn't sent |
| `postId` | The post's numeric id from the source | `null` on every summary row (`blocked_by_robots`, `error`) |
| `title` | Post title as returned by the source | `null` on every summary row |
| `postDate` | Publish timestamp, ISO 8601, as returned by the source | `null` on every summary row |
| `audience` | The source's own access-level flag, passed through unchanged | `"everyone"` (free) or `"only_paid"` (paywalled) observed live; the source may use other values, and this Actor never interprets or renames any of them |
| `bodyHtmlLength` | Byte length of `bodyHtml` | Check this before trusting `bodyHtml` — it is `0` on every paywalled post observed |
| `bodyHtml` | Full post body HTML, verbatim from the source | Empty string, not `null`, on a paywalled post — this is deliberate: the post genuinely exists, its text just wasn't sent |
| `commentCount` | Comment count as reported by the source's own metadata | Not independently verified — this Actor never fetches the comment thread itself (disallowed path, see Sources and rights) |
| `canonicalUrl` | The post's public URL | `null` on every summary row |
| `status` | The billing and outcome boundary | `"ok"` = billed, delivered text. `"partial"` = free, delivered row, empty body. `"blocked_by_robots"` / `"error"` = free, summary row, no post data |
| `error` | `null` on every row where a post was actually read, including `"partial"` | Populated only on `"blocked_by_robots"` (quotes the disallowed path) and `"error"` (quotes the source's own HTTP status, or names the guard that rejected the request) |
| `checkedAt` | ISO-8601 timestamp of this row's outcome | Per-row, not per-run — rows from the same run can carry slightly different timestamps under concurrency |

A pre-built "Post overview" Dataset view (`.actor/dataset_schema.json`) exposes eight of these
fields — `publicationDomain`, `title`, `postDate`, `audience`, `bodyHtmlLength`, `commentCount`,
`status`, `found` — as a sortable table in the Console, without needing to open raw JSON rows.

### Evidence and boundaries

**The paywall is the first thing to understand about this Actor, and the evidence for it is above,
not asserted here.** A paid publication's posts arrive at `HTTP 200` with a real title, a real date, a
real `audience: "only_paid"` flag — and a `body_html` field that is a genuine zero-length string. This
was confirmed on all three posts requested from `bigtechnology.substack.com` in a single live run (see
Real happy, partial and failure output). This Actor's code checks `bodyHtml.length > 0` to decide
`status`, not the HTTP status code and not the presence of the `audience` field — a paywalled post at
`HTTP 200` is never mistaken for a delivered one, and is never billed as one.

**Custom domains are read like any other domain, with their own `robots.txt` check, because they
genuinely can differ from Substack's own file.** `bigtechnology.substack.com` redirected (`HTTP 301`)
to `www.bigtechnology.com` on the same day this page was written; this Actor followed that redirect
and checked `robots.txt` on `www.bigtechnology.com` specifically — the new host — before reading
anything there, not the `robots.txt` of the domain the request started at. In this instance, a direct
`curl` of both `www.astralcodexten.com/robots.txt` and `www.bigtechnology.com/robots.txt` showed
identical content to `substack.com/robots.txt` itself — all three are served by Substack's shared
platform infrastructure even on a custom domain, and none of the three disallow `/api/v1/posts`. That
match is not guaranteed to hold for every publication forever; a publisher who reconfigures their own
DNS or reverse-proxy setup could serve a different `robots.txt` on their custom domain at any time,
which is exactly why this Actor re-reads the file on the domain it is actually about to request, every
run, rather than trusting a build-time assumption about Substack's platform-wide file.

**The apex-vs-`www` retry is a discovered production quirk, not a documented Substack behavior.**
While verifying this build, a direct request to `https://astralcodexten.com/api/v1/posts?limit=3`
(no `www`) returned a genuine `HTTP 404` — not a redirect. The identical path on
`https://www.astralcodexten.com/api/v1/posts?limit=3` returned `HTTP 200`. This Actor's code retries
exactly once, through the same `robots.txt`-guarded fetch path, with a `www.` prefix, whenever a bare
non-`www.`, non-numeric host's first request fails — a different host still means a different
`robots.txt` check, not a bypass of the first one (see the `astralcodexten.com` happy-path example
above for the resulting `publicationDomain`). This retry is a resilience measure found empirically,
not asserted to cover every Substack publication's DNS configuration — only the one observed live.

**The `robots.txt` check runs live, on the target host, on every publication in every run — and is
cached only for the lifetime of a single run.** Within one run, if a run's input lists the same host
twice (after normalization) or a redirect returns to a previously-seen host, the second check reuses
the first fetch rather than re-requesting the same `robots.txt` file twice. Across separate runs there
is no persistence at all — a publisher who tightens their `robots.txt` after one run is picked up
correctly on the very next run. What this does **not** cover: a `robots.txt` file changing mid-run,
between the first and a later request to the same host in one run, is not detected — the ruling fetched
at the start of that run's first request to that host is the ruling used for the rest of that run. This
is a genuine, disclosed edge case, not something this evidence session triggered live.

**The SSRF/host guard, confirmed live against a real loopback address, applied identically on every
redirect hop.** `127.0.0.1` was rejected with `error: "blocked host (private/loopback/metadata)"`
before any request left the process (see the second error example above). The same
`resolveVerifiedAddresses()` check that stopped that direct case runs again on every redirect
`Location` header this Actor follows — including the real `bigtechnology.substack.com` →
`www.bigtechnology.com` redirect above — so a publication that tried to redirect into a private
address range would be stopped at that hop the same way. No live third party publishes a
redirect-to-private-IP endpoint to test that exact case end-to-end, so this page can confirm the guard
fires on a direct loopback input and can confirm the identical code path runs on every hop of a real
redirect — it cannot show a live example of both combined, because no safe, public target for that
combination exists to test against.

**Response size is capped; what happens to a request that hits the cap was not exercised live for
this page.** Every fetch this Actor makes is capped at 4,000,000 bytes and stops reading past that
point. The publications measured for this page returned roughly 68–204 KB total for 3 posts — nowhere
near the cap. If a very large `postsPerPublication` value against publications with unusually long
posts produced a response that got cut mid-stream, the truncated text would very likely fail JSON
parsing, and the code path for that failure returns `status: "error"` with a `"bad json: ..."` message
rather than returning corrupted partial content as if it were valid — this is what the code does on a
`JSON.parse` failure regardless of cause, but the specific truncation-at-cap scenario was not
independently reproduced during this evidence session. Treat it as a disclosed, reasoned expectation
from reading the code, not a live-verified fact.

**`postsPerPublication` and `maxConcurrency` above the values used in this page's evidence are not
independently load-tested here.** Every JSON example was captured at `postsPerPublication: 3`,
`maxConcurrency: 1`, matching this Actor's own acceptance-test goldens. The schema allows up to `50`
posts and `15` concurrent publications — real, enforced limits neither ceiling was pushed to during
this session. See Operating guide and Limits.

**This Actor's billing safety nets were not exercised on-platform for this page.** Every JSON example
above came from a local run (`apify run --purge`), which never engages the Apify platform's own
charging manager. Locally, `monetized` is `false` and every row is written through the same
unconditional path regardless of `status`; the log line `(0 post rows billed)` on the
`bigtechnology.substack.com` run above reflects that no `status: "ok"` row was produced, not a local
billing simulation. See Limits for what remains unverified about the mutex-protected guard itself.

### Decision routing

| `status` | Billable? | `bodyHtml` | What it means |
|---|---|---|---|
| `"ok"` | **Yes** | Non-empty | The source handed over the post's real text |
| `"partial"` | No | Empty string | The post exists but the source did not send its body — observed exclusively on paywalled posts (`audience` not `"everyone"`) in this Actor's evidence |
| `"blocked_by_robots"` | No | `null` (no post read) | The target domain's own `robots.txt` disallows this path; no content request was made |
| `"error"` | No | `null` (no post read) | The domain is not a reachable Substack publication, the request was rejected by the SSRF/host guard, or the source returned a non-200 response |

A workflow needs two rules to be correct: bill and use only rows where `status` is `"ok"`; and never
assume `"partial"` is an error just because `bodyHtml` is empty — check `audience` on a `"partial"` row
to see whether the reason was a paywall, the only reason observed in this Actor's own evidence.

### Commercial playbooks

#### Competitive content-cadence and theme tracking

An analyst or agency who already knows which Substack publications matter to a client's market runs
this Actor on a schedule against that fixed list, reading full post text to track publishing
frequency, topic shifts, and framing over time — instead of a title-only RSS feed or manually opening
each site.

#### Paywall-posture monitoring across a watchlist

A researcher tracking how aggressively a set of publications monetizes their content points this
Actor at the same publication list on a recurring schedule and reads the `audience` field on each run.
This Actor reports a single run's snapshot honestly — whether a specific post is free or paid right
now — and does not itself track change over time; a buyer who wants a trend line stores each run's
`audience` values in their own system and compares them across runs.

#### Digest and roundup content pipelines

A newsletter curator or content team feeds in the domains of several publications they already follow
and pulls the last few posts from each into one structured dataset, ready to filter to `status: "ok"`
and assemble into a roundup — without opening each site by hand or writing a bespoke parser for each
publication's own front end.

### Integration recipes

![Where this Actor sits in your workflow](https://raw.githubusercontent.com/TimmyZinin/apify-actor-assets/e1ce54c6c199c6d374dde18443e244118994a66b/hub10/substack-publication-scraper/readme-workflow.webp)

#### Scheduled watchlist

Configure an Apify Task with a fixed `publications` list and a daily or weekly schedule. Filter each
run's Dataset to `status: "ok"` before loading it into a digest, spreadsheet, or CRM — `"partial"`,
`"blocked_by_robots"` and `"error"` rows are the free, honest record of what did not deliver text, not
data to import as content.

#### Agent/MCP pattern

An agent that already knows a target publication's domain passes it directly into `publications`,
checks `status` before treating any row as usable content, and reads `audience` on a `"partial"` row
to explain to the end user *why* a specific post came back without text, rather than treating the
row as a silent failure to retry.

#### Spreadsheet/catalog append pattern

Dataset rows are appended keyed by `postId` for `"ok"`/`"partial"` rows (both represent a genuinely
found post) and by `input` + `checkedAt` for `"blocked_by_robots"`/`"error"` summary rows, which carry
no `postId`. `null` fields are preserved as `null`, not coerced into an empty string, so a downstream
query can tell "this post's body is empty because it's paywalled" apart from "this field was never
populated for this row type."

### Operating guide

**Choosing `postsPerPublication`.** The default of `10` is the schema default, not the value this
page's own live evidence was gathered at — every JSON example above used `3`, matching this Actor's
own acceptance-test goldens. Raising it fetches more posts in the same single request per publication
(the source's own `limit=N` parameter); it does not change the per-post price or add a second request.
Values well above `3` have not been independently measured for response size or truncation risk during
this evidence session — see Evidence and boundaries on the 4 MB response cap.

**Choosing `maxConcurrency`.** The default of `5` is a conservative starting point for reading several
publications in parallel; raising it (up to `15`) speeds up a run with many publications in one input
without changing what any individual publication's own request returns. This page's own evidence was
gathered at `maxConcurrency: 1` to keep each example's log output attributable to one publication at a
time.

**Reading a `"partial"` row correctly.** Check `audience` — every `"partial"` row observed while
building this page was a paywall case (`audience` not `"everyone"`), though the field technically
just means "delivered but the body was empty." This Actor never bills that row regardless of the
specific reason.

**Reading a `"blocked_by_robots"` row correctly.** This means the target domain's own `robots.txt`
disallowed the request path at the moment this run made the check — not that the publication doesn't
exist, and not a permanent verdict. Re-running against the same domain later can produce a different
result if that publisher changes their file; this Actor checks fresh on every run rather than caching
a verdict across runs (see Evidence and boundaries for the one within-run caching caveat).

**Handling redirects and the apex/`www` quirk.** You do not need to pre-resolve which exact host a
publication's API actually lives on — supply the domain you know, and check `publicationDomain` in the
result to see where the data actually came from. Both the redirect-follow path (proven on
`bigtechnology.substack.com` → `www.bigtechnology.com`) and the failure-triggered `www.` retry (proven
on `astralcodexten.com` → `www.astralcodexten.com`) run automatically and are both re-checked against
`robots.txt` on the resulting host before any content is read.

#### Related tools

Related tools for adjacent workflows in social platforms and regional directories.

| Actor | What it does |
|---|---|
| [YouTube Playlist Scraper](https://apify.com/zinin/youtube-playlist-scraper) | Pair it in the social platforms and regional directories workflow: Scrape any YouTube playlist by id or URL — including other people's public playlists, not just your own... |
| [Bluesky Profile Scraper](https://apify.com/zinin/bluesky-profile-scraper) | Pair it in the social platforms and regional directories workflow: Batch-fetch public Bluesky profiles by handle or DID via the official keyless AppView API — display name,... |
| [YouTube Channel Intel](https://apify.com/zinin/youtube-channel-intel) | Pair it in the social platforms and regional directories workflow: Public intel card for any YouTube channel: subscribers, total views, video count, join date, country, full... |
| [YouTube Channel Videos List](https://apify.com/zinin/youtube-channel-videos-list) | Pair it in the social platforms and regional directories workflow: List up to 100 most recent uploads for any YouTube channel: videoId, title, view count, and publish age... |
| [YouTube Channel Lookalike Finder](https://apify.com/zinin/youtube-channel-lookalike-finder) | Pair it in the social platforms and regional directories workflow: Find the similar/lookalike channels YouTube itself shows on any channel's About page: channelId, name,... |

### FAQ

#### Does this bypass paywalls?

No, never, under any input. A paywalled post's body is empty because Substack's own API does not send
it to an unauthenticated client — this Actor reports that honestly (`status: "partial"`,
`bodyHtmlLength: 0`) instead of pretending it has the text, and it sends no login, session, or
subscription credential of any kind that could change that answer.

#### Does it work on a publication's own custom domain, or only `*.substack.com` addresses?

Both. `bigtechnology.substack.com` in this page's own evidence resolved, via a real `HTTP 301`
redirect, to `www.bigtechnology.com` — the publication's custom domain — and this Actor followed it,
checking `robots.txt` on the destination before reading anything there.

#### What happens if a publication's domain blocks crawling?

Its posts come back as one free row with `status: "blocked_by_robots"`, quoting the exact disallowed
path from that domain's own `robots.txt`, and no request is ever made to the content path. This is
checked fresh on every run.

#### What happens if I give it a domain that isn't a Substack publication?

A free row with `status: "error"` and the source's own response quoted or summarized in `error` — an
`HTTP 404` from the domain in this page's own `example.com` example, or a similar honest failure for
another non-Substack domain.

#### Does it return comments?

No. Substack's own `robots.txt` disallows `/p/*/comment/*`, and this Actor's code never requests that
path under any input. `commentCount` is a number reported in the post's own metadata, not a fetch of
the thread itself.

#### Can it read a subscriber's private, logged-in feed?

No. This Actor sends no authentication of any kind and only ever reads what an anonymous visitor's
request to the public JSON endpoint would receive — the same boundary that produces the paywall
behavior above also means a reader's personal, logged-in feed is not something this Actor can access.

#### Can I get a publication's entire archive?

Not in one run. This Actor requests the most recent `limit` posts (`postsPerPublication`, up to 50)
per publication per run and does not paginate deeper into a publication's history. See Limits.

#### Is `/api/v1/posts` an official, documented Substack API?

No — it is a real, currently reachable JSON endpoint that Substack's own publication front ends call
to render posts, but it is not published as a stable, versioned public API with a support commitment
from Substack. See Sources and rights for what that means as a risk, stated plainly rather than
buried.

#### What happens under high concurrency — can two workers double-charge the same row?

No. The charge and the dataset write for a `status: "ok"` row happen together inside one
mutex-protected step in `lib/billing.js`, specifically to close a race where two concurrent workers
could otherwise both pass a budget check before either had charged. See Pricing for why this Actor's
default concurrency of `5` made that race worth fixing, not theoretical.

Found a wrong result, or need a check this Actor doesn't run? Open an issue on this Actor's page.

### Sources and rights

The one path this Actor requests is public and allowed by robots.txt, re-checked on the publication's own domain every run.

**The only source.** The JSON response at `https://<publication-domain>/api/v1/posts?limit=N` — the
same endpoint the publication's own site calls to render its posts, fetched anonymously, with no login
and no API key. No RSS feed, no email-digest scraping, no browser rendering.

**This is an official-but-undocumented endpoint, and that is a genuine, disclosed risk, not a
footnote.** Substack has not published `/api/v1/posts` as a stable, versioned, supported public API.
It is real and currently reachable — every JSON example on this page was read from it live — but the
platform could change its shape, rename it, or restrict it without prior notice, the way any internal
endpoint a platform did not commit to as a public contract can change. A buyer relying on this Actor
for ongoing monitoring should know that risk exists before relying on it, not discover it the day a
scheduled run starts failing.

**`robots.txt`, checked live on the actual target domain, not cached from a prior date.** A direct
request to `substack.com/robots.txt`, made independently while writing this page, disallows
`/action/`, `/publish`, `/sign-in`, `/channel-frame`, `/session-attribution-frame`,
`/visited-surface-frame`, `/feed/private`, `/feed/podcast/*/private/*.rss`, `/subscribe`,
`/lovestack/*`, `/p/*/comment/*`, `/inbox/post/*`, `/notes/post/*`, and `/embed` for `User-agent: *`.
`/api/v1/posts` is not named by any of those rules. The same file, byte-for-byte identical, was
independently confirmed live on both `www.astralcodexten.com/robots.txt` and
`www.bigtechnology.com/robots.txt` — both custom domains, both served by Substack's own shared
platform infrastructure, both permitting this path. This Actor re-checks the file on whichever domain
it is actually about to request — not `substack.com`'s file by assumption — specifically because a
publisher's own custom-domain configuration is not guaranteed to match Substack's platform default
forever.

**No explicit reuse licence.** Substack publishes no equivalent of an open-data licence for the
content of its posts. This Actor's engineering posture is limited to what `robots.txt` permits to be
crawled and what a genuinely anonymous, unauthenticated request receives; it does not constitute legal
advice about how you may use the resulting post text downstream, and each post's own text remains the
copyrighted work of its author regardless of how this Actor reads it.

**No paywall bypass, as a rule this Actor's code follows, not a claim made only on this page.**
Nothing in this Actor's logic attempts a login, a subscription purchase, or any other method of
reaching content behind a paywall — see Evidence and boundaries for the exact check the code runs.
A paywalled post is reported honestly and left there.

### Limits

**A paid publication's post body is empty, at `HTTP 200`, and this is never sold as delivered text.**
This is the single most important thing to understand about this Actor before using it: `audience` not
equal to `"everyone"` means `bodyHtmlLength: 0`, `status: "partial"`, and no charge — confirmed live on
three real posts from a real paid publication in Real happy, partial and failure output above. There is
no configuration, input flag, or future update path in this Actor that changes this; a paywalled post's
text is simply not something the source sends to an unauthenticated request, and this Actor does not
attempt to obtain it any other way.

- **No comments.** Substack's own `robots.txt` disallows the comment-thread path; this Actor's code
  never requests it, on any input. `commentCount` is a number from the post's own metadata only.
- **No private, logged-in reader feed.** This Actor sends no authentication of any kind.
- **No full archive.** Up to 50 most recent posts per publication per run, no deeper pagination — a
  publication with hundreds of posts is not fully retrievable in one run, and this Actor does not
  attempt to cross multiple runs into one combined archive itself.
- **The source endpoint is officially undocumented.** `/api/v1/posts` works today, verified live on
  this page's own evidence, but Substack has made no public commitment to keep its shape or
  availability stable. See Sources and rights.
- **`robots.txt` is cached only for the lifetime of one run.** A change to a target domain's file
  mid-run, between two requests to the same host within that run, would not be detected until the next
  run. Across separate runs, the check is always fresh.
- **The 4 MB per-request response cap has not been tested at its actual truncation point.** The
  publications measured for this page returned 68–204 KB total for 3 posts, nowhere near the cap; the
  behavior described for a genuinely truncated response (a JSON-parse failure surfacing as
  `status: "error"`) is reasoned from the code, not independently reproduced live.
- **`postsPerPublication` above `3` and `maxConcurrency` above `1` were not the values used to gather
  this page's own live JSON evidence**, even though both are real, enforced limits (`50` and `15`
  respectively) in the input schema and in `main.js`. See Evidence and boundaries and Operating guide.
- **The apex-to-`www` retry and the redirect-follow path were each confirmed on exactly one real
  publication.** Both mechanisms are general code paths, not special-cased to `astralcodexten.com` or
  `bigtechnology.substack.com` specifically, but this page's live evidence covers one concrete example
  of each, not an exhaustive survey of every Substack publication's DNS configuration.
- **This Actor's platform billing safety nets — the double-charge-proof guard, the
  pricing-misconfiguration halt, the mid-run budget stop — are real, reviewed code paths, but none of
  them were exercised during this evidence-gathering session**, because a local, off-platform run never
  engages the platform's own charging manager. They are documented from the code and from the shared
  billing-guard pattern this Actor reuses from elsewhere in this hub, not from a monetized live
  observation made specifically for this page.
- **`maxConcurrency` up to `15` has not been load-tested against a large batch of publications for
  this build** — the default of `5` is the conservative, documented starting point.

### Support boundary

Actor support covers this Actor's own input validation, the paywall-honesty behavior, the per-run
`robots.txt` guard on the actual target domain, the SSRF/host-confusion guard, the apex-to-`www`
retry, the redirect-follow behavior, and the Dataset schema described on this page — all backed by a
live, verified run referenced above. It does not cover whether Substack's undocumented
`/api/v1/posts` endpoint remains reachable or unchanged in its response shape going forward, does not
restore access if a specific publication's markup, API, or anti-automation posture changes, does not
provide a way to read paywalled content under any circumstance, and does not provide legal guidance on
how the resulting post text may be reused downstream. When reporting a problem, include the run id,
the exact publication value you supplied, the `status` and `error` returned, and the `checkedAt`
timestamp of the affected row.

***

Built by [zinin](https://apify.com/zinin). Questions? Telegram [@timzinin](https://t.me/timzinin).

# Actor input Schema

## `publications` (type: `array`):

Substack publication domains or full URLs, one per line. Works for both \*.substack.com addresses and publications on their own custom domain (e.g. astralcodexten.com).

## `postsPerPublication` (type: `integer`):

How many of the most recent posts to fetch per publication.

## `maxConcurrency` (type: `integer`):

How many publications to read in parallel.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten.com"
  ],
  "postsPerPublication": 10,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

API URL for the default dataset items produced by this run.

## `runSummary` (type: `string`):

One JSON object written once at the end of this run: requestedCount, deliveredRows, paidRows, freeRows, failedCount, incompleteRows, replaySafe + replaySafeReason, fatalError + fatalErrorReason, generatedAt.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("zinin/substack-publication-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": ["astralcodexten.com"] }

# Run the Actor and wait for it to finish
run = client.actor("zinin/substack-publication-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten.com"
  ]
}' |
apify call zinin/substack-publication-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zinin/substack-publication-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Xc3gloB8G6P5tQ5AZ/builds/yRv4t05JPUTMR12bp/openapi.json