Substack Publication Scraper: Read Recent Posts From Any Substack Publication, Custom Domain Included, With Honest Paywall Status

This Actor takes a Substack publication you already know — a bare domain like astralcodexten.com or a full URL — and returns its most recent posts through the same JSON endpoint the publication's own site already calls: <publication-domain>/api/v1/posts?limit=N. No login, no API key, no browser. One HTTP GET per publication, parsed, returned as flat JSON: title, publish date, full HTML body, comment count, canonical URL, and the source's own paywall flag.

This is deliberately narrow and deliberately honest about one thing before anything else: a paid publication's posts come back with an empty body. Substack's own API answers a paywalled post with HTTP 200 and audience: "only_paid" — but body_html is a zero-length string. That is the source's documented behavior, not a bug in this Actor, and it is not something this Actor tries to work around. Every row where the body is empty is marked status: "partial" and is never billed as delivered text. The next section, and Evidence and boundaries further down, show the actual bytes from a paid and a free publication side by side.

The billable outcome is a delivered post: one row, read from a publication you supplied, with status: "ok" — meaning the source actually handed over a non-empty post body. Every other outcome is free: a paywalled post (status: "partial", empty body, still delivered as a row but not sold as text), a publication whose target domain's own robots.txt disallows this path (status: "blocked_by_robots", no request to the content path is made at all), and a domain that isn't running a reachable Substack publication (status: "error"). This page shows real, unedited JSON for all four outcomes, captured from actual local runs of this exact build.

What you get

• Full post HTML body, title, publish date, comment count and canonical URL for the most recent posts of each publication you list — up to 50 per publication per run.
• Honest paywall disclosure, proven with real numbers, not a promise. A run against bigtechnology.substack.com returned three posts, all audience: "only_paid", all with bodyHtmlLength: 0 at HTTP 200. A same-day run against astralcodexten.com returned three posts, all audience: "everyone", with real body lengths of 1,310 / 82,935 / 21,603 bytes — see Real happy, partial and failure output for the full rows. The audience field itself is handed to you exactly as the source sends it — this Actor does not interpret, hide, or normalize it away.
• Works on a publication's own custom domain, not only *.substack.com. Plenty of publications run on their own domain behind Substack's infrastructure. A live run against bigtechnology.substack.com was answered with an HTTP 301 redirect to www.bigtechnology.com, and this Actor followed it — re-checking robots.txt on the new host before reading anything there, because a redirect target is a different domain with its own, potentially different, publisher policy.
• A runtime robots.txt check on the domain actually being requested, on every run, not a build-time assumption. This is the reason this Actor exists as its own build rather than a shared fallback: Substack publications live on domains their own publishers control, and this Actor cannot promise in advance what any one of those domains' robots.txt files will say six months from now — see Real happy, partial and failure output for a live old.reddit.com block, quoted error and all.
• A discovered apex-vs-www quirk, handled automatically. While verifying this build, astralcodexten.com (no www) answered /api/v1/posts with a direct HTTP 404 — not a redirect — while www.astralcodexten.com answered HTTP 200 on the identical path, a behavior not documented by Substack. This Actor retries once, through the same robots.txt-guarded path, with a www. prefix, whenever a bare non-www., non-numeric host's first request fails — see Evidence and boundaries for the exact trigger condition.
• No login, no API key, no paywall bypass, by design and by code. This Actor sends no credentials of any kind. It reads exactly the same public, unauthenticated JSON response any anonymous visitor's browser would receive.
• A hardened SSRF and host-confusion guard, confirmed live against a real loopback address (127.0.0.1 — see the SSRF error example below) and re-applied on every redirect hop, not only on the input value.
• A billing guard that never double-charges under concurrency. The charge and the dataset write happen inside one mutex-protected step, reused unmodified from the same billing-guard pattern already running on the other Actors in this hub — see Pricing for why that mattered enough to be worth documenting explicitly.
• Runs on Apify: schedule it, call it from the API or an MCP client, export results to JSON, CSV or Excel, or push rows straight into your own pipeline.

Who uses it

• Content analysts and agencies monitoring a specific set of newsletters — competitive content research, publication-cadence tracking, or building a digest from a known list of Substack publications, rather than discovering new ones.
• Competitive intelligence teams who already know which publications matter to them and want the actual text of recent posts, not just a feed summary, to analyze themes, framing, or topic shifts over time.
• Newsletter curators and digest builders pulling the last few posts from several publications into one structured dataset for a roundup, without opening each site by hand.
• Developers and agent/workflow builders who want one flat JSON row per post — status, audience, bodyHtmlLength — that a script, an n8n node, or an LLM agent can branch on directly, without parsing Substack's HTML or guessing at an undocumented response shape themselves.
• Not a paywall bypass tool, under any framing — see Limits and Sources and rights. This is a boundary, not a missing feature this Actor intends to fill later.
• Not for comment mining. Substack's own robots.txt disallows /p/*/comment/*; this Actor never requests it. commentCount is metadata only — never a fetch of the thread.
• Not for a complete historical archive. This Actor reads the most recent limit posts per publication per run (up to 50), not the full back catalog, and does not paginate deeper into a publication's history. See Limits.

How to run

1. Populate Publications with one or more values — a bare domain (astralcodexten.com) or a full URL (https://www.astralcodexten.com) both work. Up to 20 publications per run; duplicates are silently deduplicated before any request is made.
2. Leave Posts per publication at its default of 10, or lower it toward the 3 used in this page's own evidence (1–50 range) — see Operating guide for what is and is not verified above 3.
3. Leave Max concurrency at its default of 5 unless you have a reason to raise it (up to 15) or lower it (down to 1).
4. Press Start. The prefilled input (["astralcodexten.com"]) is a real, live, free publication — a default run with no changes produces a small, real, non-empty dataset at close to zero cost, so you can see the actual shape of the data before pointing this Actor at a larger list.
5. Read the Dataset: one row per post for a publication that returned content, or one summary row for a publication that was blocked by robots.txt or was not a reachable Substack publication. Every row shares the same 14-field shape, and a pre-built "Post overview" table view is also available in the Console — see Field dictionary for both.
6. Check status before trusting bodyHtml. "ok" is the only status where bodyHtml is guaranteed non-empty. "partial" means the row was delivered but the body is empty by the source's own design — see Pricing for why.
7. A run-level summary is written to the OUTPUT record of this run's key-value store: how many publications were attempted, how many rows were delivered, how many of those were charged and how many were free, how many failed, how many came back incomplete, whether re-running the same input is safe, and whether the run ended in a fatal error. Read it to reconcile your Apify invoice against what you actually received. The Dataset itself stays the authoritative record of the content; the summary tells you what happened to the run as a whole.

Pricing

$0.005 per Actor start, plus $0.001 per delivered post. The primary pricing noun on this page, in the Dataset, and in this Actor's platform pricing record is the same throughout: the delivered post — one row, read from a publication you supplied, where the source actually handed over a non-empty post body (status: "ok"). 100 delivered posts cost about $0.105 in total — metered on Apify's invoice as apify-actor-start + result-found.

A paywalled post is never billed as delivered text. A row with status: "partial" — the source answered HTTP 200 with audience set to something other than "everyone" and body_html empty — is written to the Dataset for free, so you still see that the post exists, its title, its date, and its paywall status, without paying for text you were never sent. A publication blocked by its own robots.txt (status: "blocked_by_robots") and a publication that could not be read at all (status: "error") are also always free.

This is enforced in code, not left as a documentation promise. Charging and writing a row happen together, inside one mutex-protected step (lib/billing.js), and only a row already carrying status: "ok" at that point is ever passed to the paid path; every other status is pushed through a separate, unconditionally free path. lib/billing.js's own header comment documents why this matters here specifically: an earlier version of this pattern checked the remaining budget and then pushed the row as two separate steps, which is safe only when a single caller runs at a time — this Actor's default concurrency is 5 (up to 15), so two workers could otherwise both pass the budget check before either had actually charged, double-billing the buyer at the budget boundary. The guard used here is copied, not re-derived, from the same proven pattern already running on the other Actors in this factory's hub, specifically to avoid reintroducing a bug that had already been found and fixed once elsewhere.

This Actor's own external data cost is close to zero — one GET request per publication, roughly 68 KB per post on the publications measured for this page, no proxy, no LLM, no paid third-party API — so the per-post price reflects extraction, the paywall-honesty guarantee, the per-run robots guard and delivery verification described on this page, not a resold data bill.

Input contract

One field is required.

• publications — array of strings, up to 20 items, required. Each item is either a bare domain (astralcodexten.com) or a full URL (https://www.astralcodexten.com); this Actor normalizes both forms down to a lowercase hostname before making any request. A value is rejected before any network call (empty input handling, no request made) if it fails to parse as a URL once https:// is prepended, if it carries embedded userinfo (user@host — a classic host-confusion vector where the readable "host" is not where a request would actually go), or if the parsed hostname contains .. or a non-printable character. Duplicate hosts across the list are silently deduplicated.
• postsPerPublication — integer, 1 to 50, default 10. How many of the most recent posts to request per publication (limit=N in the source URL). This page's own live evidence was gathered at 3, matching this Actor's own acceptance-test goldens; see Operating guide for what is honestly unverified above that value.
• maxConcurrency — integer, 1 to 15, default 5. How many publications this run reads at once.

Two runnable input examples, matching the values used to produce the live JSON in the next section — these are input illustrations, not output; Real happy, partial and failure output below shows the actual Dataset rows these exact inputs produced on a real run of this build.

{
  "publications": ["astralcodexten.com"],
  "postsPerPublication": 3,
  "maxConcurrency": 1
}

(A free, live publication. This is the Actor's own prefilled domain, run at the same postsPerPublication value used throughout this page's evidence so the JSON below is exactly reproducible.)

{
  "publications": [
    "bigtechnology.substack.com",
    "old.reddit.com",
    "example.com"
  ],
  "postsPerPublication": 3,
  "maxConcurrency": 1
}

(Mixes three different outcomes in one batch — a paid publication, a robots.txt-blocked domain, and a non-Substack domain — to show a buyer doesn't need to pre-filter their list. Each produces a different, correctly labeled status, shown individually below.)

Real happy, partial and failure output

Every row in the Dataset shares one flat, 14-field shape regardless of outcome — input, publicationDomain, found, postId, title, postDate, audience, bodyHtmlLength, bodyHtml, commentCount, canonicalUrl, status, error, checkedAt. All five examples below are copied verbatim from real local runs of this exact main.js build (apify run --purge against APIFY_LOCAL_STORAGE_DIR), executed while this page was being written, against the live domains named in each block. Nothing here is invented or reconstructed from the schema — every field value below is what the Dataset actually contained after each run finished.

Happy path — a free publication, full text delivered

astralcodexten.com is a real, publicly accessible, free Substack publication. This exact row is the full, untruncated Dataset entry for its most recent post at the time of this run:

{
  "input": "astralcodexten.com",
  "publicationDomain": "www.astralcodexten.com",
  "found": true,
  "postId": 211610979,
  "title": "Open Thread 447",
  "postDate": "2026-08-17T19:52:17.873Z",
  "audience": "everyone",
  "bodyHtmlLength": 1310,
  "bodyHtml": "<p>This is the weekly visible open thread. Post about anything you want, ask random questions, whatever. ACX has an unofficial <a href=\"https://www.reddit.com/r/slatestarcodex/\">subreddit</a>, <a href=\"https://discord.gg/RTKtdut\">Discord</a>, and <a href=\"https://www.datasecretslox.com/index.php\">bulletin board</a>, and <a href=\"https://www.lesswrong.com/community?filters%5B0%5D=SSC\">in-person meetups around the world</a>. Most content is free, some is subscriber only; you can subscribe <strong><a href=\"https://astralcodexten.substack.com/subscribe?\">here</a></strong>. Also:</p><div><hr></div><p><strong>1: </strong>AI Security Bootcamp is running another (very timely) cohort, and has asked me to signal-boost them again.</p>",
  "commentCount": 9,
  "canonicalUrl": "https://www.astralcodexten.com/p/open-thread-447",
  "status": "ok",
  "error": null,
  "checkedAt": "2026-08-17T20:31:15.241Z"
}

Notice publicationDomain is www.astralcodexten.com, not the bare astralcodexten.com that was supplied as input — the apex-fallback quirk described in What you get. The same run also returned two larger posts from the same publication, confirming this is not a small-body coincidence: "Your Book Review: The Escape Artist" at bodyHtmlLength: 82935, and "The Foothills Of Bay Area House Party" at bodyHtmlLength: 21603 — both also audience: "everyone", status: "ok".

Partial path — a real paid publication, proven not to leak paid text

bigtechnology.substack.com is a real, live, paid Substack publication. This is the full, untruncated Dataset entry for its most recent post — note bodyHtml is a genuinely empty string, not a truncated one:

{
  "input": "bigtechnology.substack.com",
  "publicationDomain": "www.bigtechnology.com",
  "found": true,
  "postId": 209279848,
  "title": "When Artificial Intelligence Is Too Valuable To Sell",
  "postDate": "2026-07-31T19:30:17.712Z",
  "audience": "only_paid",
  "bodyHtmlLength": 0,
  "bodyHtml": "",
  "commentCount": 0,
  "canonicalUrl": "https://www.bigtechnology.com/p/when-artificial-intelligence-is-too",
  "status": "partial",
  "error": null,
  "checkedAt": "2026-08-17T20:31:30.461Z"
}

All three posts requested from this publication in the same run came back the same way: audience: "only_paid", bodyHtmlLength: 0, status: "partial" — "When Artificial Intelligence Is Too Valuable To Sell", "Everything You Need to Know About Kimi K3, the Latest Model From China to Shake Up AI", and "AI Model Prices Are Falling At The Worst Moment For The U.S. Frontier Labs". Note also publicationDomain: "www.bigtechnology.com" — this publication's own domain, reached via the HTTP 301 redirect this Actor followed from bigtechnology.substack.com, with robots.txt on the new host checked before the redirect was followed (see Evidence and boundaries). This run's log line read exactly: Done: 1 publication(s) with data, 0 failed of 1 (0 post rows billed).

Blocked path — a target domain whose own robots.txt disallows this path entirely

old.reddit.com is not a Substack publication, but it is a real, live domain whose own robots.txt reads User-Agent: * / Disallow: / — everything is disallowed. This Actor checked that file before attempting any content request and stopped:

{
  "input": "old.reddit.com",
  "publicationDomain": "old.reddit.com",
  "found": false,
  "postId": null,
  "title": null,
  "postDate": null,
  "audience": null,
  "bodyHtmlLength": null,
  "bodyHtml": null,
  "commentCount": null,
  "canonicalUrl": null,
  "status": "blocked_by_robots",
  "error": "robots.txt on old.reddit.com disallows /api/v1/posts for User-agent: *",
  "checkedAt": "2026-08-17T20:31:38.449Z"
}

No request was made to /api/v1/posts on this host at all — this run's log line confirms the block happened before the content fetch, not after an unauthorized read. This row is free.

Error path — a domain that is not a reachable Substack publication

example.com is a real, live, ordinary domain with no Substack publication behind it. Its own robots.txt does not block anything relevant, so this Actor proceeded to the content request, which the domain answered honestly with HTTP 404:

{
  "input": "example.com",
  "publicationDomain": "example.com",
  "found": false,
  "postId": null,
  "title": null,
  "postDate": null,
  "audience": null,
  "bodyHtmlLength": null,
  "bodyHtml": null,
  "commentCount": null,
  "canonicalUrl": null,
  "status": "error",
  "error": "http 404",
  "checkedAt": "2026-08-17T20:31:45.133Z"
}

This row is also free — status: "error" is never billed, same as "blocked_by_robots" and "partial". The distinction that matters to you as a buyer: "blocked_by_robots" means the target domain itself refused to be crawled and this Actor never attempted the content read; "error" means the content read was attempted and the source answered that nothing was there.

A second error example — the SSRF/host guard, not a content-level error

A distinct kind of "error" row, worth showing separately because it never reaches the network at all. 127.0.0.1 was supplied directly as a "publication domain" — a loopback address, the kind of value a redirect-based SSRF attack would try to smuggle in:

{
  "input": "127.0.0.1",
  "publicationDomain": "127.0.0.1",
  "found": false,
  "postId": null,
  "title": null,
  "postDate": null,
  "audience": null,
  "bodyHtmlLength": null,
  "bodyHtml": null,
  "commentCount": null,
  "canonicalUrl": null,
  "status": "error",
  "error": "blocked host (private/loopback/metadata)",
  "checkedAt": "2026-08-17T20:31:51.340Z"
}

This row is also free. The same guard code runs identically on every redirect hop this Actor follows, not only on the initial host — see Evidence and boundaries for why this page can show the direct loopback case live but not a combined redirect-to-private-IP case.

Field dictionary

Field	Meaning	Important boundary
input	Exactly the publication value you supplied, after normalization to a lowercase host	Echoed even when the request was rejected before any network call — lets you match every input back to its outcome
publicationDomain	The domain the post was actually read from, after following any redirect or apex-to-www fallback	Can differ from input — see the bigtechnology.substack.com → www.bigtechnology.com and astralcodexten.com → www.astralcodexten.com examples above
found	true once a post record was actually read from the source, false on every summary row	true even when status is "partial" — the post was found, its body just wasn't sent
postId	The post's numeric id from the source	null on every summary row (blocked_by_robots, error)
title	Post title as returned by the source	null on every summary row
postDate	Publish timestamp, ISO 8601, as returned by the source	null on every summary row
audience	The source's own access-level flag, passed through unchanged	"everyone" (free) or "only_paid" (paywalled) observed live; the source may use other values, and this Actor never interprets or renames any of them
bodyHtmlLength	Byte length of bodyHtml	Check this before trusting bodyHtml — it is 0 on every paywalled post observed
bodyHtml	Full post body HTML, verbatim from the source	Empty string, not null, on a paywalled post — this is deliberate: the post genuinely exists, its text just wasn't sent
commentCount	Comment count as reported by the source's own metadata	Not independently verified — this Actor never fetches the comment thread itself (disallowed path, see Sources and rights)
canonicalUrl	The post's public URL	null on every summary row
status	The billing and outcome boundary	"ok" = billed, delivered text. "partial" = free, delivered row, empty body. "blocked_by_robots" / "error" = free, summary row, no post data
error	null on every row where a post was actually read, including "partial"	Populated only on "blocked_by_robots" (quotes the disallowed path) and "error" (quotes the source's own HTTP status, or names the guard that rejected the request)
checkedAt	ISO-8601 timestamp of this row's outcome	Per-row, not per-run — rows from the same run can carry slightly different timestamps under concurrency

A pre-built "Post overview" Dataset view (.actor/dataset_schema.json) exposes eight of these fields — publicationDomain, title, postDate, audience, bodyHtmlLength, commentCount, status, found — as a sortable table in the Console, without needing to open raw JSON rows.

Evidence and boundaries

The paywall is the first thing to understand about this Actor, and the evidence for it is above, not asserted here. A paid publication's posts arrive at HTTP 200 with a real title, a real date, a real audience: "only_paid" flag — and a body_html field that is a genuine zero-length string. This was confirmed on all three posts requested from bigtechnology.substack.com in a single live run (see Real happy, partial and failure output). This Actor's code checks bodyHtml.length > 0 to decide status, not the HTTP status code and not the presence of the audience field — a paywalled post at HTTP 200 is never mistaken for a delivered one, and is never billed as one.

Custom domains are read like any other domain, with their own robots.txt check, because they genuinely can differ from Substack's own file. bigtechnology.substack.com redirected (HTTP 301) to www.bigtechnology.com on the same day this page was written; this Actor followed that redirect and checked robots.txt on www.bigtechnology.com specifically — the new host — before reading anything there, not the robots.txt of the domain the request started at. In this instance, a direct curl of both www.astralcodexten.com/robots.txt and www.bigtechnology.com/robots.txt showed identical content to substack.com/robots.txt itself — all three are served by Substack's shared platform infrastructure even on a custom domain, and none of the three disallow /api/v1/posts. That match is not guaranteed to hold for every publication forever; a publisher who reconfigures their own DNS or reverse-proxy setup could serve a different robots.txt on their custom domain at any time, which is exactly why this Actor re-reads the file on the domain it is actually about to request, every run, rather than trusting a build-time assumption about Substack's platform-wide file.

The apex-vs-www retry is a discovered production quirk, not a documented Substack behavior. While verifying this build, a direct request to https://astralcodexten.com/api/v1/posts?limit=3 (no www) returned a genuine HTTP 404 — not a redirect. The identical path on https://www.astralcodexten.com/api/v1/posts?limit=3 returned HTTP 200. This Actor's code retries exactly once, through the same robots.txt-guarded fetch path, with a www. prefix, whenever a bare non-www., non-numeric host's first request fails — a different host still means a different robots.txt check, not a bypass of the first one (see the astralcodexten.com happy-path example above for the resulting publicationDomain). This retry is a resilience measure found empirically, not asserted to cover every Substack publication's DNS configuration — only the one observed live.

The robots.txt check runs live, on the target host, on every publication in every run — and is cached only for the lifetime of a single run. Within one run, if a run's input lists the same host twice (after normalization) or a redirect returns to a previously-seen host, the second check reuses the first fetch rather than re-requesting the same robots.txt file twice. Across separate runs there is no persistence at all — a publisher who tightens their robots.txt after one run is picked up correctly on the very next run. What this does not cover: a robots.txt file changing mid-run, between the first and a later request to the same host in one run, is not detected — the ruling fetched at the start of that run's first request to that host is the ruling used for the rest of that run. This is a genuine, disclosed edge case, not something this evidence session triggered live.

The SSRF/host guard, confirmed live against a real loopback address, applied identically on every redirect hop. 127.0.0.1 was rejected with error: "blocked host (private/loopback/metadata)" before any request left the process (see the second error example above). The same resolveVerifiedAddresses() check that stopped that direct case runs again on every redirect Location header this Actor follows — including the real bigtechnology.substack.com → www.bigtechnology.com redirect above — so a publication that tried to redirect into a private address range would be stopped at that hop the same way. No live third party publishes a redirect-to-private-IP endpoint to test that exact case end-to-end, so this page can confirm the guard fires on a direct loopback input and can confirm the identical code path runs on every hop of a real redirect — it cannot show a live example of both combined, because no safe, public target for that combination exists to test against.

Response size is capped; what happens to a request that hits the cap was not exercised live for this page. Every fetch this Actor makes is capped at 4,000,000 bytes and stops reading past that point. The publications measured for this page returned roughly 68–204 KB total for 3 posts — nowhere near the cap. If a very large postsPerPublication value against publications with unusually long posts produced a response that got cut mid-stream, the truncated text would very likely fail JSON parsing, and the code path for that failure returns status: "error" with a "bad json: ..." message rather than returning corrupted partial content as if it were valid — this is what the code does on a JSON.parse failure regardless of cause, but the specific truncation-at-cap scenario was not independently reproduced during this evidence session. Treat it as a disclosed, reasoned expectation from reading the code, not a live-verified fact.

postsPerPublication and maxConcurrency above the values used in this page's evidence are not independently load-tested here. Every JSON example was captured at postsPerPublication: 3, maxConcurrency: 1, matching this Actor's own acceptance-test goldens. The schema allows up to 50 posts and 15 concurrent publications — real, enforced limits neither ceiling was pushed to during this session. See Operating guide and Limits.

This Actor's billing safety nets were not exercised on-platform for this page. Every JSON example above came from a local run (apify run --purge), which never engages the Apify platform's own charging manager. Locally, monetized is false and every row is written through the same unconditional path regardless of status; the log line (0 post rows billed) on the bigtechnology.substack.com run above reflects that no status: "ok" row was produced, not a local billing simulation. See Limits for what remains unverified about the mutex-protected guard itself.

Decision routing

status	Billable?	bodyHtml	What it means
"ok"	Yes	Non-empty	The source handed over the post's real text
"partial"	No	Empty string	The post exists but the source did not send its body — observed exclusively on paywalled posts (audience not "everyone") in this Actor's evidence
"blocked_by_robots"	No	null (no post read)	The target domain's own robots.txt disallows this path; no content request was made
"error"	No	null (no post read)	The domain is not a reachable Substack publication, the request was rejected by the SSRF/host guard, or the source returned a non-200 response

A workflow needs two rules to be correct: bill and use only rows where status is "ok"; and never assume "partial" is an error just because bodyHtml is empty — check audience on a "partial" row to see whether the reason was a paywall, the only reason observed in this Actor's own evidence.

Commercial playbooks

Competitive content-cadence and theme tracking

An analyst or agency who already knows which Substack publications matter to a client's market runs this Actor on a schedule against that fixed list, reading full post text to track publishing frequency, topic shifts, and framing over time — instead of a title-only RSS feed or manually opening each site.

Paywall-posture monitoring across a watchlist

A researcher tracking how aggressively a set of publications monetizes their content points this Actor at the same publication list on a recurring schedule and reads the audience field on each run. This Actor reports a single run's snapshot honestly — whether a specific post is free or paid right now — and does not itself track change over time; a buyer who wants a trend line stores each run's audience values in their own system and compares them across runs.

Digest and roundup content pipelines

A newsletter curator or content team feeds in the domains of several publications they already follow and pulls the last few posts from each into one structured dataset, ready to filter to status: "ok" and assemble into a roundup — without opening each site by hand or writing a bespoke parser for each publication's own front end.

Integration recipes

Scheduled watchlist

Configure an Apify Task with a fixed publications list and a daily or weekly schedule. Filter each run's Dataset to status: "ok" before loading it into a digest, spreadsheet, or CRM — "partial", "blocked_by_robots" and "error" rows are the free, honest record of what did not deliver text, not data to import as content.

Agent/MCP pattern

An agent that already knows a target publication's domain passes it directly into publications, checks status before treating any row as usable content, and reads audience on a "partial" row to explain to the end user why a specific post came back without text, rather than treating the row as a silent failure to retry.

Spreadsheet/catalog append pattern

Dataset rows are appended keyed by postId for "ok"/"partial" rows (both represent a genuinely found post) and by input + checkedAt for "blocked_by_robots"/"error" summary rows, which carry no postId. null fields are preserved as null, not coerced into an empty string, so a downstream query can tell "this post's body is empty because it's paywalled" apart from "this field was never populated for this row type."

Operating guide

Choosing postsPerPublication. The default of 10 is the schema default, not the value this page's own live evidence was gathered at — every JSON example above used 3, matching this Actor's own acceptance-test goldens. Raising it fetches more posts in the same single request per publication (the source's own limit=N parameter); it does not change the per-post price or add a second request. Values well above 3 have not been independently measured for response size or truncation risk during this evidence session — see Evidence and boundaries on the 4 MB response cap.

Choosing maxConcurrency. The default of 5 is a conservative starting point for reading several publications in parallel; raising it (up to 15) speeds up a run with many publications in one input without changing what any individual publication's own request returns. This page's own evidence was gathered at maxConcurrency: 1 to keep each example's log output attributable to one publication at a time.

Reading a "partial" row correctly. Check audience — every "partial" row observed while building this page was a paywall case (audience not "everyone"), though the field technically just means "delivered but the body was empty." This Actor never bills that row regardless of the specific reason.

Reading a "blocked_by_robots" row correctly. This means the target domain's own robots.txt disallowed the request path at the moment this run made the check — not that the publication doesn't exist, and not a permanent verdict. Re-running against the same domain later can produce a different result if that publisher changes their file; this Actor checks fresh on every run rather than caching a verdict across runs (see Evidence and boundaries for the one within-run caching caveat).

Handling redirects and the apex/www quirk. You do not need to pre-resolve which exact host a publication's API actually lives on — supply the domain you know, and check publicationDomain in the result to see where the data actually came from. Both the redirect-follow path (proven on bigtechnology.substack.com → www.bigtechnology.com) and the failure-triggered www. retry (proven on astralcodexten.com → www.astralcodexten.com) run automatically and are both re-checked against robots.txt on the resulting host before any content is read.

Related tools

Related tools for adjacent workflows in social platforms and regional directories.

Actor	What it does
YouTube Playlist Scraper	Pair it in the social platforms and regional directories workflow: Scrape any YouTube playlist by id or URL — including other people's public playlists, not just your own...
Bluesky Profile Scraper	Pair it in the social platforms and regional directories workflow: Batch-fetch public Bluesky profiles by handle or DID via the official keyless AppView API — display name,...
YouTube Channel Intel	Pair it in the social platforms and regional directories workflow: Public intel card for any YouTube channel: subscribers, total views, video count, join date, country, full...
YouTube Channel Videos List	Pair it in the social platforms and regional directories workflow: List up to 100 most recent uploads for any YouTube channel: videoId, title, view count, and publish age...
YouTube Channel Lookalike Finder	Pair it in the social platforms and regional directories workflow: Find the similar/lookalike channels YouTube itself shows on any channel's About page: channelId, name,...

FAQ

Does this bypass paywalls?

No, never, under any input. A paywalled post's body is empty because Substack's own API does not send it to an unauthenticated client — this Actor reports that honestly (status: "partial", bodyHtmlLength: 0) instead of pretending it has the text, and it sends no login, session, or subscription credential of any kind that could change that answer.

Does it work on a publication's own custom domain, or only *.substack.com addresses?

Both. bigtechnology.substack.com in this page's own evidence resolved, via a real HTTP 301 redirect, to www.bigtechnology.com — the publication's custom domain — and this Actor followed it, checking robots.txt on the destination before reading anything there.

What happens if a publication's domain blocks crawling?

Its posts come back as one free row with status: "blocked_by_robots", quoting the exact disallowed path from that domain's own robots.txt, and no request is ever made to the content path. This is checked fresh on every run.

What happens if I give it a domain that isn't a Substack publication?

A free row with status: "error" and the source's own response quoted or summarized in error — an HTTP 404 from the domain in this page's own example.com example, or a similar honest failure for another non-Substack domain.

Does it return comments?

No. Substack's own robots.txt disallows /p/*/comment/*, and this Actor's code never requests that path under any input. commentCount is a number reported in the post's own metadata, not a fetch of the thread itself.

Can it read a subscriber's private, logged-in feed?

No. This Actor sends no authentication of any kind and only ever reads what an anonymous visitor's request to the public JSON endpoint would receive — the same boundary that produces the paywall behavior above also means a reader's personal, logged-in feed is not something this Actor can access.

Can I get a publication's entire archive?

Not in one run. This Actor requests the most recent limit posts (postsPerPublication, up to 50) per publication per run and does not paginate deeper into a publication's history. See Limits.

Is /api/v1/posts an official, documented Substack API?

No — it is a real, currently reachable JSON endpoint that Substack's own publication front ends call to render posts, but it is not published as a stable, versioned public API with a support commitment from Substack. See Sources and rights for what that means as a risk, stated plainly rather than buried.

What happens under high concurrency — can two workers double-charge the same row?

No. The charge and the dataset write for a status: "ok" row happen together inside one mutex-protected step in lib/billing.js, specifically to close a race where two concurrent workers could otherwise both pass a budget check before either had charged. See Pricing for why this Actor's default concurrency of 5 made that race worth fixing, not theoretical.

Found a wrong result, or need a check this Actor doesn't run? Open an issue on this Actor's page.

Sources and rights

The one path this Actor requests is public and allowed by robots.txt, re-checked on the publication's own domain every run.

The only source. The JSON response at https://<publication-domain>/api/v1/posts?limit=N — the same endpoint the publication's own site calls to render its posts, fetched anonymously, with no login and no API key. No RSS feed, no email-digest scraping, no browser rendering.

This is an official-but-undocumented endpoint, and that is a genuine, disclosed risk, not a footnote. Substack has not published /api/v1/posts as a stable, versioned, supported public API. It is real and currently reachable — every JSON example on this page was read from it live — but the platform could change its shape, rename it, or restrict it without prior notice, the way any internal endpoint a platform did not commit to as a public contract can change. A buyer relying on this Actor for ongoing monitoring should know that risk exists before relying on it, not discover it the day a scheduled run starts failing.

robots.txt, checked live on the actual target domain, not cached from a prior date. A direct request to substack.com/robots.txt, made independently while writing this page, disallows /action/, /publish, /sign-in, /channel-frame, /session-attribution-frame, /visited-surface-frame, /feed/private, /feed/podcast/*/private/*.rss, /subscribe, /lovestack/*, /p/*/comment/*, /inbox/post/*, /notes/post/*, and /embed for User-agent: *. /api/v1/posts is not named by any of those rules. The same file, byte-for-byte identical, was independently confirmed live on both www.astralcodexten.com/robots.txt and www.bigtechnology.com/robots.txt — both custom domains, both served by Substack's own shared platform infrastructure, both permitting this path. This Actor re-checks the file on whichever domain it is actually about to request — not substack.com's file by assumption — specifically because a publisher's own custom-domain configuration is not guaranteed to match Substack's platform default forever.

No explicit reuse licence. Substack publishes no equivalent of an open-data licence for the content of its posts. This Actor's engineering posture is limited to what robots.txt permits to be crawled and what a genuinely anonymous, unauthenticated request receives; it does not constitute legal advice about how you may use the resulting post text downstream, and each post's own text remains the copyrighted work of its author regardless of how this Actor reads it.

No paywall bypass, as a rule this Actor's code follows, not a claim made only on this page. Nothing in this Actor's logic attempts a login, a subscription purchase, or any other method of reaching content behind a paywall — see Evidence and boundaries for the exact check the code runs. A paywalled post is reported honestly and left there.

Limits

A paid publication's post body is empty, at HTTP 200, and this is never sold as delivered text. This is the single most important thing to understand about this Actor before using it: audience not equal to "everyone" means bodyHtmlLength: 0, status: "partial", and no charge — confirmed live on three real posts from a real paid publication in Real happy, partial and failure output above. There is no configuration, input flag, or future update path in this Actor that changes this; a paywalled post's text is simply not something the source sends to an unauthenticated request, and this Actor does not attempt to obtain it any other way.

• No comments. Substack's own robots.txt disallows the comment-thread path; this Actor's code never requests it, on any input. commentCount is a number from the post's own metadata only.
• No private, logged-in reader feed. This Actor sends no authentication of any kind.
• No full archive. Up to 50 most recent posts per publication per run, no deeper pagination — a publication with hundreds of posts is not fully retrievable in one run, and this Actor does not attempt to cross multiple runs into one combined archive itself.
• The source endpoint is officially undocumented. /api/v1/posts works today, verified live on this page's own evidence, but Substack has made no public commitment to keep its shape or availability stable. See Sources and rights.
• robots.txt is cached only for the lifetime of one run. A change to a target domain's file mid-run, between two requests to the same host within that run, would not be detected until the next run. Across separate runs, the check is always fresh.
• The 4 MB per-request response cap has not been tested at its actual truncation point. The publications measured for this page returned 68–204 KB total for 3 posts, nowhere near the cap; the behavior described for a genuinely truncated response (a JSON-parse failure surfacing as status: "error") is reasoned from the code, not independently reproduced live.
• postsPerPublication above 3 and maxConcurrency above 1 were not the values used to gather this page's own live JSON evidence, even though both are real, enforced limits (50 and 15 respectively) in the input schema and in main.js. See Evidence and boundaries and Operating guide.
• The apex-to-www retry and the redirect-follow path were each confirmed on exactly one real publication. Both mechanisms are general code paths, not special-cased to astralcodexten.com or bigtechnology.substack.com specifically, but this page's live evidence covers one concrete example of each, not an exhaustive survey of every Substack publication's DNS configuration.
• This Actor's platform billing safety nets — the double-charge-proof guard, the pricing-misconfiguration halt, the mid-run budget stop — are real, reviewed code paths, but none of them were exercised during this evidence-gathering session, because a local, off-platform run never engages the platform's own charging manager. They are documented from the code and from the shared billing-guard pattern this Actor reuses from elsewhere in this hub, not from a monetized live observation made specifically for this page.
• maxConcurrency up to 15 has not been load-tested against a large batch of publications for this build — the default of 5 is the conservative, documented starting point.

Support boundary

Actor support covers this Actor's own input validation, the paywall-honesty behavior, the per-run robots.txt guard on the actual target domain, the SSRF/host-confusion guard, the apex-to-www retry, the redirect-follow behavior, and the Dataset schema described on this page — all backed by a live, verified run referenced above. It does not cover whether Substack's undocumented /api/v1/posts endpoint remains reachable or unchanged in its response shape going forward, does not restore access if a specific publication's markup, API, or anti-automation posture changes, does not provide a way to read paywalled content under any circumstance, and does not provide legal guidance on how the resulting post text may be reused downstream. When reporting a problem, include the run id, the exact publication value you supplied, the status and error returned, and the checkedAt timestamp of the affected row.

---

Built by zinin. Questions? Telegram @timzinin.