Substack Publication Scraper

Point it at a Substack publication and get its posts back as a table, one row each: the title, the link, the date it went out, who wrote it, how long it is, how many comments and reactions it drew, and whether it is behind the paywall. No account, no cookies, no login, no browser.

• Reads the whole archive, not just the front page. A publication with 1,474 posts going back to 2020 came back complete in 31 requests, with no repeats.
• Every row says whether the post was free or subscriber-only, so you can split the two without opening anything.
• Publications on their own domain work the same way as substack.com ones - give it the domain.
• Several publications in one run, each row tagged with the one it came from.
• Runs with empty input return a labelled sample row, free, so you can see the shape before you spend anything.

Price

$0.26 per 1,000 posts, plus a $0.0005 start fee per run.

This is a flat rate on every plan, free or paid. There are no volume tiers, no minimum spend, no subscription and no add-on fees. What you read here is what you pay on day one and on day four hundred.

Posts	Total cost
100	$0.0265
1,000	$0.2605
10,000	$2.6005
100,000	$26.0005

What is actually charged

• One post-scraped event per post row written to the dataset. Nothing else is metered per row.
• Free: the sample row an empty run returns, and every diagnostic row — a blocked target, a dead URL, a publication that does not exist. Those rows all carry "charged": false.
• A publication that does not exist, has moved, or returns nothing costs you nothing - you get an uncharged row saying which.
• Duplicate posts are dropped before they are charged, so a long archive that shifts mid-run cannot bill you twice for the same post.
• A run that finds nothing costs the start fee and nothing more.
• Rows never leave the dataset without a charge, and are never charged without a row. The billed event is a named one, so there is no price quietly attached to apify-default-dataset-item — the trick that makes some scrapers bill you for their own error messages.

Input

{
  "publications": [
    "astralcodexten",
    "www.bigtechnology.com"
  ],
  "maxPostsPerPublication": 50,
  "sort": "new"
}

Field	What it does
publications	One publication per line, up to 20. A bare handle works (astralcodexten), so does astralcodexten.substack.com, and so does a publication's own domain (www.bigtechnology.com). A link to a single post works too - the publication is taken from it.
maxPostsPerPublication	How many posts to take from each publication. Default 50, hard ceiling 2,000. A publication with fewer posts returns fewer rows. Keep it low while testing - you pay per post.
sort	new walks the archive newest-first, which is what you want for "the last N posts". top gives the publication's most popular instead, and community puts discussion threads first. Only matters when you ask for fewer posts than the archive holds.
proxyUrls	Leave empty. Fill it in only if you want the traffic to leave through proxy servers you already pay for, as http://user:pass@host:port.

Run it with empty input and you get one clearly labelled sample row, free, so you can see the output shape before you spend anything.

Output

One row per post. A real row from a real run:

{
  "ok": true,
  "charged": true,
  "recordType": "post",
  "publication": "noahpinion.substack.com",
  "title": "Roundup #88: Is it time to panic yet?",
  "url": "https://www.noahpinion.blog/p/roundup-88-is-it-time-to-panic-yet",
  "postDate": "2026-09-18T09:04:11.386Z",
  "audience": "everyone",
  "isPaywalled": false,
  "subtitle": "AI risk; AI and growth; AI and Saas; Rich people; Christians vs. Nazis; Japanese culture; State capacity; City size",
  "description": "AI risk; AI and growth; AI and Saas; Rich people; Christians vs. Nazis; Japanese culture; State capacity; City size",
  "previewText": "This week’s roundup has a lot of AI in it. Fortunately or unfortunately, it seems like a lot of the news is going to revolve around AI for the rest of our lives.",
  "wordCount": 3591,
  "commentCount": 47,
  "reactionCount": 274,
  "restackCount": 21,
  "authors": "Noah Smith",
  "postType": "newsletter",
  "slug": "roundup-88-is-it-time-to-panic-yet",
  "postId": "216235258",
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/1eb36da1-4628-4b66-b553-a7b97a2d9af2_992x708.jpeg",
  "language": "en",
  "scrapedAt": "2026-09-20T08:36:02.020Z"
}

Field notes

• publication — the host the row came from, exactly as the run resolved it. Handy when you scrape several at once.
• url — the post's canonical link. For a publication on its own domain this is that domain, not the substack.com one.
• postDate — when the post went out, normalised to ISO 8601 in UTC.
• audience — everyone or only_paid, straight from the publication.
• isPaywalled — audience as a boolean, because that is the question most people are actually asking.
• previewText — the short opening extract Substack itself publishes. Whether it is there depends on the publication rather than the post: some return it on every post, some on none at all, and it is null when absent. It is not the article - see Limitations.
• wordCount — the publication's own count for the full post, including the part you cannot read if it is paywalled.
• commentCount — top-level comments at the moment the row was read. It keeps moving on a live post.
• reactionCount — total reactions at read time.
• restackCount — how many times it was restacked on Substack Notes.
• authors — the published bylines, comma-separated. A guest post carries the guest's name.
• postType — what kind of item it is - newsletter for an ordinary post, restack for a shared note.
• postId — Substack's own id. Stable, and safe to use as a primary key when you re-run.

Every real row carries "charged": true. Sample rows carry "_sample": true and diagnostic rows carry "_diagnostic": true with an errorCode you can filter on, and neither is ever billed.

How it works

• It calls the publication's own public archive listing and reads the post records it returns. No page rendering, no headless browser, no login, and nothing is signed in.
• The archive is paged 50 at a time until your row limit is reached or the publication runs out of posts. Each page advances by the number of records it actually returned, so nothing is skipped and nothing is read twice.
• Requests leave through a large pool of rotating addresses. The archive throttles a single address that asks too quickly, so the run moves to another one rather than sitting and waiting.
• Before a single row is charged, the response has to be a list of post records with ids and dates on them. An address that answers with a web page instead, or with an empty list, produces an uncharged row explaining what happened.

What people use it for

• Pulling a publication's whole back catalogue into a spreadsheet so you can sort it by reactions or comments and see what actually landed.
• Watching a handful of publications on a schedule and diffing on postId to catch only what is new since yesterday.
• Working out how often a writer posts, how long their posts run, and what share sits behind the paywall - the three numbers are all in the row already.
• Building a reading list or a newsletter round-up from several publications at once without opening twenty tabs.
• Research datasets. The rows are flat and typed, so they load into a database without cleaning.

What you get, and what you do not

This returns what a Substack archive listing publishes about each post. That is a lot - title, subtitle, link, date, author, word count, comments, reactions, restacks, cover image, and the free/paid flag.

It does not return the text of the posts. Not for paid posts, and not for free ones either. The archive listing carries no article bodies for anybody, so a paid subscription would not get you any more of it than this does. What you do get is previewText, the short opening extract Substack publishes itself - though whether a publication puts one out at all is the publication's own choice, so on some you get it on every post and on others you get it on none.

So if what you need is the articles, this is the wrong purchase. If what you need is a table of everything a publication has ever sent, with the paid ones marked, this is exactly it.

Reading the output

Every run writes three kinds of row, and they are easy to tell apart:

• Real rows carry "charged": true and "recordType": "post". One billed event each.
• The sample row carries "_sample": true and "charged": false. There is exactly one, it only appears when you gave no publications, and it exists so you can see the output shape before spending anything.
• Diagnostic rows carry "_diagnostic": true, "charged": false and an errorCode you can switch on: NOT_FOUND when no publication answers at that address, NO_RESULTS when the address answered but not with an archive, RATE_LIMITED when the target throttled the run, NETWORK when it could not be reached, TIME_BUDGET when the run ran out of time before reaching a publication. Each carries a plain-English error and the publication it belongs to.

If you only want the data, filter on charged == true. The count of those rows always equals the number of events you were billed for, so the dataset is its own invoice.

Limitations

• No article text, for free or paid posts alike. The archive listing carries none.
• previewText is Substack's own short extract, and whether you get it is decided by the publication, not by you. Some publications return it on every post; others return it on none and the field is null throughout. One 1,474-post archive tested came back with no preview text at all.
• A publication that has moved to its own domain returns nothing on its old substack.com address - Substack answers that with an empty list rather than an error. You get an uncharged row saying so; pass the publication's own domain instead.
• Comment counts, reaction counts and restacks are a snapshot at read time and will differ from what you see if you open the post a minute later.
• No comments, no subscriber numbers, no revenue, no email open rates. None of that is public.
• No podcast audio fields. Across every publication tested the archive returned those empty, so they are not emitted rather than shipped as columns that are always null.
• Posts a publication has hidden or geo-blocked are returned as the archive lists them; the run does not second-guess that.
• The hard ceiling is 2,000 posts per publication and 20 publications per run. For more, split the work across runs.
• sort only changes which posts you get when you ask for fewer than the publication has. Ask for the whole archive and you get the whole archive either way.

Questions

Can I get the full text of the posts?

No. The archive listing this reads does not carry article bodies for anyone, free or paid, so there is nothing a subscription would add. You may get previewText, the short extract Substack publishes itself, but that is up to the publication and plenty of them publish none. If you need the articles themselves, this is not the right tool.

How do I scrape a publication that uses its own domain?

Give it that domain - www.bigtechnology.com rather than bigtechnology. Publications on their own domain answer there and only there, and quite a few of the larger ones have moved.

What happens if I misspell a publication?

You get one uncharged diagnostic row with errorCode: "NOT_FOUND", and the run carries on to your other publications. You are never billed for a publication that returned nothing.

Can it read the entire archive, or just recent posts?

The entire archive. It pages until the publication runs out of posts. A 1,474-post archive going back to 2020 came back complete, in 31 requests, with no duplicates. Set maxPostsPerPublication to 2,000 if you want everything; leave it at 50 for the recent ones.

How do I tell the paid posts from the free ones?

Filter on isPaywalled. It is set on every row, taken straight from the publication's own audience setting.

Will the run fail if a publication is unreachable?

No. It writes an uncharged diagnostic row explaining what happened and carries on. A failed run would still bill you the start fee, which would mean paying to be told something went wrong.

Can I run this on a schedule?

Yes. Nothing is held between runs, so the same input is safe to repeat. Use postId to work out which posts are new since last time.