Substack Scraper — Posts, Comments & Publications

Any Substack, as a flat table. Give this Actor a handle (astralcodexten), a publication address (https://www.lennysnewsletter.com) or a single post URL — mixed together, in any order — and get one row per post: the title, the full text, the word count, the publish date, the author, the paywall flag, likes, restacks, comment count, tags and the cover image. Switch comments on and every comment becomes a row too, with parentId and depth so the thread rebuilds exactly. Every publication also gets a row of its own: author, bio, tagline, logo, subscription prices, launch date and a subscriber hint. Add a keyword and you get every publication Substack's own directory can find for it.

No API key. No login. No subscription of yours or anyone else's. $0.25 per 1,000 posts, full text and proxy included, publications that could not be read are free, and a run that returns nothing costs nothing at all.

Try it in 30 seconds

{
  "publications": ["astralcodexten", "https://www.lennysnewsletter.com"],
  "maxPostsPerPublication": 20,
  "includeBody": true,
  "includePublicationInfo": true
}

Two publications, the twenty most recent posts of each, full text included. A bare handle, a *.substack.com address and a custom domain all work — and a handle whose publication has moved to its own domain is followed automatically.

Want one post? {"publications": ["https://www.astralcodexten.com/p/royce-on-san-francisco"]}. Want yesterday only? {"since": "2026-09-10"} — the walk stops as soon as the archive passes the cutoff, so a daily run is one or two requests per publication. Want a list of publications? {"searchQueries": ["machine learning"], "maxResultsPerQuery": 100}.

Output — one row per post, plus one row per comment and one per publication; the fields you will use most are title, url, publishedAt, text, likes and commentCount (full list under Output reference). Anything that could not be read comes back as a free diagnostic row (ok: false, errorType, error) instead of a charge.

Price — $0.25 per 1,000 posts on the FREE tier with the full text included, $0.20 per 1,000 comments, $2 per 1,000 publication rows, plus $0.001 per run; failed rows, the listing requests themselves and a run that returns nothing are all free. Pay-per-event, no API key, no browser, limited permissions — it works over the Apify MCP server (mcp.apify.com) and with agentic (x402) payments.

From code — client.actor("insight.solutions/substack-scraper").call(run_input={…}) with apify-client, or

POST https://api.apify.com/v2/acts/insight.solutions~substack-scraper/run-sync-get-dataset-items

What comes back

One post row per post (nulls omitted here for readability):

{
  "ok": true,
  "rowType": "post",
  "input": "astralcodexten",
  "publication": "www.astralcodexten.com",
  "publicationName": "Astral Codex Ten",
  "publicationId": "89120",
  "postId": "212242052",
  "slug": "royce-on-san-francisco",
  "url": "https://www.astralcodexten.com/p/royce-on-san-francisco",
  "title": "Royce On San Francisco",
  "type": "newsletter",
  "audience": "everyone",
  "paywalled": false,
  "publishedAt": "2026-09-10T11:10:18.859Z",
  "updatedAt": "2026-09-10T11:14:07.394Z",
  "authors": [{ "id": "12009663", "name": "Scott Alexander", "handle": "astralcodexten" }],
  "authorNames": "Scott Alexander",
  "wordCount": 2393,
  "text": "On housing prices:\nThe San Franciscans at this moment, living in their rag palaces…",
  "bodyComplete": true,
  "likes": 155,
  "reactions": { "❤": 155 },
  "commentCount": 76,
  "restacks": 2,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/582979e2…jpeg",
  "language": "en",
  "position": 1,
  "scrapedAt": "2026-09-11T21:12:44.000Z",
  "source": "substack",
  "sourceUrl": "https://www.astralcodexten.com/api/v1/posts/royce-on-san-francisco"
}

One comment row per comment, when you ask for them:

{
  "rowType": "comment",
  "publication": "www.astralcodexten.com",
  "postId": "212242052",
  "postUrl": "https://www.astralcodexten.com/p/royce-on-san-francisco",
  "commentId": "334272881",
  "parentId": "334012158",
  "depth": 1,
  "authorName": "Judith Stove",
  "authorHandle": "judithstove",
  "date": "2026-09-10T21:19:05.486Z",
  "body": "Yep, I was thinking 'is this AI doing nineteenth-century pastiche?'…",
  "likes": 3,
  "childrenCount": 2,
  "edited": true,
  "deleted": false
}

And one publication row per publication:

{
  "rowType": "publication",
  "publication": "www.astralcodexten.com",
  "publicationName": "Astral Codex Ten",
  "publicationId": "89120",
  "url": "https://www.astralcodexten.com",
  "subdomain": "astralcodexten",
  "customDomain": "www.astralcodexten.com",
  "authorName": "Scott Alexander",
  "heroText": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
  "createdAt": "2020-08-30T04:18:18.308Z",
  "firstPostDate": "2020-08-30T21:01:57.050Z",
  "paymentsEnabled": true,
  "plans": [
    { "interval": "month", "amount": 1000, "currency": "usd", "nickname": "$10 a month" },
    { "interval": "year", "amount": 10000, "currency": "usd", "nickname": "$100 a year" }
  ],
  "monthlyPriceUsd": 10,
  "annualPriceUsd": 100,
  "hasPodcast": false,
  "postCount": 73
}

Every row — post, comment, publication or diagnostic — carries the same columns, null where they do not apply, so the dataset exports as one rectangular table however you mixed the inputs. The dataset has four ready-made views: Posts, Comments, Publications and Problems.

Use cases

• Read a competitor's whole archive as text — headline, subtitle, word count and body for every post they have published, in one table you can search.
• Track a topic across hundreds of newsletters. Search the directory for "machine learning", take the publications, then run their archives with since on a daily schedule and only ever fetch what is new.
• Feed a RAG index or a summariser. text is already plain text with paragraph breaks, not HTML you have to clean, and bodyComplete tells you whether you are indexing a whole post or a paywalled preview.
• Study an audience, not just a writer. Comment rows carry author, timestamp, likes and reply depth — a thread rebuilt exactly, ready for sentiment or network analysis.
• Build a market map. Publication rows carry prices, launch date, subscriber hints and the publication's own ranking blurb, which is what a "who is doing this well and what do they charge" spreadsheet is made of.
• Watch one post. Paste its URL and re-run it: likes, restacks and comment count are on the row every time.

How it works, and why it keeps working

Every Substack publication serves the same small JSON API at its own address — name.substack.com or the custom domain, the same endpoints either way. This Actor reads four of them, plus the publication's home page. All are the ones Substack's own website calls to render itself; none needs a key, a cookie or a session.

• The archive is paged by what came back, not by what was asked for. The first page of a real publication answered limit=50 with 23 posts, and the next one with 50. Walking by "offset += the page's length, stop on an empty array" is the only rule that survives that; stopping at the first short page would read a fifth of an archive and call it the archive.
• The list is cheap and the text is not. One request buys up to fifty posts' metadata; the text is one request per post. includeBody: false is therefore about fifty times fewer requests, which is the right setting for building an index and the wrong one for reading the writing.
• A publication that moved is followed. A *.substack.com address that now redirects to a custom domain is followed once, and every later request in that job goes straight to where it landed.
• A refusal is not fatal. A blocked request rotates the proxy session once and tries again from a different exit IP. Only a second refusal ends that job, with everything already delivered kept and one free diagnostic row saying where it stopped.
• Requests to one publication are spaced 250–600 ms apart. Nothing forces this; it is the difference between reading a public API and hammering somebody's newsletter.
• Partial results are kept. Hit maxRunSecs or your charge ceiling halfway through an archive and you keep — and are billed for — exactly the rows that were written, plus a free row saying what was missed.

How it compares

• The full text is the default, not an add-on. text is the post as a reader sees it, converted from the published HTML with paragraph and list breaks intact, entities decoded and image furniture dropped.
• Paywalls are labelled, not pretended about. A paid post returns the free preview Substack serves to the public; the row says paywalled: true and bodyComplete: false so you always know which one you have. See Limits.
• One schema for four things. A post, a comment, a publication profile and a search hit come back as the same columns, so mixing them is a filter rather than a reconciliation project.
• Failures are free and legible. A publication that does not exist, a post that was unpublished, an empty search or a block produce a diagnostic row with an errorType you can branch on — and no charge. A run that returns nothing at all finishes SUCCEEDED with zero results, a status message that says so, and a free diagnostic row per entry telling you why; it finishes FAILED only when no entry was usable, or the Actor itself hit an error.
• No login, ever. No account, no cookie jar, no session token, no paid subscription of yours borrowed for the run. Public endpoints only, which is also why it runs under Apify's limited permissions.

Input reference

Output reference

Shared by every row: ok, rowType, input, publication, publicationName, publicationId, error, errorType, scrapedAt, source, sourceUrl.

post rows — postId, slug, url, title, subtitle, description, type, audience, paywalled, publishedAt, updatedAt, authors[], authorNames, wordCount, text, bodyHtml, bodyComplete, likes, reactions, commentCount, restacks, coverImage, podcastUrl, podcastDurationSec, sectionName, tags[], language, position.

comment rows — postId, postUrl, commentId, parentId, depth, authorName, authorHandle, authorId, date, body, likes, childrenCount, edited, deleted.

publication rows — url, title (the publication's name), subdomain, customDomain, authorName, authorHandle, authorId, authorBio, heroText, description, logoUrl, coverPhotoUrl, createdAt, firstPostDate, language, type, paymentsEnabled, plans[], monthlyPriceUsd, annualPriceUsd, subscriberCountHint, rankingDetail, hasPodcast, twitter, copyright, explicit, inviteOnly, query, position, postCount.

diagnostic rows — ok: false plus errorType, one of:

Pricing

$0.25 per 1,000 posts with their text on the FREE and BRONZE tiers. Proxy, compute and dataset storage are inside those numbers; there is no usage line on top. Set a maximum total charge on the run and the Actor stops fetching when it is reached, finishing successfully with what it already delivered.

What you are never charged for

• Any row with ok: false — a publication that does not exist, an unpublished post, an empty search, a block, a timeout, a malformed entry.
• The archive listing requests, the publication home page and the comment fetches: you pay per row, not per request.
• A run that returns no rows at all. A run whose entries were all usable but produced nothing — a publication that does not exist, a post that was unpublished, a search with no matches, a block — finishes SUCCEEDED with zero results, a status message that says so, and a free diagnostic row per entry telling you why; it bills nothing, the start fee included. A run finishes FAILED only when there was nothing usable to attempt, or the Actor itself hit an error.
• Work the run did not do. Stop it early, or let it hit maxRunSecs or your charge ceiling, and only the rows already in your dataset are billed.

Limits, and the ones that might bite

• Public posts only, and that is the whole story on paywalls. A paid post returns the same free preview any logged-out reader sees — in one measured case 4,016 of 4,604 words, in others a couple of paragraphs. paywalled: true and bodyComplete: false mark it. This Actor does not log in, does not accept a subscriber cookie, and will not return the paid remainder of a post.
• subscriberCountHint is usually null. Most publications hide their subscriber numbers, and Substack then publishes nothing — not a zero. Search hits are more forthcoming than home pages: 18 of 20 in one capture carried an order-of-magnitude figure ("4.8K+") and a launch blurb.
• The publication profile has no description field of its own. The description column on a publication row comes from the home page's Open Graph tag, so it is null for publications found through search.
• Archive search is Substack's, not yours. search passes your text to the publication's own search; matching and ranking are theirs.
• sort: top is not date-ordered, so since cannot stop the walk early under it — it only filters what comes back.
• Two spellings of one publication are two jobs. astralcodexten and https://www.astralcodexten.com are the same publication, but nothing knows that until the first request follows the redirect. List each publication once.
• A very long post is clipped at 500,000 characters of text (and of HTML), a comment at 20,000. Nothing real comes close.
• No Notes, no restack graph, no subscriber lists, no email addresses, no analytics. None of that is public, and none of it is here.
• The upstream format may change. These are the endpoints Substack's own site uses, not a documented API with a version number. When a field moves, rows keep coming with that column null, and the fix is a rebuild — the parsers are pinned against captured responses so a change is visible in a test rather than in your data.

Use it from an AI agent, or from code

The Actor is one call, and the dataset is one download.

# pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("insight.solutions/substack-scraper").call(run_input={
    "publications": ["astralcodexten"],
    "maxPostsPerPublication": 50,
    "includeBody": True,
    "since": "2026-01-01",
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "post" and row["ok"]:
        print(row["publishedAt"], row["wordCount"], row["title"])

Every row carries sourceUrl, the exact endpoint it was read from, so any value can be traced back to the request that produced it.

FAQ

Does it need my Substack login? No — and it will not take one. Public endpoints only.

Can it get posts behind a paywall? No. It returns the free preview Substack serves to everyone, marked paywalled: true and bodyComplete: false.

Does a custom domain work? Yes. The API lives at the publication's own address, whatever that address is, and a *.substack.com handle that redirects to a custom domain is followed automatically.

How do I get just the new posts each morning? Set since to yesterday and schedule the run. With sort: new the walk stops at the cutoff, so it costs one or two requests per publication however long the archive is.

How do I rebuild a comment thread? Sort by position/date and nest on parentId; depth is already computed. Top-level comments have parentId: null.

Why did a publication come back with no publication row? Its home page did not answer, or did not carry the profile blob. The posts are unaffected and a free diagnostic row explains it.

Can I get a post's text without paying per post? Not per post — the text is one request per post, which is what the post price covers. includeBody: false gives you everything except the text at the same per-row price, from one request per fifty posts.

What happens if I list the same publication twice? It is read, and billed, once. Two different spellings of the same publication are the exception — see Limits.

Legal and data-protection notes

• Public endpoints only. Every endpoint this Actor reads is one the publication's own website calls to render a page for a logged-out visitor. The Actor never logs in, never accepts cookies or session tokens, and never uses an API key or subscription belonging to anyone.
• Post content belongs to its author. Titles, text and images are copyrighted works, and copyright on the publication row is often where the author states their terms. Indexing, analysis, quotation and linking are the ordinary uses and are what this is built for; republishing is your call and your responsibility.
• Comments are written by identifiable people. Author names, handles and user IDs are personal data in most jurisdictions even though they are published. Have a lawful basis before you process them, and do not use them for anything the commenters would be surprised by.
• Paywalled content stays paywalled. This Actor returns only what Substack serves to the public. Do not use it in an attempt to reconstruct paid posts.
• Not affiliated with Substack Inc. or with any publication whose content you retrieve. All names and trademarks belong to their owners and are used only to describe which public endpoints this Actor reads.

Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

Video, audio & social

• YouTube Transcript API — captions as timed segments, text, SRT or VTT, with language fallback and translation.
• YouTube Comments API — comments and replies with likes, pinned and hearted flags, newest or top sort.
• YouTube Channel API — a channel's videos, Shorts and live streams, plus YouTube search.
• Podcast Search, Episodes & Charts API — Apple Podcasts search, charts and full episode feeds.
• Bluesky Scraper — profiles, posts, followers and follows from the public AT Protocol API.
• Telegram Channel Scraper — posts, views and channel stats from public Telegram channels.
• Hacker News API — stories, comments, users, front page and a structured "Who is hiring?" parser from the official HN APIs.
• GitHub Repos, Trending & Releases API — repository facts, trending lists, releases and search, no token needed for small runs.
• Discourse Forum API — topics, posts and categories from any Discourse community via its own JSON endpoints, usernames only.

News, documents & the web

• Google News Search, Topics & Real Article URLs — news search and topic feeds with the publisher's real URL decoded.
• Website to Markdown — Content Extractor for LLMs & RAG — any site as clean Markdown, text and heading-aware chunks.
• WordPress Posts & Pages API — posts and pages from any WordPress site as clean text or Markdown, with categories, tags and images.
• Website Metadata API — any URL's meta tags, Open Graph, Twitter cards, hreflang, feeds, JSON-LD and microdata, with normalised Product, Article, Event and FAQ entities.
• Subdomain Finder API — every subdomain of a domain from certificate-transparency logs, resolved over DNS, with an optional live HTTP check.
• Internet Archive API — archive.org search, item metadata, files and reviews.
• Wayback Machine Toolkit — archived URL inventories, snapshots and text diffs between dates.
• Website Technology Detector — the tech stack behind any site, with the evidence for each detection.
• Domain Intelligence API — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
• SEO Page Audit — sitemap crawl with on-page checks, structured data and broken-link reports.
• Keyword Suggestions API — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
• Website Contact Extractor — emails, phone numbers and social profiles from any list of websites.
• Web Search Results API — Bing and DuckDuckGo organic results with snippets, no key, no browser.
• Company Enrichment API — a domain in, a company profile out: firmographics, contacts, tech stack, DNS and hiring signal.
• Company Dossier API — one company in, twelve sections out: profile, tech, contacts, DNS, open roles, news, SEC filings, federal awards, recalls, YC batch and apps.
• Press Releases API — GlobeNewswire and PR Newswire releases plus any newsroom feed, by keyword, company, ticker or subject.
• Federal Register API — rules, proposed rules, notices and the Public Inspection desk with dockets, comment deadlines and CFR references.
• Academic Papers Search API — OpenAlex, Crossref, arXiv and PubMed in one row per paper: abstract, citations, open-access PDF, authors and venue.
• RSS & Atom Feed Monitor — any RSS, Atom or JSON feed (or an OPML file) in, only the new items out, with keyword filters and a webhook.
• Website Change Monitor — watch any pages, diff the text between runs, get change rows with added/removed lines, keyword alerts and a webhook.
• Wikipedia & Wikidata API — article text, search, daily pageviews and Wikidata entity facts, any language edition.

Business, finance & jobs

• Congress & Insider Trades API — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
• Federal Contracts, Grants & Lobbying API — SAM.gov opportunities, USAspending awards, Grants.gov notices and Senate lobbying filings in one schema.
• SEC EDGAR API — filings, XBRL financials and full-text search by ticker or CIK.
• Clinical Trials & FDA API — ClinicalTrials.gov studies plus openFDA recalls, labels, approvals, 510(k)s and adverse-event reports.
• Product & Vehicle Recalls API — CPSC, NHTSA, FDA and USDA recalls, vehicle complaints and ratings, plus a VIN decoder.
• Y Combinator Companies, Batches & Founders — the YC directory with founders and social links, filterable by batch, industry and hiring status.
• Career Site Jobs API — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
• Workday Jobs API — every posting on any Workday career site, with facets, search, posted dates and full descriptions.
• New Job Postings Monitor — new, closed and changed postings on the career sites you watch.
• Hiring Signals API — Open Roles & Hiring Surge by Company — one row per company per run: open roles, what opened and closed, department and seniority breakdowns, and a hiring-surge flag.
• Remote Jobs API — RemoteOK, Remotive, We Work Remotely, Himalayas, Jobicy and more in one schema, deduplicated.
• Shopify Products API — any Shopify store's catalogue, variants, prices and stock signals.
• WooCommerce Products API — any WooCommerce store's catalogue, prices, variants and stock through its public Store API.
• Shopify App Store API — Shopify App Store listings by category, collection or developer: pricing plans, ratings and merchant reviews.
• Luma Events API — upcoming Luma events by city, category or calendar: dates, venue, price, capacity and organiser calendar, no host names.
• Shopify Store Monitor — price drops, sales, restocks, sell-outs and new products on any Shopify store, one row per change.
• Public Tenders API — EU TED, UK Find a Tender and Contracts Finder notices by keyword, CPV code, country, stage and deadline.
• Nonprofit & IRS 990 Lookup API — search US nonprofits and get EIN, NTEE code and multi-year Form 990 financials.
• OpenStreetMap Places API — businesses and points of interest by category and area from OpenStreetMap: name, address, coordinates, website, phone, opening hours.

Apps & games

• App Store & Google Play Reviews API — reviews from both stores with ratings, versions and developer replies.
• App Store Top Charts & App Search API — Apple top charts by country and genre, plus app search and details.
• App Store Keyword Rank Tracker — where any app ranks for any keyword on the App Store and Google Play, with rank changes and ASO suggestions.
• Steam Reviews API — Steam reviews with playtime, helpfulness and game details.
• Steam Game Data API — prices, tags, review scores, live player counts and top charts.
• Sports Scores & Schedules API — scores, schedules, standings, teams and headlines for the NFL, NBA, MLB, NHL, soccer and more, no key.
• App Store Monitor — App Store and Google Play listings watched between runs: new versions with release notes, price, rating and listing changes, one row per change.