# Substack Scraper — Posts, Comments & Publications (`insight.solutions/substack-scraper`) Actor

Scrape Substack as data. One row per post: title, full text, word count, paywall flag, likes, restacks, tags, author. Plus comment threads with reply depth, publication profiles with prices and subscriber hints, and Substack-wide publication search. Public posts only, no login, no API key.

- **URL**: https://apify.com/insight.solutions/substack-scraper.md
- **Developed by:** [Insight Solutions](https://apify.com/insight.solutions) (community)
- **Categories:** News, Social media, Marketing
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.15 / 1,000 post returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Posts, Comments & Publications

**Any Substack, as a flat table.** Give this Actor a handle (`astralcodexten`), a
publication address (`https://www.lennysnewsletter.com`) or a single post URL —
mixed together, in any order — and get one row per post: the title, the **full
text**, the word count, the publish date, the author, the paywall flag, likes,
restacks, comment count, tags and the cover image. Switch comments on and every
comment becomes a row too, with `parentId` and `depth` so the thread rebuilds
exactly. Every publication also gets a row of its own: author, bio, tagline,
logo, subscription prices, launch date and a subscriber hint. Add a keyword and
you get every publication Substack's own directory can find for it.

No API key. No login. No subscription of yours or anyone else's. **$0.25 per
1,000 posts, full text and proxy included**, publications that could not be read
are free, and a run that returns nothing costs nothing at all.

### Try it in 30 seconds

```json
{
  "publications": ["astralcodexten", "https://www.lennysnewsletter.com"],
  "maxPostsPerPublication": 20,
  "includeBody": true,
  "includePublicationInfo": true
}
```

Two publications, the twenty most recent posts of each, full text included. A
bare handle, a `*.substack.com` address and a custom domain all work — and a
handle whose publication has moved to its own domain is followed automatically.

Want one post? `{"publications": ["https://www.astralcodexten.com/p/royce-on-san-francisco"]}`.
Want yesterday only? `{"since": "2026-09-10"}` — the walk stops as soon as the
archive passes the cutoff, so a daily run is one or two requests per publication.
Want a list of publications? `{"searchQueries": ["machine learning"], "maxResultsPerQuery": 100}`.

**Output** — one row per post, plus one row per comment and one per publication;
the fields you will use most are `title`, `url`, `publishedAt`, `text`, `likes`
and `commentCount` (full list under *Output reference*). Anything that could not
be read comes back as a free diagnostic row (`ok: false`, `errorType`, `error`)
instead of a charge.

**Price** — **$0.25 per 1,000 posts** on the FREE tier with the full text
included, $0.20 per 1,000 comments, $2 per 1,000 publication rows, plus $0.001
per run; failed rows, the listing requests themselves and a run that returns
nothing are all free. Pay-per-event, no API key, no browser, limited permissions
— it works over the Apify MCP server (`mcp.apify.com`) and with agentic (x402)
payments.

**From code** —
`client.actor("insight.solutions/substack-scraper").call(run_input={…})` with
`apify-client`, or `POST
https://api.apify.com/v2/acts/insight.solutions~substack-scraper/run-sync-get-dataset-items`.
Full snippets under *Use it from an AI agent, or from code*.

### What comes back

One `post` row per post (nulls omitted here for readability):

```jsonc
{
  "ok": true,
  "rowType": "post",
  "input": "astralcodexten",
  "publication": "www.astralcodexten.com",
  "publicationName": "Astral Codex Ten",
  "publicationId": "89120",
  "postId": "212242052",
  "slug": "royce-on-san-francisco",
  "url": "https://www.astralcodexten.com/p/royce-on-san-francisco",
  "title": "Royce On San Francisco",
  "type": "newsletter",
  "audience": "everyone",
  "paywalled": false,
  "publishedAt": "2026-09-10T11:10:18.859Z",
  "updatedAt": "2026-09-10T11:14:07.394Z",
  "authors": [{ "id": "12009663", "name": "Scott Alexander", "handle": "astralcodexten" }],
  "authorNames": "Scott Alexander",
  "wordCount": 2393,
  "text": "On housing prices:\nThe San Franciscans at this moment, living in their rag palaces…",
  "bodyComplete": true,
  "likes": 155,
  "reactions": { "❤": 155 },
  "commentCount": 76,
  "restacks": 2,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/582979e2…jpeg",
  "language": "en",
  "position": 1,
  "scrapedAt": "2026-09-11T21:12:44.000Z",
  "source": "substack",
  "sourceUrl": "https://www.astralcodexten.com/api/v1/posts/royce-on-san-francisco"
}
```

One `comment` row per comment, when you ask for them:

```jsonc
{
  "rowType": "comment",
  "publication": "www.astralcodexten.com",
  "postId": "212242052",
  "postUrl": "https://www.astralcodexten.com/p/royce-on-san-francisco",
  "commentId": "334272881",
  "parentId": "334012158",
  "depth": 1,
  "authorName": "Judith Stove",
  "authorHandle": "judithstove",
  "date": "2026-09-10T21:19:05.486Z",
  "body": "Yep, I was thinking 'is this AI doing nineteenth-century pastiche?'…",
  "likes": 3,
  "childrenCount": 2,
  "edited": true,
  "deleted": false
}
```

And one `publication` row per publication:

```jsonc
{
  "rowType": "publication",
  "publication": "www.astralcodexten.com",
  "publicationName": "Astral Codex Ten",
  "publicationId": "89120",
  "url": "https://www.astralcodexten.com",
  "subdomain": "astralcodexten",
  "customDomain": "www.astralcodexten.com",
  "authorName": "Scott Alexander",
  "heroText": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
  "createdAt": "2020-08-30T04:18:18.308Z",
  "firstPostDate": "2020-08-30T21:01:57.050Z",
  "paymentsEnabled": true,
  "plans": [
    { "interval": "month", "amount": 1000, "currency": "usd", "nickname": "$10 a month" },
    { "interval": "year", "amount": 10000, "currency": "usd", "nickname": "$100 a year" }
  ],
  "monthlyPriceUsd": 10,
  "annualPriceUsd": 100,
  "hasPodcast": false,
  "postCount": 73
}
```

Every row — post, comment, publication or diagnostic — carries the **same
columns**, null where they do not apply, so the dataset exports as one
rectangular table however you mixed the inputs. The dataset has four ready-made
views: **Posts**, **Comments**, **Publications** and **Problems**.

### Use cases

- **Read a competitor's whole archive as text** — headline, subtitle, word count
  and body for every post they have published, in one table you can search.
- **Track a topic across hundreds of newsletters.** Search the directory for
  "machine learning", take the publications, then run their archives with
  `since` on a daily schedule and only ever fetch what is new.
- **Feed a RAG index or a summariser.** `text` is already plain text with
  paragraph breaks, not HTML you have to clean, and `bodyComplete` tells you
  whether you are indexing a whole post or a paywalled preview.
- **Study an audience, not just a writer.** Comment rows carry author, timestamp,
  likes and reply depth — a thread rebuilt exactly, ready for sentiment or
  network analysis.
- **Build a market map.** Publication rows carry prices, launch date, subscriber
  hints and the publication's own ranking blurb, which is what a "who is doing
  this well and what do they charge" spreadsheet is made of.
- **Watch one post.** Paste its URL and re-run it: likes, restacks and comment
  count are on the row every time.

### How it works, and why it keeps working

Every Substack publication serves the same small JSON API at **its own address** —
`name.substack.com` or the custom domain, the same endpoints either way. This
Actor reads four of them, plus the publication's home page. All are the ones
Substack's own website calls to render itself; none needs a key, a cookie or a
session.

- **The archive is paged by what came back, not by what was asked for.** The
  first page of a real publication answered `limit=50` with 23 posts, and the
  next one with 50. Walking by "offset += the page's length, stop on an empty
  array" is the only rule that survives that; stopping at the first short page
  would read a fifth of an archive and call it the archive.
- **The list is cheap and the text is not.** One request buys up to fifty posts'
  metadata; the text is one request per post. `includeBody: false` is therefore
  about fifty times fewer requests, which is the right setting for building an
  index and the wrong one for reading the writing.
- **A publication that moved is followed.** A `*.substack.com` address that now
  redirects to a custom domain is followed once, and every later request in that
  job goes straight to where it landed.
- **A refusal is not fatal.** A blocked request rotates the proxy session once
  and tries again from a different exit IP. Only a second refusal ends that job,
  with everything already delivered kept and one free diagnostic row saying where
  it stopped.
- **Requests to one publication are spaced 250–600 ms apart.** Nothing forces
  this; it is the difference between reading a public API and hammering
  somebody's newsletter.
- **Partial results are kept.** Hit `maxRunSecs` or your charge ceiling halfway
  through an archive and you keep — and are billed for — exactly the rows that
  were written, plus a free row saying what was missed.

### How it compares

- **The full text is the default, not an add-on.** `text` is the post as a reader
  sees it, converted from the published HTML with paragraph and list breaks
  intact, entities decoded and image furniture dropped.
- **Paywalls are labelled, not pretended about.** A paid post returns the free
  preview Substack serves to the public; the row says `paywalled: true` and
  `bodyComplete: false` so you always know which one you have. See *Limits*.
- **One schema for four things.** A post, a comment, a publication profile and a
  search hit come back as the same columns, so mixing them is a filter rather
  than a reconciliation project.
- **Failures are free and legible.** A publication that does not exist, a post
  that was unpublished, an empty search or a block produce a diagnostic row with
  an `errorType` you can branch on — and no charge. A run that returns nothing at
  all finishes **SUCCEEDED with zero results**, a status message that says so,
  and a free diagnostic row per entry telling you why; it finishes **FAILED**
  only when no entry was usable, or the Actor itself hit an error.
- **No login, ever.** No account, no cookie jar, no session token, no paid
  subscription of yours borrowed for the run. Public endpoints only, which is
  also why it runs under Apify's **limited permissions**.

### Input reference

| Input | Type | Default | What it does |
|---|---|---|---|
| `publications` | array | — | Handles (`astralcodexten`), addresses (`name.substack.com`, custom domains — the API lives at the same address) or single post URLs (`…/p/slug`). Share links from `open.substack.com` work too. Deduplicated by address |
| `maxPostsPerPublication` | integer | `50` | Posts per publication, newest first. `0` = the whole archive |
| `includeBody` | boolean | `true` | Fetch each post's full text — one extra request per post. Off = archive listing only, ~50 posts per request |
| `sort` | select | `new` | `new` (archive order) or `top` (Substack's own ranking of the whole archive) |
| `search` | string | — | Substack's archive search, run *inside* each publication you listed |
| `since` | string | — | ISO date or timestamp. With `sort: new` the walk stops as soon as the archive passes it |
| `postTypes` | array | all four | `newsletter`, `podcast`, `thread`, `video` |
| `includeHtml` | boolean | `false` | Add `bodyHtml`, the post's markup as published |
| `includeComments` | boolean | `false` | One extra request per commented post; one row per comment |
| `maxCommentsPerPost` | integer | `100` | Counted depth-first, so you keep the top of the thread. `0` = all |
| `commentSort` | select | `best_first` | `best_first`, `newest_first`, `oldest_first` |
| `includePublicationInfo` | boolean | `true` | One `publication` row per publication, from its home page |
| `searchQueries` | array | `[]` | Substack's publication directory, by keyword |
| `maxResultsPerQuery` | integer | `20` | Publications per query; 20 per page, Substack stops at 200 |
| `maxPagesPerPublication` | integer | `40` | Hard stop on archive depth — 40 pages is 2,000 posts |
| `maxConcurrency` | integer | `4` | Publications read in parallel, each on its own proxy session |
| `maxRunSecs` | integer | `240` | Wall-clock budget for the whole run |
| `proxyConfiguration` | object | `{ "useApifyProxy": true }` | Datacenter proxy by default; switch to residential only if you see `blocked` rows |

### Output reference

Shared by every row: `ok`, `rowType`, `input`, `publication`, `publicationName`,
`publicationId`, `error`, `errorType`, `scrapedAt`, `source`, `sourceUrl`.

**`post` rows** — `postId`, `slug`, `url`, `title`, `subtitle`, `description`,
`type`, `audience`, `paywalled`, `publishedAt`, `updatedAt`, `authors[]`,
`authorNames`, `wordCount`, `text`, `bodyHtml`, `bodyComplete`, `likes`,
`reactions`, `commentCount`, `restacks`, `coverImage`, `podcastUrl`,
`podcastDurationSec`, `sectionName`, `tags[]`, `language`, `position`.

**`comment` rows** — `postId`, `postUrl`, `commentId`, `parentId`, `depth`,
`authorName`, `authorHandle`, `authorId`, `date`, `body`, `likes`,
`childrenCount`, `edited`, `deleted`.

**`publication` rows** — `url`, `title` (the publication's name), `subdomain`,
`customDomain`, `authorName`, `authorHandle`, `authorId`, `authorBio`,
`heroText`, `description`, `logoUrl`, `coverPhotoUrl`, `createdAt`,
`firstPostDate`, `language`, `type`, `paymentsEnabled`, `plans[]`,
`monthlyPriceUsd`, `annualPriceUsd`, `subscriberCountHint`, `rankingDetail`,
`hasPodcast`, `twitter`, `copyright`, `explicit`, `inviteOnly`, `query`,
`position`, `postCount`.

**`diagnostic` rows** — `ok: false` plus `errorType`, one of:

| `errorType` | What it means |
|---|---|
| `not-found` | No publication at that address, or the post has been unpublished or renamed |
| `no-results` | The archive, the archive search or the publication search matched nothing |
| `blocked` | Two requests from two different exit IPs were both refused |
| `timeout` | The run's time budget ran out before this entry was finished |
| `invalid-input` | The entry could not be read as a handle, an address or a post URL |
| `unavailable` | Something unexpected — the message says what |

### Pricing

| Event | Price (FREE/BRONZE) | SILVER | GOLD |
|---|---|---|---|
| `post` — one post row, full text included | **$0.00025** | $0.0002 | $0.00015 |
| `comment` — one comment row | $0.0002 | $0.0002 | $0.0002 |
| `publication` — one publication profile or search hit | $0.002 | $0.002 | $0.002 |
| `actor-start` — once per run that returned something | $0.001 | $0.001 | $0.001 |

**$0.25 per 1,000 posts** with their text on the FREE and BRONZE tiers. Proxy,
compute and dataset storage are inside those numbers; there is no usage line on
top. Set a maximum total charge on the run and the Actor stops fetching when it
is reached, finishing successfully with what it already delivered.

#### What you are never charged for

- Any row with `ok: false` — a publication that does not exist, an unpublished
  post, an empty search, a block, a timeout, a malformed entry.
- The archive listing requests, the publication home page and the comment
  fetches: you pay per **row**, not per request.
- A run that returns no rows at all. A run whose entries were all usable but
  produced nothing — a publication that does not exist, a post that was
  unpublished, a search with no matches, a block — finishes **SUCCEEDED with
  zero results**, a status message that says so, and a free diagnostic row per
  entry telling you why; it bills nothing, the start fee included. A run
  finishes **FAILED** only when there was nothing usable to attempt, or the
  Actor itself hit an error.
- Work the run did not do. Stop it early, or let it hit `maxRunSecs` or your
  charge ceiling, and only the rows already in your dataset are billed.

### Limits, and the ones that might bite

- **Public posts only, and that is the whole story on paywalls.** A paid post
  returns the same free preview any logged-out reader sees — in one measured case
  4,016 of 4,604 words, in others a couple of paragraphs. `paywalled: true` and
  `bodyComplete: false` mark it. This Actor does not log in, does not accept a
  subscriber cookie, and will not return the paid remainder of a post.
- **`subscriberCountHint` is usually null.** Most publications hide their
  subscriber numbers, and Substack then publishes nothing — not a zero. Search
  hits are more forthcoming than home pages: 18 of 20 in one capture carried an
  order-of-magnitude figure ("4.8K+") and a launch blurb.
- **The publication profile has no description field of its own.** The
  `description` column on a publication row comes from the home page's Open Graph
  tag, so it is null for publications found through search.
- **Archive search is Substack's, not yours.** `search` passes your text to the
  publication's own search; matching and ranking are theirs.
- **`sort: top` is not date-ordered**, so `since` cannot stop the walk early
  under it — it only filters what comes back.
- **Two spellings of one publication are two jobs.** `astralcodexten` and
  `https://www.astralcodexten.com` are the same publication, but nothing knows
  that until the first request follows the redirect. List each publication once.
- **A very long post is clipped** at 500,000 characters of text (and of HTML), a
  comment at 20,000. Nothing real comes close.
- **No Notes, no restack graph, no subscriber lists, no email addresses, no
  analytics.** None of that is public, and none of it is here.
- **The upstream format may change.** These are the endpoints Substack's own site
  uses, not a documented API with a version number. When a field moves, rows keep
  coming with that column null, and the fix is a rebuild — the parsers are pinned
  against captured responses so a change is visible in a test rather than in your
  data.

### Use it from an AI agent, or from code

The Actor is one call, and the dataset is one download.

```python
## pip install apify-client
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("insight.solutions/substack-scraper").call(run_input={
    "publications": ["astralcodexten"],
    "maxPostsPerPublication": 50,
    "includeBody": True,
    "since": "2026-01-01",
})

for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["rowType"] == "post" and row["ok"]:
        print(row["publishedAt"], row["wordCount"], row["title"])
```

Every row carries `sourceUrl`, the exact endpoint it was read from, so any value
can be traced back to the request that produced it.

### FAQ

**Does it need my Substack login?** No — and it will not take one. Public
endpoints only.

**Can it get posts behind a paywall?** No. It returns the free preview Substack
serves to everyone, marked `paywalled: true` and `bodyComplete: false`.

**Does a custom domain work?** Yes. The API lives at the publication's own
address, whatever that address is, and a `*.substack.com` handle that redirects
to a custom domain is followed automatically.

**How do I get just the new posts each morning?** Set `since` to yesterday and
schedule the run. With `sort: new` the walk stops at the cutoff, so it costs one
or two requests per publication however long the archive is.

**How do I rebuild a comment thread?** Sort by `position`/`date` and nest on
`parentId`; `depth` is already computed. Top-level comments have `parentId: null`.

**Why did a publication come back with no `publication` row?** Its home page did
not answer, or did not carry the profile blob. The posts are unaffected and a
free diagnostic row explains it.

**Can I get a post's text without paying per post?** Not per post — the text is
one request per post, which is what the `post` price covers. `includeBody: false`
gives you everything except the text at the same per-row price, from one request
per fifty posts.

**What happens if I list the same publication twice?** It is read, and billed,
once. Two different spellings of the same publication are the exception — see
*Limits*.

### Legal and data-protection notes

- **Public endpoints only.** Every endpoint this Actor reads is one the
  publication's own website calls to render a page for a logged-out visitor. The
  Actor never logs in, never accepts cookies or session tokens, and never uses an
  API key or subscription belonging to anyone.
- **Post content belongs to its author.** Titles, text and images are copyrighted
  works, and `copyright` on the publication row is often where the author states
  their terms. Indexing, analysis, quotation and linking are the ordinary uses
  and are what this is built for; republishing is your call and your
  responsibility.
- **Comments are written by identifiable people.** Author names, handles and user
  IDs are personal data in most jurisdictions even though they are published.
  Have a lawful basis before you process them, and do not use them for anything
  the commenters would be surprised by.
- **Paywalled content stays paywalled.** This Actor returns only what Substack
  serves to the public. Do not use it in an attempt to reconstruct paid posts.
- **Not affiliated with Substack Inc. or with any publication whose content you
  retrieve.** All names and trademarks belong to their owners and are used only
  to describe which public endpoints this Actor reads.

### Our other Actors

Every Insight Solutions Actor is pay-per-result with no browser, no login and no API key, and every one of them returns free diagnostic rows instead of billing for failures. Prices are per 1,000 results.

**Video, audio & social**

- [YouTube Transcript API](https://apify.com/insight.solutions/youtube-transcript-api) — captions as timed segments, text, SRT or VTT, with language fallback and translation.
- [YouTube Comments API](https://apify.com/insight.solutions/youtube-comments-api) — comments and replies with likes, pinned and hearted flags, newest or top sort.
- [YouTube Channel API](https://apify.com/insight.solutions/youtube-channel-api) — a channel's videos, Shorts and live streams, plus YouTube search.
- [Podcast Search, Episodes & Charts API](https://apify.com/insight.solutions/podcast-api) — Apple Podcasts search, charts and full episode feeds.
- [Bluesky Scraper](https://apify.com/insight.solutions/bluesky-scraper) — profiles, posts, followers and follows from the public AT Protocol API.
- [Telegram Channel Scraper](https://apify.com/insight.solutions/telegram-channel-scraper) — posts, views and channel stats from public Telegram channels.
- [Hacker News API](https://apify.com/insight.solutions/hacker-news-api) — stories, comments, users, front page and a structured "Who is hiring?" parser from the official HN APIs.
- [GitHub Repos, Trending & Releases API](https://apify.com/insight.solutions/github-repos-api) — repository facts, trending lists, releases and search, no token needed for small runs.
- [Discourse Forum API](https://apify.com/insight.solutions/discourse-forum-api) — topics, posts and categories from any Discourse community via its own JSON endpoints, usernames only.

**News, documents & the web**

- [Google News Search, Topics & Real Article URLs](https://apify.com/insight.solutions/google-news-api) — news search and topic feeds with the publisher's real URL decoded.
- [Website to Markdown — Content Extractor for LLMs & RAG](https://apify.com/insight.solutions/website-content-extractor) — any site as clean Markdown, text and heading-aware chunks.
- [WordPress Posts & Pages API](https://apify.com/insight.solutions/wordpress-posts-api) — posts and pages from any WordPress site as clean text or Markdown, with categories, tags and images.
- [Website Metadata API](https://apify.com/insight.solutions/website-metadata-api) — any URL's meta tags, Open Graph, Twitter cards, hreflang, feeds, JSON-LD and microdata, with normalised Product, Article, Event and FAQ entities.
- [Subdomain Finder API](https://apify.com/insight.solutions/subdomain-finder-api) — every subdomain of a domain from certificate-transparency logs, resolved over DNS, with an optional live HTTP check.
- [Internet Archive API](https://apify.com/insight.solutions/internet-archive-api) — archive.org search, item metadata, files and reviews.
- [Wayback Machine Toolkit](https://apify.com/insight.solutions/wayback-toolkit) — archived URL inventories, snapshots and text diffs between dates.
- [Website Technology Detector](https://apify.com/insight.solutions/website-tech-detector) — the tech stack behind any site, with the evidence for each detection.
- [Domain Intelligence API](https://apify.com/insight.solutions/domain-intelligence-api) — DNS, RDAP registration, TLS certificate and HTTP facts in one row per domain.
- [SEO Page Audit](https://apify.com/insight.solutions/seo-page-audit) — sitemap crawl with on-page checks, structured data and broken-link reports.
- [Keyword Suggestions API](https://apify.com/insight.solutions/keyword-suggestions-api) — Google, YouTube, Bing, Amazon and eBay autocomplete with alphabet and question expansions.
- [Website Contact Extractor](https://apify.com/insight.solutions/website-contact-extractor) — emails, phone numbers and social profiles from any list of websites.
- [Web Search Results API](https://apify.com/insight.solutions/web-search-api) — Bing and DuckDuckGo organic results with snippets, no key, no browser.
- [Company Enrichment API](https://apify.com/insight.solutions/company-enrichment-api) — a domain in, a company profile out: firmographics, contacts, tech stack, DNS and hiring signal.
- [Company Dossier API](https://apify.com/insight.solutions/company-dossier-api) — one company in, twelve sections out: profile, tech, contacts, DNS, open roles, news, SEC filings, federal awards, recalls, YC batch and apps.
- [Press Releases API](https://apify.com/insight.solutions/press-releases-api) — GlobeNewswire and PR Newswire releases plus any newsroom feed, by keyword, company, ticker or subject.
- [Federal Register API](https://apify.com/insight.solutions/federal-register-api) — rules, proposed rules, notices and the Public Inspection desk with dockets, comment deadlines and CFR references.
- [Academic Papers Search API](https://apify.com/insight.solutions/academic-papers-api) — OpenAlex, Crossref, arXiv and PubMed in one row per paper: abstract, citations, open-access PDF, authors and venue.
- [RSS & Atom Feed Monitor](https://apify.com/insight.solutions/rss-feed-monitor) — any RSS, Atom or JSON feed (or an OPML file) in, only the new items out, with keyword filters and a webhook.
- [Website Change Monitor](https://apify.com/insight.solutions/website-change-monitor) — watch any pages, diff the text between runs, get change rows with added/removed lines, keyword alerts and a webhook.
- [Wikipedia & Wikidata API](https://apify.com/insight.solutions/wikipedia-api) — article text, search, daily pageviews and Wikidata entity facts, any language edition.

**Business, finance & jobs**

- [Congress & Insider Trades API](https://apify.com/insight.solutions/congress-insider-trades-api) — STOCK Act periodic transaction reports and SEC Form 4 insider trades in one schema.
- [Federal Contracts, Grants & Lobbying API](https://apify.com/insight.solutions/federal-contracts-grants-api) — SAM.gov opportunities, USAspending awards, Grants.gov notices and Senate lobbying filings in one schema.
- [SEC EDGAR API](https://apify.com/insight.solutions/sec-edgar-api) — filings, XBRL financials and full-text search by ticker or CIK.
- [Clinical Trials & FDA API](https://apify.com/insight.solutions/clinical-trials-fda-api) — ClinicalTrials.gov studies plus openFDA recalls, labels, approvals, 510(k)s and adverse-event reports.
- [Product & Vehicle Recalls API](https://apify.com/insight.solutions/product-recalls-api) — CPSC, NHTSA, FDA and USDA recalls, vehicle complaints and ratings, plus a VIN decoder.
- [Y Combinator Companies, Batches & Founders](https://apify.com/insight.solutions/yc-companies-directory) — the YC directory with founders and social links, filterable by batch, industry and hiring status.
- [Career Site Jobs API](https://apify.com/insight.solutions/ats-jobs-api) — jobs straight from Greenhouse, Lever, Ashby, Workable and 10+ other ATS career sites.
- [Workday Jobs API](https://apify.com/insight.solutions/workday-jobs-api) — every posting on any Workday career site, with facets, search, posted dates and full descriptions.
- [New Job Postings Monitor](https://apify.com/insight.solutions/job-postings-monitor) — new, closed and changed postings on the career sites you watch.
- [Hiring Signals API — Open Roles & Hiring Surge by Company](https://apify.com/insight.solutions/hiring-signals-api) — one row per company per run: open roles, what opened and closed, department and seniority breakdowns, and a hiring-surge flag.
- [Remote Jobs API](https://apify.com/insight.solutions/remote-jobs-api) — RemoteOK, Remotive, We Work Remotely, Himalayas, Jobicy and more in one schema, deduplicated.
- [Shopify Products API](https://apify.com/insight.solutions/shopify-products-api) — any Shopify store's catalogue, variants, prices and stock signals.
- [WooCommerce Products API](https://apify.com/insight.solutions/woocommerce-products-api) — any WooCommerce store's catalogue, prices, variants and stock through its public Store API.
- [Shopify App Store API](https://apify.com/insight.solutions/shopify-app-store-api) — Shopify App Store listings by category, collection or developer: pricing plans, ratings and merchant reviews.
- [Luma Events API](https://apify.com/insight.solutions/luma-events-api) — upcoming Luma events by city, category or calendar: dates, venue, price, capacity and organiser calendar, no host names.
- [Shopify Store Monitor](https://apify.com/insight.solutions/shopify-store-monitor) — price drops, sales, restocks, sell-outs and new products on any Shopify store, one row per change.
- [Public Tenders API](https://apify.com/insight.solutions/public-tenders-api) — EU TED, UK Find a Tender and Contracts Finder notices by keyword, CPV code, country, stage and deadline.
- [Nonprofit & IRS 990 Lookup API](https://apify.com/insight.solutions/nonprofit-990-api) — search US nonprofits and get EIN, NTEE code and multi-year Form 990 financials.
- [OpenStreetMap Places API](https://apify.com/insight.solutions/osm-places-api) — businesses and points of interest by category and area from OpenStreetMap: name, address, coordinates, website, phone, opening hours.

**Apps & games**

- [App Store & Google Play Reviews API](https://apify.com/insight.solutions/app-reviews-api) — reviews from both stores with ratings, versions and developer replies.
- [App Store Top Charts & App Search API](https://apify.com/insight.solutions/app-charts-api) — Apple top charts by country and genre, plus app search and details.
- [App Store Keyword Rank Tracker](https://apify.com/insight.solutions/app-store-keyword-rank-tracker) — where any app ranks for any keyword on the App Store and Google Play, with rank changes and ASO suggestions.
- [Steam Reviews API](https://apify.com/insight.solutions/steam-reviews-api) — Steam reviews with playtime, helpfulness and game details.
- [Steam Game Data API](https://apify.com/insight.solutions/steam-store-stats-api) — prices, tags, review scores, live player counts and top charts.
- [Sports Scores & Schedules API](https://apify.com/insight.solutions/sports-scores-api) — scores, schedules, standings, teams and headlines for the NFL, NBA, MLB, NHL, soccer and more, no key.
- [App Store Monitor](https://apify.com/insight.solutions/app-store-monitor) — App Store and Google Play listings watched between runs: new versions with release notes, price, rating and listing changes, one row per change.

# Actor input Schema

## `publications` (type: `array`):

One entry per publication: a **Substack handle** (`astralcodexten`), a **publication address** (`astralcodexten.substack.com` or a custom domain like `https://www.lennysnewsletter.com`), or a **single post URL** (`https://www.astralcodexten.com/p/royce-on-san-francisco`), which returns that one post instead of the archive. Share links (`open.substack.com/pub/…/p/…`) work too, a moved handle is followed to its new domain, and duplicates are read, and billed, once.

## `maxPostsPerPublication` (type: `integer`):

How many posts to return for each publication, newest first. Set **0 for the whole archive**, which for a publication running since 2020 is several hundred posts and, with `includeBody` on, several hundred requests. The archive list costs one request per 50 posts whatever you ask for; the bodies are what cost.

## `includeBody` (type: `boolean`):

**On (default):** one extra request per post, and every row carries the post's full text in `text`. **Off:** the archive listing alone — title, subtitle, date, author, word count, likes, comment count, tags, cover image — at one request per 50 posts and no text column. Turning it off is roughly fifty times fewer requests, so it is the setting to use when you are building an index rather than reading the writing.

## `sort` (type: `string`):

**Newest first** is the archive as the publication shows it, and the only order in which the `since` cutoff can stop paging early. **Most popular** is Substack's own ranking of the whole archive — useful for "what does this publication get read for", not for "what did they publish last month".

## `postTypes` (type: `array`):

Which kinds of post to keep. Substack labels every post `newsletter`, `podcast`, `thread` or `video`. Leave all four selected for everything. Filtering does not reduce the archive requests — the list has to be read to find them — but it does reduce the rows you are charged for.

## `searchQueries` (type: `array`):

Search Substack's own publication directory. Each query returns up to `maxResultsPerQuery` **publication** rows — name, author, address, prices, launch blurb, subscriber order of magnitude — and no posts. This is the input for building a list: "every Substack about machine learning", "who writes about monetary policy".

## `maxResultsPerQuery` (type: `integer`):

How many publications each query returns. The directory answers twenty per page and stops at 200 for one query, however many more exist.

## `search` (type: `string`):

Return only posts matching this text, using Substack's own archive search (it reads titles and post text). Leave empty for the whole archive. This searches *inside* the publications you listed; to find publications themselves, use `searchQueries` instead.

## `since` (type: `string`):

An ISO date (`2026-01-01`) or timestamp. With **Newest first** the walk stops as soon as the archive passes the cutoff, so "everything since Monday" costs one or two requests rather than the whole archive — which is what makes this Actor cheap to run on a schedule.

## `includeComments` (type: `boolean`):

Read every post's comment thread as well: one extra request per post that has comments, and one row per comment with `parentId` and `depth` so the thread can be rebuilt. Off by default — a lively post is 76 comment rows, and a hundred posts of them is a different sort of job from reading a hundred posts.

## `maxCommentsPerPost` (type: `integer`):

How many comment rows each post may produce, counted depth-first so what you keep is the top of the thread rather than an arbitrary slice. Set 0 for every comment. Ignored unless `includeComments` is on.

## `commentSort` (type: `string`):

The order Substack returns the thread in — the same three orders the site's own dropdown offers. Replies always follow the comment they answer whichever you pick.

## `includeHtml` (type: `boolean`):

Add a `bodyHtml` column holding the post as the publisher wrote it — links, images, footnotes, block quotes. The plain-text `text` column is always there when `includeBody` is on; this is for when you want the markup. It makes rows several times larger and it is off by default.

## `includePublicationInfo` (type: `boolean`):

Write one `publication` row per publication: name, author, bio, hero text, logo, created date, first post date, subscription prices, subscriber hint, podcast flag. It costs one request for the publication's home page (a few hundred KB), once per run however many posts you asked for.

## `maxPagesPerPublication` (type: `integer`):

A hard stop on how deep the archive walk goes: 40 pages is 2,000 posts, which is longer than almost any Substack. It exists so that `maxPostsPerPublication: 0` on a publication with a decade of daily posts cannot run all afternoon.

## `maxConcurrency` (type: `integer`):

How many publications, post URLs or queries are read at once. Each parallel worker keeps its own proxy session, so one blocked publication burns only its own exit IP. Requests to a *single* publication are always spaced a few hundred milliseconds apart, whatever this is set to.

## `maxRunSecs` (type: `integer`):

Wall-clock budget for the whole run. When it is reached the Actor stops fetching, keeps and bills for every row it already wrote, and files a free diagnostic row for each entry it did not finish. Nothing is charged for work that did not happen.

## `proxyConfiguration` (type: `object`):

Substack answered datacenter addresses on every endpoint this Actor uses, so the default is Apify's datacenter proxy — much cheaper than residential, and the proxy cost is already inside the per-row price. Switch to residential only if you see `blocked` rows. Sessions rotate automatically when an exit IP is refused.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten",
    "https://www.lennysnewsletter.com",
    "noahpinion",
    "https://www.astralcodexten.com/p/royce-on-san-francisco"
  ],
  "maxPostsPerPublication": 20,
  "includeBody": true,
  "sort": "new",
  "postTypes": [
    "newsletter",
    "podcast",
    "thread",
    "video"
  ],
  "searchQueries": [
    "machine learning",
    "urban planning"
  ],
  "maxResultsPerQuery": 20,
  "search": "AI",
  "since": "2026-01-01",
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "commentSort": "best_first",
  "includeHtml": false,
  "includePublicationInfo": true,
  "maxPagesPerPublication": 40,
  "maxConcurrency": 4,
  "maxRunSecs": 240,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One row per post — title, full text, word count, paywall flag, likes, comments, tags, author — plus one row per comment and one per publication profile. Entries that could not be read get a free diagnostic row saying why. Delivered as JSON items in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com"
    ],
    "maxPostsPerPublication": 20,
    "includeBody": true,
    "sort": "new",
    "postTypes": [
        "newsletter",
        "podcast",
        "thread",
        "video"
    ],
    "searchQueries": [],
    "maxResultsPerQuery": 20,
    "search": "",
    "since": "",
    "includeComments": false,
    "maxCommentsPerPost": 100,
    "commentSort": "best_first",
    "includeHtml": false,
    "includePublicationInfo": true,
    "maxConcurrency": 4,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("insight.solutions/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com",
    ],
    "maxPostsPerPublication": 20,
    "includeBody": True,
    "sort": "new",
    "postTypes": [
        "newsletter",
        "podcast",
        "thread",
        "video",
    ],
    "searchQueries": [],
    "maxResultsPerQuery": 20,
    "search": "",
    "since": "",
    "includeComments": False,
    "maxCommentsPerPost": 100,
    "commentSort": "best_first",
    "includeHtml": False,
    "includePublicationInfo": True,
    "maxConcurrency": 4,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("insight.solutions/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 20,
  "includeBody": true,
  "sort": "new",
  "postTypes": [
    "newsletter",
    "podcast",
    "thread",
    "video"
  ],
  "searchQueries": [],
  "maxResultsPerQuery": 20,
  "search": "",
  "since": "",
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "commentSort": "best_first",
  "includeHtml": false,
  "includePublicationInfo": true,
  "maxConcurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call insight.solutions/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,insight.solutions/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/csY4jz5AegblbTge1/builds/4e35EYpz6II48zDlP/openapi.json