Substack Community Chat Scraper

Scrape posts and comments from a Substack publication's community chat and get clean, structured JSON — one dataset item per post, with the full comment thread embedded.

Substack "chat" is the conversational feed many publications run (questions, recommendations, discussions). This Actor paginates a publication's chat, pulls each published post, fetches its comments, and returns everything in a predictable schema you can pipe into your own pipeline.

---

Features

• 📨 Posts + comments — every published, non-paywalled post with its embedded comment thread (including threaded replies via parent_comment_id).
• 👤 Author & reaction metadata — name, handle, bio, role, subscriber tier, plus emoji reaction counts.
• 🔗 Link previews — title/description/image for link-type posts.
• 📆 Date window & limit — scrape everything or bound by startDate / endDate / maxPosts.
• 🌐 Apify Proxy support — route through residential proxies to avoid Cloudflare blocks on datacenter IPs.
• 🧱 Cloudflare-aware — uses Chrome TLS/JA3 impersonation so requests look like a real browser.

---

What you need

1. The publication ID

Open the publication's chat in your browser. The URL looks like:

https://substack.com/chat/3643280

The number (3643280) is the publicationId.

2. A session cookie (substack.sid)

The chat API requires authentication. Sign in to Substack in your browser, open DevTools → Application → Cookies → substack.com, and copy the value of the substack.sid cookie into the sessionCookie input. (You can also paste the entire Cookie: header string.)

⚠️ Comments are subscriber-gated. To read a chat's comments, the account behind your cookie must be subscribed to that publication. If it isn't, the comments endpoint returns HTTP 402 and posts come back with an empty comments list (post bodies still scrape fine).

Tip: A free subscription is often enough. Subscribe (free) to the publication with the same account, then run the Actor — for many publications the comments unlock immediately. Some publications gate chat comments behind a paid subscription; for those you'll need to subscribe at the paid tier.

---

Input

Field	Type	Required	Description
publicationId	integer	✅	Numeric publication id (the number in https://substack.com/chat/<id>).
sessionCookie	string (secret)	✅	Your substack.sid cookie value, or the full Cookie header.
maxPosts	integer	—	Max posts to scrape. Omit to scrape all within the date window.
startDate	string	—	Only posts on/after this date (YYYY-MM-DD or ISO-8601).
endDate	string	—	Only posts on/before this date (YYYY-MM-DD or ISO-8601).
proxyConfiguration	object	—	Apify Proxy settings (residential recommended).

Example input

{
  "publicationId": 3643280,
  "sessionCookie": "s%3A...your-substack.sid...",
  "maxPosts": 50,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

---

Output

One dataset item per post. Example:

{
  "id": "0179ee7b-3151-4e87-b3ed-0cdc55bf06dd",
  "publication_id": 3643280,
  "created_at": "2026-05-02T14:30:00+00:00",
  "updated_at": "2026-05-02T15:00:00+00:00",
  "status": "published",
  "type": "text",
  "body": "Where are you going? What cities would you like to DO SEE EAT and DRINK in?",
  "link_url": null,
  "is_paywalled": false,
  "comment_count": 2,
  "reaction_count": 12,
  "reactions": { "🔥": 8, "❤": 4 },
  "audience": "everyone",
  "author": {
    "id": 98765, "name": "DO. SEE. EAT. DRINK.", "handle": "dosee",
    "photo_url": "https://...", "bio": null, "pub_role": "admin",
    "subscriber_tier": null, "is_bestseller": false
  },
  "link_metadata": null,
  "comments": [
    {
      "id": "c-1", "post_id": "0179ee7b-...", "body": "What about Sedona, Arizona?",
      "created_at": "2026-05-02T15:10:00+00:00",
      "author": { "id": 9002, "name": "Q El Crosby", "handle": "qelc" },
      "reactions": {}, "parent_comment_id": null
    }
  ],
  "scraped_at": "2026-05-03T02:00:00+00:00"
}

---

Pricing — pay per event

This Actor uses the pay-per-event model:

Event	When	Price (USD)
actor-start	Once per run	$0.01
post-scraped	Per post returned to the dataset	$0.004

A 50-post run therefore costs about $0.01 + 50 × $0.004 = $0.21. (Final prices are set on the Apify Console at publish time.)

---

Limitations

• Comments require a subscription to the target publication (see above); paywalled chats need a paid subscription.
• Only published, non-paywalled posts are returned; deleted/draft and paywalled posts are skipped.
• This is a bounded full scrape — there is no incremental/delta mode.