Substack Scraper - Posts, Newsletters & Comments

Collects public post archives and post metadata from Substack publications and custom domains. Optional content fetching returns plain text exposed by Substack's public post endpoint; paid posts may only expose a preview. Optional comment collection recursively returns visible public comments. Publication search is available through Substack's public search endpoint, but that endpoint can return empty results even for broad queries.

Use cases

• Monitor public newsletter publishing and engagement
• Research publication topics, authors, and post cadence
• Collect public comment discussions
• Build content feeds for AI agents / MCP tools

Input

{
  "publications": ["https://www.lennysnewsletter.com"],
  "postUrls": [],
  "searchQuery": "product management",
  "maxPostsPerPublication": 5,
  "includeContent": false,
  "includeComments": false,
  "sinceDate": "2026-09-01"
}

publications accepts Substack URLs, subdomains such as lenny or @lenny, and custom domains. postUrls adds individual posts. The prefill fetches up to five recent public posts from Lenny's Newsletter and returns charged rows quickly. sinceDate is inclusive and uses YYYY-MM-DD.

Output

Post rows include publication identity, post ID, title, subtitle, URL, date, audience, paywall flag, authors, reactions, comment count, word count, cover image, and description. Set includeContent to include plain text. Set includeComments to emit comment rows. Search results are emitted as newsletter rows.

Trimmed live post example:

{"type":"post","publication":{"name":"Lenny's Newsletter","url":"https://www.lennysnewsletter.com"},"postId":215694124,"title":"All of the Lenny & Friends Summit talks are now online!","subtitle":"Plus, some reflections and takeaways from the day","slug":"all-of-the-lenny-and-friends-summit","url":"https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit","postDate":"2026-09-29T13:15:57.512Z","audience":"everyone","isPaywalled":false,"authors":["Lenny Rachitsky"],"likes":274,"commentCount":6,"wordcount":1276,"coverImage":"https://substack-post-media.s3.amazonaws.com/public/images/9fed347c-0e02-4190-a590-1236719b99de_9213x6142.jpeg","description":"Plus, some reflections and takeaways from the day","scrapedAt":"2026-10-04T00:00:00.000Z"}

Trimmed live comment example:

{"type":"comment","postId":215694124,"commentId":348735847,"author":"Justin Wormley","body":"thank you for existing","date":"2026-09-29T15:30:12.240Z","likes":3,"parentId":null}

Free summary and error rows report progress and endpoint issues. Search rows are charged but searches may legitimately return no matches.

Price and limits

Charged event: post-result for each post, comment, or newsletter row. Price: $1.00 per 1,000 charged rows. Summary and error rows are free. Public APIs can throttle requests, omit paid post bodies, or change their response shape. Requests time out after 20 seconds and transient rate-limit/server errors are retried with backoff. The actor fetches one publication at a time.

FAQ

Can it access paid posts? No authentication is used. Paid posts can be marked by their audience/access fields, but the API only provides text available publicly, often a preview.

Why is publication search empty? The endpoint returned HTTP 200 with an empty results list for the tested terms. Search is best-effort and may be limited by Substack.

Are custom domains supported? Yes, if the domain exposes the public /api/v1/archive and post endpoints.