Substack Scraper: Posts, Full Text & Newsletter Data

Turn any Substack newsletter into clean, structured data. Paste newsletter URLs (custom domains work too) and get every post with its full text as LLM-ready Markdown, plain text and HTML, plus likes, comments, restacks, tags, authors, paid/free status and the newsletter's public subscriber count. Add a keyword to search all of Substack at once.

It reads the same public JSON that every Substack site loads in your browser, so it is fast (100 posts in about 5 seconds), cheap, and doesn't break when Substack changes its page design. No browser, no proxies, no login.

What you get

• Every post in the archive, newest first: title, subtitle, URL, authors, publish date, word count, cover image, tags, section, podcast link.
• Full text, three ways: clean Markdown (headings, links, lists, images, captions kept; subscribe buttons and share widgets removed), plain text, and the original HTML.
• Engagement: likes (reactions), comment count, restacks.
• Paid vs free: audience (free / paid / founding / subscribers_only) and a paywalled flag.
• Comments (optional): public comments and replies with author, date, likes and thread position.
• Newsletter profile: name, author, description, public subscriber count ("Over 1,200,000 subscribers"), paid tier, language, launch date.
• Search across Substack (optional): find posts about any topic from any newsletter.

Use cases

• AI, RAG and fine-tuning: feed newsletters into a vector database or a custom GPT as clean Markdown with source URLs and dates. Schedule it daily with since: "1 day" to keep the index fresh.
• Creator and market research: benchmark newsletters in your niche: posting cadence, free vs paid mix, what topics get the most likes, comments and restacks.
• Competitive intelligence: track what competitors, investors or analysts publish, the moment they publish it.
• Media monitoring and PR: search all of Substack for your brand, product or topic.
• Lead generation and partnerships: find active writers in a niche with their audience size for sponsorships, guest posts and outreach.
• Personal archives: back up your own newsletter, or a favorite one, as Markdown.

Input

Field	What it does	Default
publications	Newsletter URLs (https://name.substack.com), custom domains (www.lennysnewsletter.com), bare subdomains (lenny), author handles (@lenny) or single post URLs (…/p/slug)	2 examples
searchQuery	Keyword search across all of Substack	empty
maxSearchResults	Max posts from the search	20
maxPostsPerPublication	Newest posts first; 0 = whole archive	50
since	Only posts on/after a date (2026-01-01) or within a window (7 days)	any date
audience	all, free or paid posts	all
includeContent	Full text as Markdown, text and HTML	true
includeHtml	Also keep the raw HTML (turn off for smaller datasets)	true
includeComments	Add comments and replies to each post	false
maxCommentsPerPost	Comment cap per post	100
includePublicationInfo	Add author, description and subscriber count; save a PUBLICATIONS record	true

{
  "publications": ["https://www.lennysnewsletter.com", "oneusefulthing", "@garymarcus"],
  "maxPostsPerPublication": 100,
  "since": "2026-01-01",
  "includeContent": true,
  "includeComments": false,
  "searchQuery": "AI agents",
  "maxSearchResults": 50
}

Output

One row per post:

{
  "publication": "One Useful Thing",
  "publication_url": "https://www.oneusefulthing.org",
  "publication_id": 1180644,
  "post_id": 204003560,
  "title": "The twilight of the chatbots",
  "subtitle": "How work changes along the exponential",
  "slug": "the-twilight-of-the-chatbots",
  "url": "https://www.oneusefulthing.org/p/the-twilight-of-the-chatbots",
  "type": "newsletter",
  "author": "Ethan Mollick",
  "authors": [{ "name": "Ethan Mollick", "handle": "oneusefulthing", "id": 846835 }],
  "published_at": "2026-06-30T22:18:12.226Z",
  "audience": "free",
  "paywalled": false,
  "word_count": 1271,
  "reactions": 808,
  "comment_count": 103,
  "restacks": 78,
  "cover_image": "https://substackcdn.com/image/fetch/...",
  "tags": [],
  "description": "How work changes along the exponential",
  "content_markdown": "If you feel like things are accelerating in AI, you are probably right. Better AI models...\n\n![](https://substackcdn.com/image/...)\n\n...",
  "content_text": "If you feel like things are accelerating in AI, you are probably right...",
  "content_html": "<p>If you feel like things are accelerating in AI...</p>",
  "content_word_count": 1255,
  "publication_author": "Ethan Mollick",
  "publication_subscribers": "Over 488,000 subscribers",
  "source": "publication",
  "scraped_at": "2026-10-22T18:52:49.558Z"
}

With includeComments, each post also has comments: [{ comment_id, parent_id, depth, author_name, author_handle, body, published_at, edited_at, reactions, restacks }].

The key-value store also holds:

• PUBLICATIONS: one full profile per newsletter (name, subdomain, custom domain, author, description, subscriber count, paid tier, language, podcast, logo, created date, first post date).
• SUMMARY: posts found per input, and why an input failed (for example, a site that isn't on Substack).

Pricing

Pay only for what you get. Full text is included in the post price.

Event	Price	1,000 =
Post (metadata + full text)	$0.001	$1.00
Comment (optional)	$0.0003	$0.30

No extra run-start fee beyond Apify's minimal default. Inputs that aren't Substack newsletters cost nothing. Set a maximum cost per run in Apify and the Actor stops cleanly when it is reached.

FAQ

Does it get paid (paywalled) posts? It returns exactly what Substack shows any visitor who isn't a paying subscriber: the title, stats and the free preview of the post, with paywalled: true and audience: "paid". It never tries to get around a paywall. Comments on paid posts are usually hidden from non-subscribers too, so they come back empty. Use audience: "free" to skip paid posts entirely.

Do custom domains work? Yes. www.lennysnewsletter.com, www.noahpinion.blog, newsletter.pragmaticengineer.com and any other Substack on its own domain work the same as name.substack.com. Subdomains that redirect to a custom domain are followed.

What's the difference between word_count and content_word_count? word_count is Substack's own count for the full post. content_word_count is the number of words actually returned, which is lower for paywalled previews.

How fresh is the data? Live. Every run reads the newsletter at that moment. Schedule runs with since: "1 day" to collect only new posts.

Is the Markdown good enough for an LLM? Yes. Substack's subscribe boxes, share buttons and image-expand clutter are removed, while headings, lists, links, quotes, images and captions are kept.

Is subscriber count exact? Substack only shows rounded public figures ("Over 121,000 subscribers", "Tens of thousands of paid subscribers"), and that's what you get.

Is this allowed? The Actor only reads public pages and public data that Substack serves to every visitor, at a polite rate, with no login. You are responsible for how you use the content: respect the authors' copyright and Substack's terms, and don't republish full posts without permission.

Something missing? Open an issue on the Actor's Issues tab and it will usually be added quickly.