Substack Scraper - Posts, Leaderboards & RAG

Turn any Substack newsletter — or the top newsletters of any category leaderboard — into clean, AI-ready data: engagement stats, full Markdown / text for free posts, RAG chunks with token estimates, and only-new-since-last-run monitoring.

No login. Paywall-safe: paid posts get metadata + public preview only — full content is never fetched.

Pay-per-event (when enabled by the owner):

Event	What you pay for	Price
post-metadata (primary)	One post: title, date, reactions, restacks, comments, paywall flag, preview	$1.00 / 1,000
post-full-content	Free post with Markdown/text (+ optional RAG chunks). Instead of metadata, never both	$2.00 / 1,000
leaderboard-publication	One leaderboard publication row (rank, author, public subscriber labels)	$2.00 / 1,000
apify-actor-start	Run start (default)	$0.00005

Respects Max cost per run. Do not keep apify-default-dataset-item enabled (would double-charge).

Why this Actor

Capability	This Actor
Leaderboard discovery → posts in one run	✅ paid / trending boards + newest posts
Only new posts since last run	✅ onlyNew + stateKey
RAG-ready chunks + estimated tokens	✅ chunkForRag, tokensEst
Markdown, plain text, HTML	✅ contentFormats
Paywall flag / free-only filter	✅ paid content never fetched
Engagement & reading time	✅ reactions, restacks, comments, word count
Custom domains	✅ e.g. www.lennysnewsletter.com

Use cases

AI / RAG knowledge bases · scheduled newsletter alerts · creator & market research · content benchmarking.

Input examples

Monitor publications (only new, Markdown + chunks):

{ "publications": ["www.lennysnewsletter.com", "astralcodexten"], "maxPostsPerPublication": 20, "chunkForRag": true, "onlyNew": true, "stateKey": "my-ai-feeds" }

Top rising Technology newsletters + latest free posts:

{ "leaderboardCategories": ["4"], "leaderboardType": "trending", "leaderboardTop": 10, "maxPostsPerPublication": 3, "onlyFree": true }

Category IDs (examples): 4 Technology, 62 Business, 96 Culture — full list: https://substack.com/api/v1/categories

Output

Two recordType values: post and publication. Dataset views: Posts, Leaderboard publications, RAG content. Run counters in KV RUN_STATS. See sample-output.json.

Responsible use

Public /api/v1/ endpoints only (not disallowed by Substack robots.txt). No login, no paid content, no comment authors, no emails. Default 400 ms delay. Token counts are estimates (~4 chars/token). Not affiliated with Substack Inc. Article copyright stays with authors.

Run locally

npm install && npm start (input: storage/key_value_stores/default/INPUT.json) or apify run.

Tutorial: feed Substack newsletters into your RAG app (and keep it fresh)

Goal: turn a set of Substack newsletters into Markdown chunks ready for embeddings, then pull only new posts every day.

Actor: Substack Scraper - Posts, Leaderboards & RAG (infinite_seascape/substack-rag-monitor). Pricing: $1 / 1,000 posts (metadata), $2 / 1,000 free posts with full content, $2 / 1,000 leaderboard rows, + $0.00005 per run start.

1. Pick your sources

Any of these work in publications: lenny, @astralcodexten, https://name.substack.com, a custom domain like www.lennysnewsletter.com, a /p/ post link, or substack.com/@handle.

Don't know which newsletters to follow? Discover them:

{ "leaderboardCategories": ["4"], "leaderboardType": "trending", "leaderboardTop": 10, "maxPostsPerPublication": 0 }

→ the 10 rising Technology newsletters with rank and public subscriber labels (category IDs: https://substack.com/api/v1/categories).

2. Get RAG-ready content

{
  "publications": ["www.lennysnewsletter.com", "@astralcodexten"],
  "maxPostsPerPublication": 30,
  "onlyFree": true,
  "contentFormats": ["markdown"],
  "chunkForRag": true,
  "chunkTokens": 500
}

Each free post includes markdown, tokensEst and chunks: [{ index, text, tokensEst }]. Paid posts are never fetched in full — with onlyFree: true they're skipped; otherwise you get metadata + public preview only.

3. Load into a vector DB (Python sketch)

from apify_client import ApifyClient
client = ApifyClient("<YOUR_TOKEN>")
run = client.actor("infinite_seascape/substack-rag-monitor").call(run_input={...})
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    if post["recordType"] != "post" or not post.get("chunks"):
        continue
    for c in post["chunks"]:
        upsert(id=f'{post["postId"]}-{c["index"]}', text=c["text"],
               metadata={"url": post["url"], "title": post["title"], "date": post["postDate"]})

(upsert = your Pinecone / Qdrant / pgvector / Chroma call.)

4. Keep it fresh

Add "onlyNew": true, "stateKey": "my-kb" and schedule daily (Console → Schedules, cron 0 6 * * *). Each run returns only posts not seen before for that stateKey, so you embed — and pay for — new content only.

5. Control spend

Set Max cost per run. The Actor stops cleanly at the limit and, if there's budget for metadata but not full content, it falls back to the cheaper metadata row.

Notes

• Uses public Substack JSON endpoints not disallowed by robots.txt; no login, no comments' author data, no emails.
• Token counts are estimates (~4 characters per token).
• Respect authors' copyright when reusing content. Not affiliated with Substack Inc.