Substack Posts Scraper - Newsletter Data Extractor

Extract posts, engagement metrics, and newsletter data from Substack publications. Perfect for content research and competitive analysis.

Substack Posts Scraper collects structured data from any public Substack publication through its public archive API. Give it one or more publication URLs and it returns every post with its title, subtitle, author, publish date, engagement metrics (likes, comments, restacks), word count, cover image, tags, and a text excerpt - no authentication required.

Use Cases

• Content research — Analyze what topics, formats, and headlines successful newsletters publish.
• Competitive analysis — Track engagement metrics (likes, comments, restacks) of posts in your niche.
• Trend monitoring — Watch a publication's posting cadence, tag usage, and audience targeting over time.
• Data journalism — Build datasets of newsletter content for reporting on the newsletter economy.
• Market intelligence — Identify high-performing publications and content patterns before entering a niche.

Input

Field	Type	Required	Description
startUrls	Array	Yes	Substack publication URLs to scrape (e.g. https://newsletter.pragmaticengineer.com). Post URLs are also accepted - the publication is derived automatically.
maxItems	Number	No	Maximum number of posts to scrape per publication, newest first (default: 100, max: 5000).
proxyConfiguration	Object	No	Proxy settings. The public Substack API usually does not require a proxy, but one can be enabled for high-volume runs.

Output

The Actor outputs a dataset with one record per post:

{
  "url": "https://newsletter.pragmaticengineer.com/p/what-is-technology",
  "slug": "what-is-technology",
  "title": "What is Technology?",
  "subtitle": "A framework for understanding the tech industry",
  "description": "A framework for understanding the tech industry",
  "publishedAt": "2024-05-14T07:00:00.000Z",
  "author": "Gergely Orosz",
  "authorHandle": "pragmaticengineer",
  "publicationName": "The Pragmatic Engineer",
  "publicationUrl": "https://newsletter.pragmaticengineer.com",
  "likes": 523,
  "comments": 87,
  "restacks": 41,
  "reactions": { "\u2764\uFE0F": 523 },
  "wordCount": 3120,
  "excerpt": "I keep being asked what \"technology\" actually means...",
  "coverImage": "https://substackcdn.com/image/fetch/...",
  "language": "en",
  "tags": ["technology", "career"],
  "type": "newsletter",
  "audience": "everyone",
  "scrapedAt": "2026-10-05T08:45:00.000Z"
}

A STATS record in the default key-value store summarizes the run (posts scraped, publications processed, errors).

Pricing

Pay per event: $0.002 per post scraped (post-scraped event). You are only charged for posts actually extracted and delivered to your dataset.

Limitations

• Only public posts from public publications are accessible; paywalled content is not extracted.
• Full post bodies are not included - the output carries the official text excerpt, engagement metrics, and metadata. Full text is available on the canonical URL.
• The Substack archive API returns at most 50 posts per request; very large maxItems values take proportionally longer.
• Engagement metrics reflect the values at scrape time and change over time.

FAQ

Q: Can I scrape multiple publications in one run? A: Yes. Add as many publication URLs to startUrls as you like. The maxItems cap applies per publication.

Q: Do I need a proxy? A: Usually no - the public Substack API responds to direct requests. For sustained high-volume runs you can enable the Apify proxy via proxyConfiguration.

Q: What URL formats are accepted? A: Publication homepages (https://name.substack.com), custom domains (https://newsletter.example.com), and individual post URLs all work.

Support

Open an issue on this Actor's Apify page and I will take a look.

Changelog

• v0.0.2 — Source recovery build: reconstructed Actor with full input/output/dataset schemas, README, and resilient pagination over the public archive API.
• v0.0.1 — Initial release.