Substack Intelligence Scraper - Posts & Comments

🔎 What does Substack Intelligence Scraper do?

Substack Intelligence Scraper turns public Substack publications into structured post, comment, author, and newsletter datasets. It works with *.substack.com publications, custom domains, archive pages, and individual post URLs.

Use it as a fast Substack API alternative for competitive research, newsletter monitoring, content analysis, sponsorship research, and AI-ready corpora. The Actor uses Substack's public HTTP endpoints and server-rendered data, so it does not need a browser or customer-managed proxy settings.

Key capabilities include:

• Complete public archive pagination with 0 for unlimited depth
• Public post HTML, normalized plain text, and Markdown
• Nested comment threads plus optional analytics-friendly comment rows
• Publication, audience, author, tag, podcast, reaction, and restack metadata
• Date, post type, paywall, include-keyword, and exclude-keyword filters
• Direct collection from a single /p/... post URL
• Persistent onlyNew monitoring for scheduled runs
• Per-source and per-post error rows, so one bad URL does not discard a batch

Ready-to-run examples

• Monitor new Substack newsletter posts
• Export Substack comments and replies
• Build a Substack RAG dataset

👥 Who is this Substack scraper for?

• Newsletter teams monitoring competitors and topic coverage
• Researchers comparing publishing cadence and reader engagement
• Sponsorship and partnership teams evaluating public publications
• Data journalists studying newsletter ecosystems
• AI teams preparing searchable text or RAG datasets
• Creators auditing their own public archives

⚙️ How to scrape Substack newsletters

1. Add one or more newsletter homepages, custom domains, archive pages, or post URLs.
2. Keep the safe default of 50 posts per publication or use 0 for a complete archive.
3. Choose whether to fetch public content and select HTML, text, Markdown, or all formats.
4. Optionally enable comments and select nested, separate-row, or combined output.
5. Apply date, content-type, paywall, or keyword filters.
6. Start the Actor and open the Output tab.
7. Download JSON, CSV, Excel, XML, RSS, or connect through the Apify API.

Apify can run the same configuration on a schedule, send a webhook when it finishes, and connect the dataset to Make, Zapier, Google Sheets, cloud storage, or your own application.

📥 Input

See the Input tab for every option. A typical run looks like this:

{
    "urls": ["https://example.substack.com", "https://newsletter.example.com/p/a-public-post"],
    "maxPostsPerNewsletter": 100,
    "includeContent": true,
    "contentFormat": "all",
    "includeComments": true,
    "commentOutput": "both",
    "includePublicationInfo": true,
    "publicationOutput": "both",
    "contentType": "all",
    "onlyFree": false,
    "includeKeywords": ["AI", "product strategy"],
    "onlyNew": false
}

maxPostsPerNewsletter limits matching posts, not archive pages scanned. Keyword matching is case-insensitive and checks titles, subtitles, descriptions, public preview text, and tags.

📤 What Substack data can be extracted?

Data	Important fields
Post identity	postId, title, subtitle, slug, url
Timing and access	publishedAt, updatedAt, postType, audience, isPaid
Public content	bodyHtml, bodyText, bodyMarkdown, truncatedBodyText
Engagement	reactionCount, reactions, commentCount, restacks
Discovery	tags, language, coverImage, description, wordCount
Authors	authors, authorName, authorHandle, public bios and photos
Publication	name, URL, subscriber signals, branding, benefits, paid-plan status
Comments	body, date, author, reactions, pinned status, reply depth and parent ID

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. The Output tab includes dedicated views for posts, comments, publications, and all records.

Example post row:

{
    "itemType": "post",
    "postId": 123456789,
    "title": "A practical guide to product strategy",
    "url": "https://example.substack.com/p/product-strategy",
    "publishedAt": "2026-08-01T12:00:00.000Z",
    "postType": "newsletter",
    "isPaid": false,
    "wordCount": 2450,
    "tags": ["Product", "Strategy"],
    "reactionCount": 120,
    "commentCount": 18,
    "bodyText": "Public article content...",
    "bodyMarkdown": "## Public article content...",
    "publicationName": "Example Newsletter",
    "scrapedAt": "2026-08-09T12:00:00.000Z"
}

💵 How much does it cost to scrape Substack?

The recommended Store pricing is pay per event with no start fee: $2.00 per 1,000 posts and $0.50 per 1,000 comments, with lower prices for higher Apify subscription tiers. Publication and recoverable error rows are free. Comments are charged once when collected, whether you choose nested, separate, or combined comment output.

For cost-efficient runs:

• Start with includeContent: false to survey metadata quickly.
• Keep comments disabled unless reader discussion is required.
• Use a date window for recurring research.
• Keep the default 50-post cap for the first run.

🔁 How to monitor new Substack posts

Enable onlyNew and run the same input on an Apify schedule. The Actor stores emitted post IDs in a configuration-specific named key-value store. Later runs suppress previously delivered posts while preserving separate state for different URL and filter combinations.

For RSS delivery, request the last task dataset in RSS format and map title, url, description, and publishedAt to feed fields.

🛠️ Substack scraper API

Run the Actor from Node.js with apify-client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('webdata_labs/substack-intelligence-scraper').call({
    urls: ['https://on.substack.com'],
    maxPostsPerNewsletter: 20,
    contentFormat: 'text',
    onlyNew: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

The same input works through the REST API, Apify CLI, webhooks, schedules, and MCP integrations.

⚠️ Limits and responsible use

The Actor collects only data available without authentication. It does not log in, bypass access controls, or return paid post bodies. Paid posts can still expose public metadata and preview text. Substack can change undocumented public endpoints or page payloads, so recoverable failures are emitted as structured error rows.

Public comments and author profiles can contain personal data. Use the dataset only for a legitimate purpose and comply with Substack's terms, GDPR, CCPA, copyright rules, and other applicable laws. If you are unsure whether your use is legitimate, consult qualified legal counsel.

❓ FAQ and support

Does it support custom Substack domains?

Yes. Add the full custom-domain URL and the Actor discovers its canonical publication metadata.

Can it scrape a single post?

Yes. A URL containing /p/post-slug is treated as a direct post and does not expand into the full archive.

Why is a paid post body empty?

The Actor intentionally omits paid post bodies. Public metadata, engagement, tags, word count, and preview text can still be returned.

How are duplicate inputs handled?

Duplicate source strings, publication archives, and post IDs are deduplicated within each run. onlyNew extends post deduplication across scheduled runs with the same monitoring configuration.

For bugs or feature requests, use the Actor's Issues tab and include the public source URL plus the run ID. Do not include private credentials or subscription cookies.