Substack Posts Scraper

Export public Substack posts from publication URLs or custom domains into a structured dataset. Substack Posts Scraper collects titles, dates, public previews, authors and available engagement counts for newsletter research, content analysis and media monitoring.

It does not bypass paywalls. Body text and metadata depend on what each publication exposes publicly.

Quick start

1. Open the Actor and enter a publication homepage or domain.
2. Start with five posts and keep the other settings unchanged.
3. Run the Actor, then open the dataset to review and export the results.

{
  "publicationUrls": [{"url": "https://www.lennysnewsletter.com"}],
  "maxPostsPerPublication": 5,
  "includeBodyHtml": false,
  "includeRssFallback": true
}

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

Ready-to-run examples

Choose an example to start with a small, preconfigured input.

• Export Lenny’s Newsletter posts — Collect up to ten public posts from one product newsletter.
• Collect AI newsletter posts — Collect up to ten posts each from Import AI and One Useful Thing. These examples use the same extraction mode with different publication lists.

What you get

• Publication archives: One dataset row per saved post, with public identifiers and source context.
• Content and engagement: Titles, previews, available body text, authors, reaction counts and comment counts. This is not comment-text extraction.
• Focused collections: Optional publication archive search and inclusive date filters.
• Optional body HTML: Include the public body HTML when available. Enabling this does not unlock subscriber-only content.

Who is it for?

• Researchers and journalists: Compare public newsletter coverage and cite source post URLs.
• Content teams: Collect competitor topics and public engagement signals.
• AI workflow builders: Pass available post text to a separate summarization or classification step.

Input settings

Setting	JSON key	Meaning
Publication URLs or domains	publicationUrls	Publication homepages or domains. Use URL objects as in Quick start.
Maximum posts per publication	maxPostsPerPublication	1–1,000 saved posts per publication, default 20.
Search term	searchTerm	Optional publication archive search, not global Substack search.
Published after / before	dateFrom, dateTo	Inclusive dates in YYYY-MM-DD form.
Include body HTML	includeBodyHtml	Default false. Public text can still be returned without HTML.
Publication URLs to resume	resumePublicationUrls	Additional URLs processed before the main list.
Use feed fallback	includeRssFallback	Default true. A feed can return fewer fields or posts.
Publication concurrency	concurrency	1–10 publications at a time, default 3.

Provide at least one publication URL. The main and resume lists together support at most 100 entries. The form requires publicationUrls, so retain it when adding resume URLs.

For a topic-focused input, add "searchTerm": "AI" to the Quick start input. Search applies to publication archives only. Feed fallback does not apply the search term. Date filters apply to posts with a usable date, and records with missing or unparseable dates are retained.

Output data

One row represents one saved post. Fields can be null when absent from the public response. Feed results usually expose fewer fields.

Fields	Meaning
publicationInput, publicationUrl, publicationDomain	Original input and normalized publication context.
postId, publicationId, canonicalUrl, slug	Post and publication identifiers and public post location.
title, subtitle, socialTitle, description	Available titles and preview.
postDate, updatedAt	Publication and update dates as returned by the source.
authorName, byline	Public author attribution.
audience, isPaidOnly, isLimited, type	Available audience and content indicators. Feed flags do not establish paid-access status.
bodyText, truncatedBodyText, bodyHtml	Public text or preview. bodyHtml is included only when enabled.
wordCount, coverImage	Available length and image.
reactionCount, commentCount	Available counts, not reaction or comment records.
tags, section, sourceEndpoint	Tags, section and source context.

Illustrative shape only, not a recorded extraction:

{
  "publicationUrl": "https://example.substack.com",
  "canonicalUrl": "https://example.substack.com/p/example-post",
  "title": "Example post",
  "postDate": "2026-10-01T10:00:00.000Z",
  "authorName": null,
  "bodyText": "Public preview text",
  "reactionCount": null,
  "commentCount": null,
  "tags": []
}

The run’s key-value storage also contains RUN_SUMMARY with saved counts, publication failures and pending URLs when available. It is separate from the post dataset.

Download your results

Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See Apify's download and export guide for the available formats and field options.

Pricing

The start event is charged once after input validation when collection begins. A run can incur this charge even if no posts are saved.

The post event is charged for each public post saved to the dataset. Missing fields do not make a saved post free. Publication failures are summarized separately, not saved as charged post rows.

See the live Pricing tab for current rates and discounts.

Tips for best results

• Start small. Try five posts from one publication before increasing your list.
• Use publication homepages. This Actor collects publication archives, not individual post URL lookups or global search results.
• Check date coverage. Narrow windows and unavailable older content can leave few or no matching posts.
• Enable HTML only when needed. It increases dataset size without making private content available.
• Compare runs in your workflow. A later run can return posts you have already collected.

Limits and caveats

• Public data only. No login, subscriber-only bodies, email lists or paywall bypass.
• Search terms are not applied to feed fallback. Feeds may contain only recent entries.
• Results and field completeness depend on the publication. There is no guaranteed full archive export or selectable engagement sort.
• A run with no saved posts currently fails. Inspect RUN_SUMMARY for available failure or pending-work details rather than treating an empty run as proof that a publication is private or deleted.
• Some runs can return useful posts alongside publication failures. Review the summary before assuming every input completed.
• This is an independent tool, not an official Substack API.

Automation and integrations

Save your settings for next time

Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.

Run on a schedule

Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

Send results to your other tools

Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.

For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.

For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.

cURL

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~substack-posts-scraper/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"publicationUrls":[{"url":"https://www.lennysnewsletter.com"}],"maxPostsPerPublication":5}'

Node.js

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/substack-posts-scraper').call({
  publicationUrls: [{ url: 'https://www.lennysnewsletter.com' }],
  maxPostsPerPublication: 5,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("fetch_cat/substack-posts-scraper").call(run_input={
    "publicationUrls": [{"url": "https://www.lennysnewsletter.com"}],
    "maxPostsPerPublication": 5,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

MCP and AI agents

You can ask your AI assistant to collect data with Substack Posts Scraper and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.

Easier setup with an Apify plugin

If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

Use Apify AI in Console

Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's Substack Posts Scraper. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

Choose your assistant

• Claude and Claude Code (Anthropic): Apify setup guide.
• ChatGPT and Codex (OpenAI): Apify setup guides.
• Grok Bot (xAI): Apify plugin guide.
• Gemini CLI (Google): MCP server setup guide. Use the server URL below with the HTTP configuration supported by Gemini CLI.

Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/substack-posts-scraper"
    }
  }
}

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

Use Apify's general server

Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use Substack Posts Scraper by Fetch Cat, or give it the identifier fetch_cat/substack-posts-scraper.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

Example requests

• “Collect five recent public posts from Lenny’s Newsletter and give me a table of titles, dates and source links.”
• “Get ten posts each from Import AI and One Useful Thing, then summarize the available public previews by topic.”
• “Collect posts from my publication list published during September 2026 and prepare a spreadsheet of the available authors and engagement counts.”

Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.

Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.

Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

Other ways for agents to pay

Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.

FAQ

Can I export Substack posts without logging in?

Yes, this Actor uses public publication data. Availability varies and it cannot retrieve content hidden behind a subscriber login.

Can I download complete articles?

Only when public responses expose the content. bodyText may contain a preview, and enabling includeBodyHtml cannot bypass a paywall.

Is this a Substack API alternative?

It provides structured public post data through Apify’s API. It is not Substack’s official API and does not support account management, subscriber data or global publication discovery.

Can I search all Substack publications?

No. Supply publication URLs first. The optional search term applies within each publication’s archive, and feed fallback does not preserve that search filter.

Why did my run return no posts?

Check publication URLs, date filters and the run summary. A feed can contain fewer posts than an archive. No saved posts currently produces a failed run, not a guarantee that the target has no content.

Can I automate CSV or Excel exports?

Yes. Use the dataset export options or retrieve records through the API and add file creation to your workflow.

Related Actors

• Substack Notes Scraper — Public short-form Notes and engagement.
• Hacker News Search Scraper — Search technology stories and comments.
• Telegram Channel Posts Scraper — Public channel content.
• Product Hunt Scraper — Product launches and discussions.
• Google News Scraper — News headlines and publishers.

Support and feedback

For custom additions or integrations, email hello@fetchcat.dev.

Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. Leave a review from the Actor page when that option is available to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥