Substack Scraper — Publication Feeds and Posts

What is Substack Scraper?

Substack Scraper extracts newsletter posts and publication metadata into structured data you can export as CSV, Excel or JSON. Paste newsletter URLs, choose your result limits, and collect titles, links, dates, public author names and optional excerpts. Substack subdomains and custom publication domains are supported.

Use it to research a newsletter shortlist, build a publication content index, or collect new and changed posts through repeat runs. Start with recent RSS/Atom posts; optional sources provide bounded older-post metadata or public post links by year.

• Simple input: paste URLs directly; no code is needed in Apify Console.
• Useful filters: select posts by date or a word or phrase.
• Exportable results: download data or retrieve it through the Apify API.
• Incremental collection: supply a saved checkpoint to skip unchanged records.
• Visible coverage: inspect source errors, missing values and the limits reached.

Notes, replies, private feeds and paid article bodies are unsupported. No source guarantees a complete archive. Use publications you are authorized to process.

How to scrape Substack newsletters

1. Open Substack Scraper in Apify Console and select Input. The Actor is currently private, so your account needs access.
2. Paste publication links into Newsletter or post URLs. Use Bulk edit to paste several URLs, one per line. An empty list uses your existing API selection or, by default, the Simon Willison example.
3. Set Total results and Results per newsletter. Both default to 10. Raise both to collect more than 10 posts from one newsletter.
4. Set Maximum cost per run in the run options, then select Start. The default source collects recent posts with excerpts of up to 300 characters.
5. Open the dataset and export JSON, CSV or Excel. The Output tab also links to the coverage report and continuation checkpoint.

Leave the optional sections collapsed for your first run. Filter posts contains dates and keywords; Content and output changes text and record type; Older posts and year indexes selects other sources. Advanced limits and continuation controls source requests, work time and checkpoints.

Substack Scraper pricing

$0.15 per 1,000 saved results + $0.002 per start. Platform usage during the run is included. The same rate applies to all Apify plans, with no Actor subscription or feature add-ons.

Saved results	Charge with one start (USD)
0	$0.002
10	$0.0035
100	$0.017
1,000	$0.152
10,000	$1.502

These are price calculations, not guaranteed collection volumes. The formula is $0.002 × starts + $0.00015 × saved results. Resurrecting an interrupted run can incur another start fee. At the supported 128–256 MB memory, each start incurs one start event.

Each delivered post, publication record, archive metadata row or year-index link is one result. Updated records delivered later count again. Unchanged records skipped with a supplied checkpoint, suppressed duplicates, filtered items and source errors have no result charge. Empty or failed runs that start still incur the start fee. Independent runs without a checkpoint can return and bill the same records again.

Use Maximum cost per run to cap spending. The minimum selectable cap is $0.002; allow at least $0.00215 for a start and one result. A spending limit can return partial results and a checkpoint. Apify billing is authoritative, and platform interruption can prevent a final checkpoint save.

Long scans that produce few new results may stop early under the included-usage allowance. In Coverage and errors, operating_cost_limit identifies this stop; operating_cost_unavailable means usage could not be checked safely. Saved results remain available, and a checkpoint is saved when possible. Review the coverage and narrow the selection before continuing; every new start has its own start fee.

Your Apify account subscription, applicable taxes and storage retained after the run are governed separately by Apify. There is no automatic increase or expiry attached to this introductory Actor rate.

What Substack data can you extract?

Choose Where to find posts to select a source:

Source	What you receive	Coverage
Recent posts (default)	Current RSS/Atom entries with post metadata and optional feed text	Current feed window only
Older posts (experimental)	Bounded archive metadata, optional public subtitles and observed engagement where supplied	Configured pages; publication sections may be omitted
Post links by year (advanced)	Titles and URLs from selected public year indexes	Selected indexes only; dates, authors, text and engagement remain unknown

To return one publication metadata record per newsletter, choose Content and output → Return → Newsletter details only and keep Recent posts as the source. Publication records include the name, URL, description, available public author names and observed feed item count. Subscriber counts and subscription prices are unsupported and remain null.

The experimental archive source uses an undocumented public site route. This Actor runs through the Apify API; it is not an official Substack Developer API integration. Neither public access nor a successful run establishes commercial reuse permission.

Text, previews and engagement

Short excerpt uses the feed description or public archive subtitle. Text supplied by the feed uses text already present in RSS/Atom and works only with Recent posts. No text returns metadata. maxTextCharacters limits plain text; textTruncated reports clipping.

Feed text may be incomplete or a preview. bodyText stays null; feed accessStatus is unknown and isPreviewOnly is null. Archive audience labels can identify free or paid access, but do not establish how much text a visitor can read. Missing engagement values stay null. Observed reactions, comments and restacks are counts, not audience reach.

Substack scraper input examples

This example requests up to 20 recent posts across two newsletters, with at most 10 from each:

{
  "startUrls": [
    {"url": "https://simonw.substack.com"},
    {"url": "https://www.lennysnewsletter.com"}
  ],
  "maxItems": 20,
  "maxPostsPerPublication": 10
}

Publication homepages, /feed, /archive and /p/post URLs are accepted. For JSON/API input, startUrls accepts URL strings, bare names such as simonw, objects containing url, or a mixture. The Console form uses newsletterUrls with URL objects; a non-empty newsletterUrls list takes precedence over startUrls. Leave it empty or omit it to use startUrls. Add URLs directly; hosted URL-list files are unsupported. The collection setting selects the source; a URL ending in /archive does not switch it automatically.

These API inputs select the same newsletter:

{"startUrls": ["https://simonw.substack.com"]}

{"startUrls": ["simonw"]}

{"startUrls": [{"url": "https://simonw.substack.com"}]}

Input settings reference

JSON/API field	Default	Purpose
newsletterUrls	[]	Console URL list as {"url": "https://..."} objects; up to 100; a non-empty list replaces startUrls
startUrls	Simon Willison example URL	API-compatible URL strings, bare names or URL objects; up to 100; used when newsletterUrls is empty
maxItems	10	Total dataset records; up to 10,000
maxPostsPerPublication	10	Delivered posts per newsletter; up to 2,000
keyword	Empty	Case-insensitive phrase in titles/descriptions; titles only for year indexes
publishedAfter	Unset	Inclusive start date or ISO timestamp, using UTC
publishedBefore	Unset	Exclusive end date or ISO timestamp, using UTC
mode	"posts"	Posts or "publications" metadata
textMode	"excerpt"	"none", "excerpt" or "feed"
maxTextCharacters	300	Maximum plain-text characters per post
collection	"feed"	"feed", "archive" or "sitemap"
maxArchivePages	2	Older-post pages per newsletter; up to 100
sitemapYears	[]	One to five explicit years for year-index discovery
maxRequests	20	Source requests including redirects and retries; up to 120
maxRunSeconds	90	Collection work-time budget, in seconds
resumeState	{}	Previous CHECKPOINT JSON for incremental collection

Date and keyword filters apply to posts, not publication records. The end date excludes the selected day. Date-filtered posts with missing or invalid dates are excluded and reported. Exact date filters are unavailable for year indexes. Keyword filtering applies only to your supplied newsletters, not platform-wide search.

Collect older Substack post metadata

{
  "startUrls": [{"url": "https://simonw.substack.com"}],
  "collection": "archive",
  "maxArchivePages": 2,
  "maxItems": 24,
  "maxPostsPerPublication": 24,
  "maxRequests": 3,
  "textMode": "none"
}

Each archive page requests up to 12 entries, plus one feed lookup per newsletter. Short pages do not prove that history has ended. Collection stops at an observed empty/repeated page, access block or configured limit. archiveComplete remains false. Older posts requires mode: "posts" and textMode: "none" or "excerpt".

Discover Substack post links by year

{
  "startUrls": [{"url": "https://www.lennysnewsletter.com"}],
  "collection": "sitemap",
  "sitemapYears": [2026],
  "maxItems": 10,
  "maxPostsPerPublication": 10,
  "maxRequests": 2,
  "textMode": "none"
}

Each selected year adds one request after the publication feed lookup. These results are labelled contentCoverage: "index_only". The index year is not an inferred publication date. Use posts mode with none/excerpt text settings; this source provides no post text.

Output: export Substack data to CSV, Excel or JSON

The default dataset contains your results. The Output tab also provides Coverage and errors (OUTPUT) and Incremental checkpoint (CHECKPOINT).

Post records include publication details, title, URL, source date, available author names and tags, observation time and coverage. Unsupported values remain null. Feed entries can include audio/video; non-post items and archive restacks are labelled separately. Notes and replies are not returned.

Abbreviated post metadata example with text omitted:

{
  "recordType": "post",
  "publicationTitle": "Simon Willison’s Newsletter",
  "title": "I built a new feature for my blog using my voice",
  "url": "https://simonw.substack.com/p/i-built-a-new-feature-for-my-blog",
  "publishedAt": "2026-10-12T22:29:32Z",
  "bodyText": null,
  "accessStatus": "unknown",
  "isPreviewOnly": null,
  "coverage": "current_feed_only",
  "archiveComplete": false,
  "changeType": "new"
}

Check OUTPUT.outcome alongside the Apify run status. COMPLETE means the observed source windows were processed; it never proves a complete archive. PARTIAL means a limit, missing target or source issue left work unfinished. FAILED means source failures prevented a usable result; CANCELLED records a cooperative stop. A successful ten-result run can legitimately report PARTIAL because it reached your result limit.

Run Substack Scraper through the API with Python

Use the Apify Python client with an Apify token that has access to this Actor. This example uses the version 3.3 client API and caps the Actor charge at $0.005:

from decimal import Decimal

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("thescrapelab/substack-scraper").call(
    run_input={
        "startUrls": [{"url": "https://simonw.substack.com"}],
        "maxItems": 10,
    },
    max_total_charge_usd=Decimal("0.005"),
)
if run is None:
    raise RuntimeError("No Actor run returned")

for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["title"], item["url"])

The dataset iterator handles pagination. Inspect the run's OUTPUT report for coverage and source errors before treating the exported data as complete.

Collect new posts and resume partial runs

Copy the previous CHECKPOINT into Continue from checkpoint (resumeState) for the next run. Keep the collection source, output mode, filters and text settings the same, including selected sitemap years. Result, request, page and time limits may change. Use {} for a new selection configuration. Independent runs do not automatically share state.

Feed continuation rechecks incomplete feeds and can conditionally recheck completed feeds. Archive continuation rescans from the first page and skips unchanged records; increase maxArchivePages to reach further back. Year-index continuation rechecks the selected years. Rechecking can use requests even when no new rows are delivered.

Recovering the same run reconciles existing dataset records before adding rows. Check the dataset and last checkpoint after interruption: valid rows may exist without a final summary. A forced interruption can prevent cleanup. When combining feed, archive and year-index exports, match by post URL; record IDs differ between sources.

FAQ and troubleshooting

Can I scrape every Substack post or Substack Notes?

Complete archives, all-section coverage, platform-wide search, Notes and replies are unsupported. A post absent from the selected source window is not classified as deleted. Publication domain changes may require record identity reconciliation.

Can this scraper read paid articles?

It does not log in, bypass paywalls or fetch paid article bodies. Feed excerpts and archive subtitles may be public previews. Access and preview flags remain unknown when the source does not establish them.

Why did I get fewer results, no results or a partial result?

Open Coverage and errors. Check result limits, filters, unchanged checkpoint records, unavailable post URLs, source errors and access blocks. Raise Results per newsletter as well as Total results when needed. A blocked source stops collection. Avoid repeatedly starting fresh runs without reviewing the report and checkpoint.

Collection uses one source request at a time, at least 1.1 seconds apart. Limits include 100 input URLs, 120 source requests, five MB per source response/decompression and ten MB of record output per run. Smaller configured limits take priority. The work budget defaults to 90 seconds, with a 35-second reserve before the Apify timeout for saving progress. Set the platform timeout long enough for both. Temporary connection failures may retry once; retries count toward the request limit. No identity rotation or challenge bypass is used.

What permission do I need to use the data?

Use sources you are authorized to process and follow applicable platform and publisher conditions for reuse, attribution and redistribution. Public visibility alone does not establish commercial permission. No Substack API account is required by this Actor, and its experimental archive route is not an official supported API.

How do I get support?

Use the Actor's Issues tab when available, or the support contact provided with your private access. Include the run link, build number, sanitized input and a small example of expected versus observed metadata. Never send tokens, private feed URLs, paid article bodies or subscriber lists. Support is best effort, without a fixed response-time SLA. Direct Apify account and invoice questions to Apify support.

How are results retained or deleted?

Review your dataset and default key-value store in Apify Console under Storage. They can contain results, input, summaries and checkpoints. Export what you need before using Apify's normal deletion controls; deleting a checkpoint removes its continuation history. Copies and exports elsewhere need separate handling.

The Actor does not set a universal retention period, automatically delete exports or interpret a missing source item as a deletion request. Follow the retention, refresh, attribution and deletion conditions attached to your authorized use.